Skip to main content
Version: 1.4

Row Format

Apache Fory™ provides a random-access row format that enables reading nested fields from binary data without full deserialization.

Overview​

Row format drastically reduces overhead when working with large objects where only partial data access is needed. It also supports memory-mapped files for ultra-low memory footprint.

Key Benefits:

FeatureDescription
Zero-Copy AccessRead nested fields without deserializing entire object
Memory EfficiencyMemory-map large datasets directly from disk
Cross-LanguageBinary format compatible between Python, Java, C++
Partial DeserializationDeserialize only specific elements you need
High PerformanceSkip unnecessary data parsing for analytics workloads

Basic Usage​

from dataclasses import dataclass
from typing import Dict, List

import pyfory

@dataclass
class Bar:
    f1: str
    f2: List[pyfory.Int64]

@dataclass
class Foo:
    f1: pyfory.Int32
    f2: List[pyfory.Int32]
    f3: Dict[str, pyfory.Int32]
    f4: List[Bar]

# Create encoder for row format
encoder = pyfory.encoder(Foo)

# Create large dataset
foo = Foo(
    f1=10,
    f2=list(range(1_000_000)),
    f3={f"k{i}": i for i in range(1_000_000)},
    f4=[Bar(f1=f"s{i}", f2=list(range(10))) for i in range(1_000_000)]
)

# Encode to row format
binary: bytes = encoder.to_row(foo).to_bytes()

# Zero-copy access - no full deserialization needed!
foo_row = pyfory.RowData(encoder.schema, binary)
print(foo_row.f2[100000])              # Access 100,000th element directly
print(foo_row.f4[100000].f1)           # Access nested field directly
print(foo_row.f4[200000].f2[5])        # Access deeply nested field directly

PyArrow Schema Conversion​

Row format can convert PyArrow schemas through pyfory.format when the format optional dependency is installed:

import pyarrow as pa
from pyfory.format import from_arrow_schema, to_arrow_schema

arrow_schema = pa.schema(
    [
        pa.field("id", pa.int32(), nullable=False),
        pa.field("scores", pa.list_(pa.float64())),
    ]
)

fory_schema = from_arrow_schema(arrow_schema)
roundtrip_arrow_schema = to_arrow_schema(fory_schema)

This PyArrow conversion surface is separate from cross-language dense-array field annotations. In object serialization, pyfory.PyArray[T] means the standard-library Python array.array carrier, not PyArrow.

Cross-Language Compatibility​

Row format works seamlessly across languages. The same binary data can be accessed from Java and C++.

Java​

public class Bar {
  String f1;
  List<Long> f2;
}

public class Foo {
  int f1;
  List<Integer> f2;
  Map<String, Integer> f3;
  List<Bar> f4;
}

RowEncoder<Foo> encoder = Encoders.bean(Foo.class);

// Encode to row format (cross-language compatible with Python)
BinaryRow binaryRow = encoder.toRow(foo);

// Zero-copy random access without full deserialization
BinaryArray f2Array = binaryRow.getArray(1);              // Access f2 list
BinaryArray f4Array = binaryRow.getArray(3);              // Access f4 list
BinaryRow bar10 = f4Array.getStruct(10);                  // Access 11th Bar
long value = bar10.getArray(1).getInt64(5);               // Access 6th element of bar.f2

// Partial deserialization - only deserialize what you need
RowEncoder<Bar> barEncoder = Encoders.bean(Bar.class);
Bar bar1 = barEncoder.fromRow(f4Array.getStruct(10));     // Deserialize 11th Bar only
Bar bar2 = barEncoder.fromRow(f4Array.getStruct(20));     // Deserialize 21st Bar only

C++​

#include "fory/encoder/row_encoder.h"
#include "fory/row/writer.h"

struct Bar {
  std::string f1;
  std::vector<int64_t> f2;
  FORY_STRUCT(Bar, f1, f2);
};

struct Foo {
  int32_t f1;
  std::vector<int32_t> f2;
  std::map<std::string, int32_t> f3;
  std::vector<Bar> f4;
  FORY_STRUCT(Foo, f1, f2, f3, f4);
};

fory::row::encoder::RowEncoder<Foo> encoder;
encoder.encode(foo);
auto row = encoder.get_writer().to_row();

// Zero-copy random access without full deserialization
auto f2_array = row->get_array(1);                   // Access f2 list
auto f4_array = row->get_array(3);                   // Access f4 list
auto bar10 = f4_array->get_struct(10);               // Access 11th Bar
int64_t value = bar10->get_array(1)->get_int64(5);   // Access 6th element of bar.f2
std::string str = bar10->get_string(0);              // Access bar.f1

Installation​

Row format requires Apache Arrow:

pip install pyfory[format]

When to Use Row Format​

  • Analytics workloads: When you only need to access specific fields
  • Large datasets: When full deserialization is too expensive
  • Memory-mapped files: Working with data larger than RAM
  • Data pipelines: Processing data without full object reconstruction
  • Cross-language data sharing: When data needs to be accessed from multiple languages