> ## Documentation Index
> Fetch the complete documentation index at: https://docs.bloklake.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Export to Parquet and DuckDB

> Write query results to Parquet files and analyze them with DuckDB, Polars or any warehouse.

Arrow responses map one-to-one onto Parquet: no conversion, no type mapping. A common setup is to
pull raw rows from Bloklake into Parquet, then do the joins, aggregations and lookups Bloklake
doesn't do in DuckDB, Polars, ClickHouse, BigQuery or Spark.

> [!NOTE]
>
> This guide reads Arrow directly with the [Arrow reader](https://docs.bloklake.com/responses/reading-arrow.md#complete-readers)
> (`bloklake_arrow.py`): its tables write to Parquet without any conversion. The
> [client libraries](https://docs.bloklake.com/clients.md) return JSON rows instead, which suit application code better.

## One file per range

Write each page as you receive it, so memory stays flat however large the export:

```python theme={null}
import pyarrow.parquet as pq
from bloklake_arrow import query

def export(body: dict, start: int, end: int, path: str, row_type: str = "logs") -> int:
    writer, frm, rows = None, start, 0
    while frm <= end:
        tables, meta = query({**body, "fromBlock": frm, "toBlock": end})
        table = tables.get(row_type)
        if table is not None and table.num_rows:
            writer = writer or pq.ParquetWriter(path, table.schema, compression="zstd")
            writer.write_table(table)
            rows += table.num_rows
        frm = int(meta["bl.next_block"])
    if writer:
        writer.close()
    return rows
```

The response schema carries the `bl.*` metadata of the first page. Strip it if your tools
complain: `table.replace_schema_metadata(None)`.

## Partitioned exports

For a large history, write one file per block range, for example every 100,000 blocks. You can
then re-export, verify or delete a single partition:

```python theme={null}
for lo in range(START, END + 1, 100_000):
    hi = min(lo + 99_999, END)
    export(body, lo, hi, f"transfers/blocks={lo:010d}-{hi:010d}.parquet")
```

Only export ranges at or below `finalized_block`, so a partition never has to be rewritten after
a reorg. See [Finality and reorgs](https://docs.bloklake.com/querying/finality-and-reorgs.md).

## Query with DuckDB

```sql theme={null}
-- topics are 32-byte BLOBs: the address is the last 40 hex characters
select '0x' || lower(substr(hex(topic2), 25)) as receiver,
       count(*) as transfers
from 'transfers/*.parquet'
group by 1
order by transfers desc
limit 10;
```

DuckDB reads the binary columns as `BLOB`. `hex()` turns one into 64 uppercase hex characters;
`substr(…, 25)` keeps the last 40 (the address; DuckDB counts from 1), and `lower()` gives the
usual lowercase form.

## Big amounts

`value`, gas prices and other amounts are 16-byte big-endian binary (32 bytes for the Arbitrum
retryable amounts). Parquet has no unsigned 128-bit integer. Three options:

* **Convert to a decimal string** when you write (simplest, as in
  [Index ERC-20 transfers](https://docs.bloklake.com/guides/erc20-transfers.md)), and cast to `HUGEINT` or `DECIMAL(38,0)`
  in queries.
* **Convert to `decimal128(38, 0)`**: fits any 128-bit value below 10³⁸ (every realistic wei
  amount), and engines can sum it natively.
* **Keep the bytes** and convert in queries when needed.

```python theme={null}
import pyarrow as pa

def to_decimal(col: pa.ChunkedArray) -> pa.Array:
    return pa.array([None if b is None else int.from_bytes(b, "big") for b in col.to_pylist()],
                    pa.decimal128(38, 0))
```

## Into a warehouse

* **ClickHouse**: `INSERT INTO t FORMAT ArrowStream` accepts the response body of a
  single-stream query directly, or load the Parquet files.
* **BigQuery, Snowflake, Spark**: load the Parquet files from object storage.

Keep the original `block_number` and `log_index` (or `transaction_index`) on every row. Together
they're a unique, stable key for re-runs and deduplication.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.