ztreamer vs lightwalletd

Summary

ztreamer reaches a serving index of the whole chain in 2.4 minutes when it runs zakura in process, and in 5.7 minutes in remote mode from the published snapshot. lightwalletd took 1 hour 44 minutes to build its cache from the same node. Once synced, ztreamer serves compact blocks 3 to 22 times faster than lightwalletd, depending on the request, with a lower tail latency at every concurrency level, and it uses about a tenth of the CPU for each block it serves. Its index is 40 % smaller on disk than the cache of lightwalletd. The two ztreamer modes serve at the same speed. One defect showed up under load: with 256 concurrent connections ztreamerd fails some requests with MDB_READERS_FULL.

• Sync: direct 146 s (44 s to open the embedded zakura, then the historical pass indexed 3,568,258 blocks in 102 s, about 35,000 blocks/s); remote 339 s (an 18.2 GB import of 409 epochs, attested by the node, then a 30 ms tail); lightwalletd 6,218 s. Direct is 42 times and remote 18 times faster than lightwalletd.
• Sync cost: direct used 502 CPU seconds; remote 4,500 (it verifies the imported ranges on about 24 cores); lightwalletd and its node 11,484, of which the node took 9,000 to answer one getblock at a time.
• Wallet download (every compact block from Sapling activation, 3,145,415 blocks, 17.9 GB of protobuf): 22.6 s on both ztreamer modes, 65.2 s on lightwalletd. ztreamer used 4 to 5 CPU seconds for it, lightwalletd 87.
• Ranges: the median 10,000-block GetBlockRange takes 4.8 ms on ztreamer and 107 ms on lightwalletd (22 times); the median 10-block range 37 µs and 232 µs.
• Concurrency: with 10,000-block requests ztreamer levels off at 1.06 to 1.13 M blocks/s from 16 clients; lightwalletd at 0.27 to 0.30 M. At 256 clients the p99 is 4.7 to 5.0 s against 29.6 s. With 100-block requests ztreamer serves 1.0 M blocks/s with a 61 ms p99 at 256 clients; lightwalletd 0.24 M blocks/s with an 814 ms p99.
• Unary RPCs (ghz): ztreamer answers GetLatestBlock and GetLightdInfo at 170,000 to 180,000 requests/s, lightwalletd at 27,000 to 43,000, because lightwalletd asks the node on each call. At 256 workers the zakura RPC server refuses part of those calls with HTTP 429, which lightwalletd passes on.
• GetTreeState at a random height is slow on every server, 170 to 260 ms median, because the tree state comes from zakura's z_gettreestate. Under load none of the servers exceeds about 80 successful requests/s.
• Efficiency: in the concurrency workloads ztreamer served 150,000 to 190,000 blocks per CPU second of server and node, lightwalletd 16,000 to 17,000.
• Memory: the anonymous memory of ztreamerd is 0.4 to 1.2 GiB under load (direct includes the embedded node). Its RSS reaches 17 to 20 GiB because the index is memory-mapped; those are file-backed pages of the page cache, which the kernel can reclaim. lightwalletd uses 0.5 GiB and its node 0.8 GiB.
• Disk: the ztreamer index is 18.7 GB / 17.4 GiB (direct) and 18.5 GB / 17.2 GiB (remote); the lightwalletd cache is 31.1 GB / 29.0 GiB. All three servers also need the zakura state (285 GB here).
• Restart on a finished index: lightwalletd 1.1 s, ztreamer remote 3.1 s, ztreamer direct 47 s (the embedded zakura opens its state first).
• Defect, MDB_READERS_FULL: once 256 ghz workers ran on 256 connections, ztreamerd answered part of the requests with Internal: MDB_READERS_FULL: Environment maxreaders limit reached, in both modes, and the errors went on in later scenarios with fewer workers. The LMDB environment opens with the default of 126 reader slots (crates/index/src/index/mod.rs, no max_readers), and with thread-local read transactions each blocking thread holds a slot; the tokio blocking pool can grow to 512 threads. The native suite, at 256 clients with one request each, saw no errors. Fix before the release: raise max_readers above the blocking-pool limit, or open the environment without TLS, or cap the blocking threads.
• Rare Unavailable: a few GetSubtreeRoots requests failed with serving snapshot generation N does not match LMDB generation N+1 while a new block was written.

Setup

Each server ran alone on the host, in this order: ztreamer direct, ztreamer remote, lightwalletd. Every run started from an empty index. The zakura state came from the same archive and was brought to the live tip before each run, so no server timed the catch-up of its node. All three servers then served the same fixed interval, with the same seeded request sequences.

• ztreamer direct: ztreamerd --zakura-config. The zakura node runs in the ztreamerd process and the index reads its state directly.
• ztreamer remote: ztreamerd --remote-rpc against a standalone zakurad. It imports the sealed ranges of the default snapshot source, https://snapshot.ztreamer.sh/mainnet, has the node attest them, and then reads the tail over JSON-RPC.
• lightwalletd: against the same standalone zakurad over JSON-RPC, with --no-tls-very-insecure and default limits. It builds its compact-block cache from genesis.

One run of each server. The node followed the live chain, so the tip moved a few blocks between runs; every historical request was in the fixed interval. The client shared the host with the server. lightwalletd built its cache from genesis, as does ztreamer. Remote mode depends on the download speed of the snapshot source; here the import moved 18.2 GB in 336 s, about 54 MB/s including verification. Throughput of ghz for streaming RPCs is limited by ghz itself (about 1,750 100-block ranges/s for both ztreamer modes); the native suite gives the server limits.

Sync speed

Time from the start of the server process until its index has the tip of the synced node and its gRPC listener accepts connections.

In direct mode ztreamerd reads blocks straight from the zakura state with eight workers, at 1.5 to 6 GB/s from disk. The slow band between heights 1.7 M and 2.1 M is the sandblasting period, whose blocks are large. In remote mode the import fetches the sealed ranges of the snapshot and checks them: the node attests the block hash at the end of each range and the tree roots of each epoch. lightwalletd asks the node for one block at a time over JSON-RPC, so its rate follows block size: about 1,500 blocks/s at the start, 65 blocks/s in the sandblasting period. After the first sync, ztreamerd in direct mode filled the range metadata of the 3,569 sealed ranges for its Replicator. That took 6 minutes 41 s after it began to serve, and raised its RSS by mapping the index. In remote mode the snapshot brought that metadata, so the pass found nothing to do.

Resource use

A sampler read /proc once a second for each process: CPU time, resident memory and disk bytes. For the remote and lightwalletd runs the totals include the standalone zakurad, which they need; in the direct run the node is inside ztreamerd.

Serving efficiency: compact blocks served per CPU second of the server and its node, in the workloads that serve blocks.

While syncing, ztreamer direct read 405 GB from disk in 146 s. In the first 44 s the embedded zakura opened its state on 6.5 cores on average and read 136 GB; the historical pass then read 269 GB of zakura state in 102 s (2.6 GB/s) on about 2 cores, so it was bound by the disk, not the CPU. Remote mode used about 24 cores while it verified the import. lightwalletd used under one core for itself and about 1.5 cores for its node for 1 hour 44 minutes. While serving, ztreamer used 4 to 6 cores under the heaviest workloads and lightwalletd 9 to 12, for a third to a quarter of the blocks. At idle all three use almost no CPU. The peak RSS of ztreamer while serving counts the mapped index; see the split below.

The restart runs restarted each server on the index of its main run, with the node already at the tip, then measured idle and loaded memory and repeated the wallet download. The wallet download after the restart took the same time as in the main run for every server.

Serving overview

Both ztreamer modes serve from the same kind of index, and their serving results agree within a few percent. lightwalletd serves blocks from its cache file and asks the node for everything else: chain info, tree states, transactions and address data.

RPC latencies

500 sequential requests per case from one client, after one warmup. Completion time includes draining the stream and decoding. Address cases use a busy and a quiet transparent address over the whole interval; GetTransaction uses two fixed transactions.

Range sizes

GetBlockRange over 1,000 seeded random ranges per size (100 at 100,000 blocks), one client, sequential.

Concurrent serving

Closed loop: each client keeps one GetBlockRange outstanding for 20 s per level, on its own connection. Two request sizes: 10,000 blocks (a wallet catching up) and 100 blocks (a wallet that is nearly synced).

10,000 blocks per request

100 blocks per request

Wallet download

One client downloads every compact block from Sapling activation to the end of the interval, 10,000 blocks per request. This is the network part of a new wallet's first sync; it excludes trial decryption, scanning and storage.

Throughput per RPC

ghz drives one RPC at a time with 1, 16, 64 and 256 concurrent workers, one connection each, for 15 s per level. Requests with heights cycle through 5,000 seeded random heights. The first request of each worker is not counted. Throughput counts successful requests only.

Raw data

All scenario summaries · Request samples (without the 2 M samples of the 100-block concurrency suite) · ghz results · Resource samples · Index sizes · Phases · All numbers. The harness is in benchmarks/2026-11-lwd-comparison. The previous page, ztreamer v0.1.0 against v0.0.1, is archived here.