I'm happy to share:
A native Go implementation of the Vortex columnar file format.
Vortex is a next-generation columnar file format (Linux Foundation project) designed for analytical workloads. It achieves significantly faster scans and better compression than Apache Parquet through adaptive encoding selection and lazy materialisation.
vortex-go provides full read and write support from pure Go — no CGo, no Rust FFI.
Key Features
- 25 encoding decoders — ALP, BitPacking, Dict, FoR, FSST, Pcodec, RLE, Zstd, and more
- Compressed writer — automatic encoding selection (dict, FoR+BitPacking, Pcodec, zone maps)
- Column projection — read only the columns you need, skip the rest
- Zone map pruning — skip entire chunks based on min/max filter predicates
- Full cross-compatibility — files readable by the Rust reference, Python vortex, and DuckDB
- CLI tools —
vtxinfofor file inspection,vtxconvertfor Parquet-to-Vortex conversion - Zero external dependencies at runtime — only Go standard library plus FlatBuffers/Protobuf
Performance
On a 7-column schema at 1M rows (Intel 14th Gen, Go 1.24):
| Metric | Parquet | Vortex (Go) | Improvement |
|---|---|---|---|
| File size | 45.6 MB | 19.3 MB | 58% smaller |
| Full scan | 188 ms | 40 ms | 5× faster |
| Projected scan (2/7 cols) | 188 ms | 16 ms | 12× faster |
| Throughput | 5.3M rows/s | 25M rows/s | 5× faster |
Zone map pruning on 100K rows (10 chunks, filter selects 10%): 8.5× faster than full scan.
The source code is available at: https://github.com/LaurieRhodes/vortex-go
- Log in to post comments