VargaLABS https://steven-varga.ca/blog/ Advanced Systems for Trading, Cryptography, and Data Engineering en-us Tue, 07 Oct 2025 21:22:37 GMT IEX-DOWNLOAD: Rust, Tick Data, and 13TB of Fun https://steven-varga.ca/blog/iex-download-rust/ https://steven-varga.ca/blog/iex-download-rust/ Thu, 25 Sep 2025 00:00:00 GMT Meet IEX-DOWNLOAD: a Rust utility that slurps down terabytes of IEX tick data A Week of Market History Vanished — Here’s What Really Happened https://steven-varga.ca/blog/iex-timestamp-glitch-wireshark/ https://steven-varga.ca/blog/iex-timestamp-glitch-wireshark/ Fri, 12 Sep 2025 00:00:00 GMT A week of IEX ticks mysteriously collapsed into December 31, 1969 — I Analyzed 6TB of Raw Stock Market Data to Uncover the 30 Most Consistently Traded Stocks — Here’s What I Found https://steven-varga.ca/blog/longest-active-stocks-from-iex-pcap/ https://steven-varga.ca/blog/longest-active-stocks-from-iex-pcap/ Fri, 05 Sep 2025 00:00:00 GMT What happens when you throw 6 terabytes of raw stock market tick data into an HDF5-powered firehose and look for the 30 most consistently active stocks? You find out who *really* moves the markets. Here’s the journey. Radix64 Encoding: Order-Preserving Strings in 64 Bits https://steven-varga.ca/blog/radix64-order-preserving-encoding/ https://steven-varga.ca/blog/radix64-order-preserving-encoding/ Sat, 23 Aug 2025 00:00:00 GMT This post introduces an order-preserving, fixed-width radix-64 encoding scheme. From Curve to Signature: A Hands-on Guide to ECDSA https://steven-varga.ca/blog/from-curve-to-signature-ecdsa-guide/ https://steven-varga.ca/blog/from-curve-to-signature-ecdsa-guide/ Mon, 12 May 2025 00:00:00 GMT A hands-on guide for generating elliptic curves and producing Ethereum-style Why Your HDF5 Dataset Can't Extend—and the H5CPP Way https://steven-varga.ca/blog/h5cpp-dataset-cant-extend/ https://steven-varga.ca/blog/h5cpp-dataset-cant-extend/ Wed, 17 May 2023 00:00:00 GMT If your HDF5 dataset refuses to extend, it's almost certainly because it wasn’t created chunked with unlimited dimensions. Here’s how to fix that—and how H5CPP wraps extendible datasets and appending into a clean, modern C++ API. Optimizing Reads of Very-Wide Compound HDF5 Types https://steven-varga.ca/blog/hdf5-wide-compound-optimization/ https://steven-varga.ca/blog/hdf5-wide-compound-optimization/ Mon, 17 Apr 2023 00:00:00 GMT Reading wide compound records with hundreds of fields can bottleneck performance. HDF5 Write Speeds: Matching Underlying Raw I/O https://steven-varga.ca/blog/hdf5-write-performance-vs-raw/ https://steven-varga.ca/blog/hdf5-write-performance-vs-raw/ Tue, 28 Mar 2023 00:00:00 GMT HDF5 writes large blocks at roughly 70–95% of underlying raw I/O—fast enough for most needs. Why Writing HDF5 Chunks Piece-by-Piece Actually Fails https://steven-varga.ca/blog/h5cpp-chunk-atomicity/ https://steven-varga.ca/blog/h5cpp-chunk-atomicity/ Fri, 24 Mar 2023 00:00:00 GMT Trying to write HDF5 chunks in fragments? Not going to work. HDF5 treats chunks as atomic units. Writing Direct HDF5 Chunks Piece-by-Piece with H5CPP https://steven-varga.ca/blog/h5cpp-direct-chunk-append/ https://steven-varga.ca/blog/h5cpp-direct-chunk-append/ Mon, 20 Mar 2023 00:00:00 GMT Need to emit partial chunks manually—perhaps from multiple queues or threads? Extendable Datasets in HDF5—Simplified with H5CPP https://steven-varga.ca/blog/h5cpp-extendable-datasets/ https://steven-varga.ca/blog/h5cpp-extendable-datasets/ Thu, 02 Mar 2023 00:00:00 GMT Building extendable datasets in HDF5 doesn’t need boilerplate. H5CPP gives you a compact, Cross-language Messaging with ZeroMQ: C++, Python, and Fortran https://steven-varga.ca/blog/zmq-multi-lang-push-pull/ https://steven-varga.ca/blog/zmq-multi-lang-push-pull/ Wed, 29 Jun 2022 00:00:00 GMT ZeroMQ is a powerful messaging library that makes inter-process communication easy. HDF5 Group Overhead: Measuring the Cost of a Million Groups https://steven-varga.ca/blog/hdf5-group-overhead/ https://steven-varga.ca/blog/hdf5-group-overhead/ Thu, 07 Apr 2022 00:00:00 GMT HDF5 group objects come with significant metadata overhead. Automated Pretty Printing for STL-like Containers in H5CPP https://steven-varga.ca/blog/automated-stl-printing/ https://steven-varga.ca/blog/automated-stl-printing/ Thu, 07 Apr 2022 00:00:00 GMT Debugging with `gdb`, `valgrind`, or custom logging setups can be painful. Custom Floating-Point and Opaque Types in HDF5 https://steven-varga.ca/blog/hdf5-custom-long-double/ https://steven-varga.ca/blog/hdf5-custom-long-double/ Thu, 17 Mar 2022 00:00:00 GMT HDF5 can represent `long double` via native or custom datatypes. HDF5 and `long double`: Precision Stored, Precision Misread https://steven-varga.ca/blog/hdf5-long-double-support/ https://steven-varga.ca/blog/hdf5-long-double-support/ Wed, 16 Mar 2022 00:00:00 GMT HDF5 fully supports `long double` writes and reads, but `h5dump` misreports values. Rewriting Attributes in HDF5 with H5CPP https://steven-varga.ca/blog/hdf5-rewrite-attributes/ https://steven-varga.ca/blog/hdf5-rewrite-attributes/ Tue, 15 Mar 2022 00:00:00 GMT While HDF5 doesn’t support extending attributes directly, you can emulate it by reading, rewriting, and replacing. Single-Thread Writer: Simplifying Parallel HDF5 I/O https://steven-varga.ca/blog/hdf5-single-thread-writer/ https://steven-varga.ca/blog/hdf5-single-thread-writer/ Mon, 14 Mar 2022 00:00:00 GMT Instead of letting many threads contend with HDF5’s global lock, Fixed-Length vs. Variable-Length Storage in HDF5 https://steven-varga.ca/blog/hdf5-fixed-vs-variable-benchmark/ https://steven-varga.ca/blog/hdf5-fixed-vs-variable-benchmark/ Thu, 29 Apr 2021 00:00:00 GMT Fixed-length storage wins for speed and predictability. Bridging HPC Structs and HDF5 COMPOUNDs with H5CPP https://steven-varga.ca/blog/pod-to-hdf5-compound/ https://steven-varga.ca/blog/pod-to-hdf5-compound/ Sun, 18 Apr 2021 00:00:00 GMT Define your data once as a C++ POD struct.