IEX-DOWNLOAD: Rust, Tick Data, and 13TB of Fun
Not long ago Brandon, a reputable headhunter from UK, asked me point-blank: “Do you know Rust?” Well, since 1993 I’ve written code in BASIC, CLIPPER, PASCAL, SQL, C, PostScript (yes, the stack-oriented beast), TEX, Intel and MIPS assembly, PHP, Java, Matlab, R, Julia, Maple, Maxima, Fortran, C#, Go, Ruby, grammar parsers like LEX and YACC, then of course JavaScript and Python — and my personal favourite: C++ metaprogramming. In other words: if it’s Turing-complete and vaguely hostile to human beings, chances are I’ve built something in it. So no, it’s not the language that matters, it’s the craft. But since Rust is today’s poster child for systems programming, I wrote IEX-DOWNLOAD entirely in Rust. My way of saying: sure, I can do that too, and I’ll even enjoy the borrow checker while I’m at it.
Key Differences (TOPS vs DEEP vs DEEP+)
| Feature | TOPS (Top-of-book) | DEEP (Aggregated) | DEEP+ (Order-by-order) |
|---|---|---|---|
| Order granularity | Only best bid/ask + last trade | Aggregated by price level (size summed) | Individual orders (each displayed order) |
| OrderID / update per order | Not present | Not present | Present |
| Hidden / non-display / reserve portions | Not shown | Not shown | Not shown |
| Memory / bandwidth load | Lowest (very compact, minimal updates) | Lower (fewer messages, coarser updates) | Higher (tracking many individual orders, cancels, modifications) |
| Use-cases | Quote feeds, NBBO tracking, top-level liquidity, lightweight apps | General depth, price level elasticity, coarse modelling, liquidity at price tiers | Detailed book shape, order flow-level strategy, detailed execution modelling, microstructure research |
Dataset as of 2025 September
| Feed | Files to Download | Total Size (≈ GB) |
|---|---|---|
| TOPS | 2,285 | 5,947.68 |
| DEEP | 2,115 | 5,955.02 |
| DEEP+ | 197 | 1,353.52 |
| TOTAL | 4,597 | 13,256.22 |
Why Bother?
Because it lets you grab over 13 TB of IEX tick data in one shot. Wait, wasn’t it 6 TB last week? Exactly. Trading data is like an iceberg: TOPS shows you the shiny tip (best bid/ask and last trade), while the real bulk is hidden underneath in DEEP and DEEP+. That’s where the weight lives — and where the fun begins.
Features at a Glance
- Progress bar with attitude → because watching terabytes flow should feel satisfying.
- PEG-based date parser → type
2025-01-??or202{4,5}-{04,05}-??and it just works, no regex headaches. - One tiny ELF → a single 3.5 MB executable (
-rwxrwxr-x 2 steven steven 3.5M Sep 23 11:00 target/release/iex-download).
No Python venvs, no dependency jungles. Drop it anywhere,chmod +x, and let it rip.
Framing It All
So, how do you use IEX-DOWNLOAD in practice?
- TOPS → quick & light (quotes + trades).
- DEEP → richer, aggregated depth (enough for most analytics).
- DEEP+ (DPLS) → the full microstructure playground (every displayed order).
And that’s the trick: IEX-DOWNLOAD gets you the firehose, and IEX2H5 turns that torrent into tidy, high-performance HDF5 files.
So there you have it: IEX-DOWNLOAD, 4,597 files and 13.26 TB of market microstructure, distilled into a 3.5 MB Rust binary. And to Brandon — thank you for asking the right question. Sometimes all it takes is a good headhunter’s nudge to turn an idea into a tool. Rust may be the new kid, but in the right hands it can move mountains… or at least download a few terabytes before lunch.