Feature · ingestion
Streaming Ingestion
Multi-GB FDR files, bounded memory, no batch jobs left for tomorrow
At a glance
- ▸ Memory-bounded streaming with configurable batch sizes
- ▸ Direct PostgreSQL COPY via libpqxx for high throughput (optional)
- ▸ HTTP/PostgREST fallback with exponential-backoff retries
- ▸ Progress events fire per batch (for UI / monitoring)
- ▸ Resumable — a failed batch doesn't lose the whole upload
What it does
Loading multi-GB FDR files into a database the naive way blows up memory or chokes on transaction size. Aviation AI Pro processes the file as a stream: parse N frames, upload them, free the memory, parse the next N. Memory ceiling holds steady whether the file is 5 MB or 5 GB.
Two ingestion paths: HTTP/PostgREST (works from anywhere) and libpqxx direct connection (way faster, requires Postgres reachability). Both retry with exponential backoff so transient network issues don’t lose data.