12.4 Ingestion Performance Tuning

Ingestion performance depends on the ingestion path, row size, batching, concurrency, indexes, and storage. Tune using representative data and measurements of end-to-end throughput and server processing response latency.

Choose an ingestion path

Use Data Input and Export to choose a path and check SDK and table-type support. This page focuses on throughput and latency for the selected path.

Measurement procedure

1. Prepare a representative schema, row size, and indexes.
2. Measure a baseline with one connection and small batches.
3. Increase batch size incrementally, recording rows/s and flush and server processing response latency.
4. Observe client CPU and memory alongside server CPU, I/O, and memory.
5. Increase connections and compare total throughput and p95/p99 response times. The p99 latency is the time within which 99% of requests complete; it reveals the impact of slow requests.
6. Test failures, reconnections, and duplicate handling.

Do not use a universal recommended batch size or thread count. Oversized batches increase memory use and error reprocessing scope; undersized batches increase network round-trip overhead.

Append operations

Follow the Append lifecycle, error handling, and duplicate handling contracts in Common Integration Concepts and the SDK Append Matrix. As you vary batch size and connections, record rows/s, p95/p99 latency, server processing responses, and failure counts together.

File loading

For file format validation, rejected-row files, exit codes, and final row verification, see Data Input and Export. Compare performance only after applying the same validation to the workload.

Classify bottlenecks

Before and after changes

Judge throughput improvements only after verifying successful and failed row counts. Also measure latency from ingestion start to server application, and check the impact on query performance and recovery time.