benchmarks-website: duckdb conn pool#7850
Conversation
Signed-off-by: Andrew Duffy <andrew@a10y.dev>
| pub db: DbHandle, | ||
| /// Serializes `/api/ingest` so concurrent ingests can't race on the same | ||
| /// rows and trigger a DuckDB write-write conflict. Reads are unaffected. | ||
| pub ingest_lock: Arc<Mutex<()>>, |
There was a problem hiding this comment.
really calls out to a RwLock around the handle or something? put the valid states into the type system
There was a problem hiding this comment.
i think we actually want a separate Connection for the ingest endpoint. it's ok for there to be multiple readonly conns in the process but there can only be one writable. so actually i should make this an Arc<Mutex<Connection>> and leave the pool for the other one in place.
Merging this PR will improve performance by 13.02%
|
| Mode | Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|---|
| ⚡ | Simulation | take_map[(0.1, 0.5)] |
1,111.5 µs | 983.5 µs | +13.02% |
| ⚡ | Simulation | take_map[(0.1, 1.0)] |
1.8 ms | 1.7 ms | +11.01% |
Comparing aduffy/duckdb-pool (f362091) with develop (ff12040)
Unable to generate the flame graphsThe performance report has correctly been generated, but there was an internal error while generating the flame graphs for this run. We're working on fixing the issue. Feel free to contact us on Discord or at support@codspeed.io if the issue persists. |
|
Closing in favor of just doing this in the other benchmarks PR |
A single visit to the benchmarks website makes ~360 concurrent API requests to this axum server.
The previous
Arc<Mutex<Connection>>was causing all of these requests to serialize (multiply by however many concurrent tabs had the benchmarks open).This creates a connection pool of DuckDB conns, and adds a mutex on the ingestion endpoint to avoid attempting to open two concurrent DB connections for writing (which DuckDB does not allow and would cause an error).