feat(server): load bases into DuckDB and serve list queries from cache

- collection-loader streams base rows via postgres and bulk-inserts into an
  in-memory DuckDB instance using the Appender API, then builds an index on
  each indexable column
- base-query-cache service routes list() calls through the prepared-statement
  path; ensureLoaded does schema-version checks with single-pass LRU eviction
- keyset param-ordering bug in the DuckDB builder fixed: placeholders appear
  head-to-tail but were being pushed tail-to-head, which made DuckDB bind the
  wrong value for each ? and throw Binder Error on typed columns
- base-row repo gains countActiveRows for the router to use in task 6
- seed script split into an importable helper so integration tests can seed a
  10k-row base deterministically without shelling out
- new integration spec compares Postgres vs DuckDB pagination end-to-end for
  a numeric sort and guards against duplicate rows from DuckDB

Integration test is skipped unless INTEGRATION_DB_URL is set.
This commit is contained in:
Philipinho
2026-04-19 21:31:05 +01:00
parent b28597125d
commit 91ad3de258
9 changed files with 1117 additions and 276 deletions
@@ -128,6 +128,21 @@ export class BaseRowRepo {
});
}
async countActiveRows(
baseId: string,
opts: WorkspaceOpts,
): Promise<number> {
const db = dbOrTx(this.db, opts.trx);
const row = await db
.selectFrom('baseRows')
.select((eb) => eb.fn.countAll<number>().as('count'))
.where('baseId', '=', baseId)
.where('workspaceId', '=', opts.workspaceId)
.where('deletedAt', 'is', null)
.executeTakeFirst();
return Number(row?.count ?? 0);
}
async getLastPosition(
baseId: string,
opts: WorkspaceOpts,