Skip to content

epic: SQE backend — serve SQLite databases via SQE alongside other backends #629

Description

@iheitlager

Use Case

sqlite-rs as a database backend that SQE serves — SQLite files as data sources, combinable with other backends (Postgres, DuckDB, etc.).

Context

SQE federates multiple database backends. Adding SQLite as a first-class backend means:

  • Query SQLite files alongside other data sources
  • Use SQLite for local/edge data that joins with remote sources
  • No C dependency for SQLite backend support

Requirements

Read path

  • Table discovery from SQLite schema
  • Type mapping (SQLite affinity → Arrow)
  • Pushdown (filters, projections, limits)
  • Scan to Arrow batches

Write path (optional)

  • INSERT/UPDATE/DELETE via SQE
  • Transaction semantics

Federation

  • Cross-backend JOINs (SQLite ⋈ Postgres)
  • Unified catalog view

Dependencies

Path

  1. Read-only backend with table discovery + scans
  2. Pushdown optimization
  3. Write support
  4. Federation testing

Success

  • sqe query "SELECT * FROM sqlite.my_table JOIN pg.other ..."
  • Performance within 2× of direct SQLite access
  • Works with existing SQE CLI and connectors

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    epicTracking issue spanning multiple tickets

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions