feat: sqlite utils and unit tests

[hanabi1224]

Summary of changes

(Part of #6249)

Changes introduced in this pull request:

• Ports Sqlite module from Lotus: https://github.com/filecoin-project/lotus/blob/v1.34.3/lib/sqlite/sqlite.go

Reference issue to close (if applicable)

Closes

Other information and links

Change checklist

• I have performed a self-review of my own code,
• I have made corresponding changes to the documentation. All new code adheres to the team's documentation standards,
• I have added tests that prove my fix is effective or that my feature works (if possible),
• I have made sure the CHANGELOG is up-to-date. All user-facing changes should be reflected in this document.

Summary by CodeRabbit

• Chores

  • Added optional SQLite support, enabling both file-backed and in-memory databases.
  • Added database initialization, migration handling, and maintenance routines to ensure schema upgrades and post-migration cleanup.

• Tests

  • Added comprehensive integration tests covering schema creation, migrations, data integrity, reopen behavior, and post-migration validations.

_{✏️ Tip: You can customize this high-level summary in your review settings.}

[coderabbitai]

Walkthrough

Adds optional sqlx-backed SQLite support: new Cargo feature and dependency, a conditional utils::sqlite module with helpers to open/configure SqlitePool and run versioned migrations, and comprehensive integration tests validating initialization and migrations.

Changes

Cohort / File(s)	Summary
Dependency / Features Cargo.toml	Added optional sqlx = { version = "0.8", default-features = false, features = ["sqlite","runtime-tokio","macros"] }; added "sqlite" to default features and feature entry "sqlite" = ["dep:sqlx"]; minor reflow of private-feature comments.
Module export src/utils/mod.rs	Exported new conditional public submodule behind feature: pub mod sqlite; (gated by #[cfg(feature = "sqlite")]).
SQLite implementation src/utils/sqlite/mod.rs	New module providing pub type SqliteQuery<'q> = Query<'q, sqlx::Sqlite, SqliteArguments<'q>>; helpers open_file(), open_memory(), open() to create/configure SqlitePool (pragmas, mmap, journaling); init_db() to create/check _meta, apply initial DDLs, run per-version migrations in transactions, update version, and perform VACUUM/WAL checkpointing when needed.
Integration tests src/utils/sqlite/tests.rs	New tests: DDLs for tables/indexes, initial inserts, DB reopen/validation, migration adding a column and more inserts, dump_tables helper and internal types TableData, TableDataRow, Value for state assertions.

Sequence Diagram(s)

sequenceDiagram
    participant App as Application
    participant Init as init_db()
    participant DB as SQLite DB
    participant Meta as _meta Table
    participant Mig as Migration

    App->>Init: call init_db(name, ddls, migrations)
    Init->>DB: query sqlite_master for `_meta`
    DB-->>Init: `_meta` exists? / not exists

    alt `_meta` missing
        Init->>DB: CREATE TABLE _meta(...)
        Init->>DB: execute initial DDLs
        Init->>Meta: INSERT version = 0
    end

    Init->>Meta: SELECT version FROM _meta
    Meta-->>Init: current_version

    loop for each migration where version > current_version
        Init->>DB: BEGIN TRANSACTION
        Init->>Mig: execute migration query
        Mig->>DB: apply changes
        Init->>Meta: UPDATE version
        Init->>DB: COMMIT
    end

    alt migrations applied and growth suspected
        Init->>DB: PRAGMA wal_checkpoint(TRUNCATE)
        Init->>DB: VACUUM
    end

    Init-->>App: return Result

Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

Suggested reviewers

• akaladarshi

🚥 Pre-merge checks | ✅ 3

✅ Passed checks (3 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title 'feat: sqlite utils and unit tests' accurately and concisely describes the main changes: adding SQLite utility functions and their corresponding unit tests.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing touches

• 📝 Generate docstrings

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 1

🤖 Fix all issues with AI agents

In `@src/utils/sqlite/tests.rs`:
- Around line 215-255: Replace the brittle SQL string parsing inside
TableData::try_from (taking TableDataRow and db: &SqlitePool) with a PRAGMA
table_info query to fetch column names reliably: run "PRAGMA table_info({name})"
via sqlx to collect the "name" field for each column into cols, and change the
row-fetching query to sqlx::query(&format!("SELECT * FROM {name} ORDER BY
rowid")) to ensure deterministic ordering when building data; keep existing
error handling and value extraction logic (the r.try_get paths) otherwise.

🧹 Nitpick comments (2)

src/utils/sqlite/mod.rs (2)

13-58: Avoid blocking FS ops in async open_file and add context.

Line 31 uses std::fs::create_dir_all inside an async function. Consider tokio::fs (or spawn_blocking) and add context to failures so diagnostics are clearer. As per coding guidelines, add context to fallible operations.

♻️ Suggested adjustment

 use anyhow::Context as _;
+use tokio::fs;
 ...
 pub async fn open_file(file: &Path) -> anyhow::Result<SqlitePool> {
     if let Some(dir) = file.parent()
         && !dir.is_dir()
     {
-        std::fs::create_dir_all(dir)?;
+        fs::create_dir_all(dir)
+            .await
+            .with_context(|| format!("creating sqlite directory {dir:?}"))?;
     }
     let options = SqliteConnectOptions::new().filename(file);
-    Ok(open(options).await?)
+    Ok(open(options)
+        .await
+        .with_context(|| format!("opening sqlite database at {file:?}"))?)
 }

---

61-105: Clarify the DDL vs migration contract for fresh DBs.

When _meta is missing, the code inserts all versions and skips migrations, so fresh DBs rely on ddls already representing the latest schema. Please document or assert this contract to avoid marking a version that hasn’t actually been applied.

📝 Doc clarification

 /// This function initializes the database by checking whether it needs to be created or upgraded.
 /// The `ddls` are the `DDL`(Data Definition Language) statements to create the tables in the database and their initial required
+/// content. NOTE: `ddls` must represent the latest schema; migrations are only for upgrading existing databases.
 /// content. The `schema_version` will be set inside the database if it is newly created. Otherwise, the

[codecov]

Codecov Report

❌ Patch coverage is 65.97938% with 33 lines in your changes missing coverage. Please review.
✅ Project coverage is 58.73%. Comparing base (db580cf) to head (c8bb624).
⚠️ Report is 5 commits behind head on main.

Files with missing lines	Patch %	Lines
src/utils/sqlite/mod.rs	65.97%	16 Missing and 17 partials ⚠️

Additional details and impacted files

Files with missing lines	Coverage Δ
src/utils/mod.rs	80.53% <ø> (ø)
src/utils/sqlite/mod.rs	65.97% <65.97%> (ø)

... and 4 files with indirect coverage changes

---

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update db580cf...c8bb624. Read the comment docs.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[LesnyRumcajs]

given we don't use it anywhere yet, let's hide it behind a feature flag

[hanabi1224]

It's not meant to be an optional feature and will be integrated in a subsequent PR.

[LesnyRumcajs]

Yeah, but until integrated, it's wasteful.

[hanabi1224]

Putting it behind a feature flag would make CI ignore the unit test unless we change the script but we will need to change everything back right after this is merged : )

[hanabi1224]

Or I can add the feature to the default list so it's excluded by mise run i --slim quick

[LesnyRumcajs]

LGTM, but probably best to keep it behind a feature flag until we actually use it.