Skip to content

Conversation

@vcarl
Copy link
Member

@vcarl vcarl commented Jan 26, 2026

We've had 2 instances of db corruption lately, which appears to be because of the Effect/non-Effect db access code. They're both connecting to the same file, but have been using different Journal modes. In the right circumstances, that will cause the file to become corrupt. AI fix explanation:

When Connection A (rollback) and Connection B (WAL) both access the same database:

  1. Connection B writes in WAL mode - appends to -wal file, updates -shm coordination
  2. Connection A doesn't know about WAL - it's looking for a -journal file and using file locks designed for rollback mode
  3. Connection A writes in rollback mode - modifies database pages directly, creates a rollback journal
  4. Conflict occurs:
    - Connection A may overwrite pages that Connection B expects to be at a certain state
    - Connection B's WAL entries reference page states that Connection A has modified
    - The -shm shared memory state becomes inconsistent with actual file state
    - B-tree page references become invalid ("2nd reference to page" errors we saw)

When Connection A (rollback) and Connection B (WAL) both access the same database:

  1. Connection B writes in WAL mode - appends to -wal file, updates -shm coordination
  2. Connection A doesn't know about WAL - it's looking for a -journal file and using file locks designed for rollback mode
  3. Connection A writes in rollback mode - modifies database pages directly, creates a rollback journal
  4. Conflict occurs:
    - Connection A may overwrite pages that Connection B expects to be at a certain state
    - Connection B's WAL entries reference page states that Connection A has modified
    - The -shm shared memory state becomes inconsistent with actual file state
    - B-tree page references become invalid ("2nd reference to page" errors we saw)
@what-the-diff
Copy link

what-the-diff bot commented Jan 26, 2026

PR Summary

  • Introduction of a centralized SQLite database connection
    Instead of creating new connections every time we need to access the database, the team has created a new variable (sqliteDb) that gets our program connected to the SQLite database just once. This makes our database operations more efficient.

  • Addition of Write-Ahead Logging (WAL) Mode for better database integrity
    The team has done a great job of protecting the database by employing the WAL mode. This mode logs changes even before they're written into the database - a huge plus for preventing any data loss during unexpected terminations.

  • Refactor of dialect export for efficient database operation
    Rather than initiating a new SQLite database connection each time, the team has now made it so our dialect now utilizes the previously mentioned sqliteDb. This means less time creating connections and more time doing the work it's supposed to.

@vcarl vcarl merged commit b22858c into main Jan 26, 2026
5 checks passed
@vcarl vcarl deleted the vc-hotfix branch January 26, 2026 16:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants