meta-corruption in attachable-subscription.test.ts

[mabels]

No description provided.

[jchris]

Meta-Corruption Investigation Summary

Root Cause Identified: CAR File Management Race Condition

Issue: During database attachment process, CAR files become "stale" and are removed from carLog before blocks can be properly validated, causing validateBlocks missing block errors.

Key Findings:

1. NOT a DbMeta corruption issue - DbMeta.cars correctly contains CAR file CIDs, not raw block CIDs
2. NOT a simple timing race - Adding sleep(1000ms) before validation doesn't reliably fix the issue
3. IS a CAR file lifecycle management problem during attachment

Evidence Pattern:

• Remote databases build up carLog with multiple CAR files during attachment
• load-car-stale car warnings indicate CAR files becoming unavailable
• validateBlocks missing block fails on legitimate CRDT head blocks
• CarLog progressively shrinks as "stale" CAR files are removed
• Final carLog often empty, explaining missing document count (2 vs 6 expected)

The Sequence:

1. Attachment begins: Remote DBs accumulate CAR files in carLog
2. CAR files written: Multiple baem... CIDs added to carLog
3. Race condition: CAR files marked as "stale" before blocks fully loaded
4. Stale cleanup: CAR files removed from carLog prematurely
5. Validation fails: Raw blocks from CRDT heads can't be found
6. Document loss: Missing blocks = missing documents in queries

Next Steps:

1. Investigate stale CAR file cleanup logic - Why are CAR files marked stale?
2. Fix CAR file availability timing - Ensure CAR files remain available during validation
3. Prevent premature carLog cleanup during attachment operations
4. Add proper async coordination between CAR file creation and block validation

Files Modified (with enhanced logging):

• /core/base/crdt-clock.ts - Added carLog logging before validation
• /core/base/crdt.ts - Added CRDT head tracking
• /core/blockstore/loader.ts - Added CAR extraction logging
• /core/tests/fireproof/attachable-subscription.test.ts - Added post-compact carLog logging

The meta-corruption issue is definitively a CAR file lifecycle management race condition during database attachment, not the originally suspected DbMeta structure corruption.

[jchris]

Deadlock Investigation - Attempted Synchronization Fixes

Attempted Fix #1: Promise.all() on Commit Queue

// In writeCar() - DEADLOCK 🔴
const remoteWrites = this.attached.remotes().map((r) => 
  this.commitQueue.enqueue(async () => {
    await r.active.car.save(block);
    return [];
  })
);
await Promise.all(remoteWrites); // ← DEADLOCK

Result: Test hung indefinitely, no logs produced after initial setup

Attempted Fix #2: commitQueue.waitIdle() in writeCar()

// In writeCar() - DEADLOCK 🔴
this.attached.remotes().forEach((r) => {
  this.commitQueue.enqueue(async () => {
    await r.active.car.save(block);
    return [];
  });
});
await this.commitQueue.waitIdle(); // ← DEADLOCK

Result: Test hung during execution, timeout after 10+ seconds

Attempted Fix #3: commitQueue.waitIdle() in writeMeta()

// In writeMeta() - DEADLOCK 🔴  
async writeMeta(cids: AnyLink[]): Promise<void> {
  const meta = { cars: cids };
  await this.commitQueue.waitIdle(); // ← DEADLOCK
  await this.attached.local().active.meta.save(meta);
}

Result: Test hung in beforeEach hook during database setup, indicating fundamental issue with queue synchronization

Root Cause of Deadlocks

The commit queue system is designed for eventual consistency, not strict synchronization:

1. Circular dependencies: Queue operations may depend on other queue operations
2. Never-idle queue: Background operations may keep queue permanently busy
3. Async-by-design: Forcing synchronous behavior breaks the queue's architecture

Confirmed Race Condition

Enhanced network logging definitively showed:

• ✅ CAR files written asynchronously to remote stores via commit queue
• ✅ DbMeta sent immediately after local writes complete
• ✅ Network requests fail when remote CAR files not yet written
• ✅ "missing car file" errors → CAR marked as "stale" → validation fails

Next Steps Required

The commit queue synchronization approach is not viable. Alternative solutions needed:

1. Different async coordination mechanism (not commit queue based)
2. Retry logic for stale CAR files instead of permanent marking
3. Synchronous remote writes during attachment (bypass queue entirely)
4. CAR availability verification before DbMeta transmission

The race condition is definitively identified but requires an architectural solution rather than commit queue synchronization.

[jchris]

How to Reproduce the Race Condition with Enhanced Logging

Running the Test with Full Logging

# From fireproof repo root:
cd /Users/jchris/code/fp/fireproof
FP_DEBUG=CRDTClock,Loader,EncryptedBlockstore pnpm run test --project memory attachable-subscription.test.ts -t 'should trigger subscriptions on inbound syncing' 2>&1 | tee meta-corruption-debug.log

What to Look For in the Logs

1. Network Request Failures 🔍

{"module":"Loader","cid":"baembeig...","msg":"NETWORK_REQUEST: About to load CAR from store"}
# Missing NETWORK_SUCCESS log = failed request
{"module":"Loader","cid":"baembeig...","msg":"load-car-stale car"}

2. CarLog State Before Validation 🔍

{"module":"CRDTClock","dbName":"remote-db-0-xxx","carLog":"baem1;baem2;baem3","msg":"PRE_VALIDATION: CarLog before block validation"}

3. Validation Failures 🔍

{"module":"CRDTClock","dbName":"remote-db-0-xxx","cid":"bafyr...","msg":"validateBlocks missing block"}

4. Empty CarLog After Test 🔍

POST_COMPACT: remote-db-0-xxx carLog: 
# ↑ Empty = all CAR files lost

Key Pattern to Confirm

1. Remote database accumulates CAR files in carLog during attachment
2. Some network requests succeed, others silently fail (no NETWORK_SUCCESS log)
3. Failed requests → load-car-stale car warnings → CAR files removed from carLog
4. Validation tries to access blocks from missing CAR files → validateBlocks missing block
5. Final carLog often empty → explains document count failure (2 instead of 6)

Current Branch

The enhanced logging is on branch mabels/meta-corruption - commit 9289c676

This logging definitively proves the write-before-notify race condition in the CAR file synchronization during database attachment.

[jchris]

Potential URL Routing Issue in CAR File Storage

Memory Gateway Logging Investigation

Running with FP_DEBUG=MemoryGatewayMeta,MemoryGatewayCar shows an interesting pattern that might explain the issue:

CAR Files Stored Successfully

CAR files appear to be written to main database storage:

{"module":"MemoryGatewayCar","url":"memory://db-z2HdkgQDgE?key=baembeif55jl...&store=car&suffix=.car","len":669,"msg":"put-car"}

CAR Files Retrieved Successfully by Main DB

Same files can be read using similar URL format:

{"module":"MemoryGatewayCar","url":"memory://db-z2HdkgQDgE?key=baembeif55jl...&store=car&suffix=.car","dataLen":669,"msg":"get-car"}

CAR Files Not Found by Remote DBs

Remote databases seem to be looking in different locations:

{"module":"Loader","url":"memory://car/remote-db-1-z2HdkgQDgE?name=remote-db-1-z2HdkgQDgE&store=car&storekey=%40db-z2HdkgQDgE-data%40&suffix=.car","error":"missing car file","msg":"load-car-stale car"}

Test Results

The assertion shows remote databases have completely empty carLogs:

MAIN_CIDS: [9 CAR CIDs]
REMOTE_CIDS: []

Possible URL Format Difference

It appears there may be a difference between:

• Storage URL format: memory://db-X?key=CARCID&store=car&suffix=.car
• Retrieval URL format: memory://car/remote-db-X?name=X&storekey=...&suffix=.car

This could explain why remote databases can't find CAR files that were successfully stored, leading to empty carLogs and validation failures.

Worth investigating if this is a URL generation issue in the attachment process or memory gateway routing.

[jchris]

@codex rebase to latest main

[chatgpt-codex-connector]

@codex rebase to latest main

For now, I can only help with PRs you've created.