Skip to content

feat: tiered data storage#7

Open
enigbe wants to merge 26 commits intomainfrom
2025-10-tiered-data-storage
Open

feat: tiered data storage#7
enigbe wants to merge 26 commits intomainfrom
2025-10-tiered-data-storage

Conversation

@enigbe
Copy link
Owner

@enigbe enigbe commented Oct 21, 2025 •
edited
Loading

What this PR does:

We introduce TierStore, a KVStore implementation that manages data across
three distinct storage layers.

The layers are:

  1. Primary: The main/remote data store.
  2. Ephemeral: A secondary store for non-critical, easily-rebuildable data
    (e.g., network graph). This tier aims to improve latency by leveraging a
    local KVStore designed for fast/local access.
  3. Backup: A tertiary store for disaster recovery. Backup operations are sent
    asynchronously/lazily to avoid blocking primary store operations.

We also permit the configuration of Node with these stores allowing
callers to set exponential back-off parameters, as well as backup and ephemeral
stores, and to build the Node with TierStore's primary store. These configuration
options also extend to our foreign interface, allowing bindings target to build the
Node with their own ffi::KVStore implementations.

A sample Python implementation is added and tested.

Additionally, we add comprehensive testing for TierStore by introducing

  1. Unit tests for TierStore core functionality.
  2. Integration tests for Node built with tiered storage.
  3. Python FFI tests for foreign ffi::KVStore implementations.

Concerns

It is worth considering the way retry logic is handled, especially because of nested
retries. TierStore comes with a basic one by default but there are KVStore implementations
that come with them baked-in (e.g. VssStore), and thus would have no need for
the wrapper-store's own logic.

@enigbe enigbe force-pushed the 2025-10-tiered-data-storage branch 9 times, most recently from 29f47f3 to 264aa7f Compare November 4, 2025 22:07
@enigbe enigbe force-pushed the 2025-10-tiered-data-storage branch 3 times, most recently from a30cbfb to 1e7bdbc Compare December 4, 2025 23:30
@enigbe enigbe force-pushed the 2025-10-tiered-data-storage branch 5 times, most recently from b5e980f to 67d47c2 Compare February 4, 2026 16:28
@enigbe enigbe force-pushed the 2025-10-tiered-data-storage branch 2 times, most recently from 95285b0 to 4b2d345 Compare February 18, 2026 11:23
@enigbe enigbe force-pushed the 2025-10-tiered-data-storage branch 2 times, most recently from cba29a3 to db1fe83 Compare February 24, 2026 23:03
benthecarman and others added 9 commits February 24, 2026 18:52
@benthecarman
prefactor: Create helper function for calculating max balance we can …
615814e 
…send

DRY up the repeated pattern of building a temporary drain transaction
to estimate fees. Both send_all_to_address (AllRetainingReserve) and
the upcoming open_channel_with_all / splice_in_with_all need this
logic, so extract it into shared helpers.
@benthecarman
Add ability to open channel with all on-chain funds
c4a45a6 
Adds open_channel_with_all which uses get_max_drain_amount to
determine the largest funding amount after accounting for on-chain fees
and anchor reserves
@benthecarman
Add ability to splice in with all on-chain funds
4a2cefc 
Adds splice_in_with_all which uses get_max_drain_amount with a
shared input to determine the largest splice-in amount after accounting
for on-chain fees and anchor reserves.
@benthecarman
Extract new_channel_anchor_reserve_sats into free function
56730e9 
DRY up the anchor reserve calculation that was duplicated between
Node::new_channel_anchor_reserve_sats and the OpenChannelRequest
handler in event.rs.
@tnull
Merge pull request lightningdevkit#788 from benthecarman/open-channel…
1cf1e6f 
…-all

Add ability to open channel / splice with all on-chain funds
@Camillarhi
Minor adjustment to fee rate handling in RBF fee bumping
6c0e4d4 
Use user-provided fee rate for bumping transaction and return an error if
it is too low, otherwise use system estimated fee rate and retry mechanism.

Also switches the RBF test to use `random_chain_source`, and removes unnecessary
sleep-based waits.
@tnull
Merge pull request lightningdevkit#802 from Camillarhi/feat/tx-rebroa…
a8b956a 
…dcast-fee-bumping-follow-up

Minor follow-ups to lightningdevkit#628
@TheBlueMatt
Add support for the simple "sigs-based auth" VSS scheme
927ce68 
At lightningdevkit/vss-server#79 we added
a new, trivial, VSS authentication scheme that ensures client
isolation without much else. This is great for testing, and we
expect some to do new-account-rate-limiting via other means, so
might well become a common default.

Here we add support to it in ldk-node.
@heyolaniran
Refactor : Python binding test
4477ca7 
Added an expected_event function to better handle various events in the same block of code instead of duplicating the code for event listening. This improves the code readability and clarity.
TheBlueMatt and others added 17 commits March 3, 2026 17:53
@TheBlueMatt
Switch to new sigs-based auth in VSS integration tests
e4da11e 
When we added the trivial sigs-based authentication scheme in VSS,
we made it the default if no other authentication scheme was
configured and default features are enabled.

This broke our integration tests as we were expecting no
authentication to be required in such a case.

Here we fix this by switching to the new sigs-based auth scheme,
removing `store_id`s to demonstrate client isolation while we're at
it.

Sadly, because we don't currently have a test framework for
LNURL-auth-based VSS, and because VSS no longer defaults to
no-auth, the upgrade-from-0.6 test has to be moved to a separate
CI job which runs VSS server with the noop auth.
@TheBlueMatt
Document when someone should want to use store_id
887ed87
@TheBlueMatt
Use non-empty store_ids in VSS integration tests
01a232e 
For now VSS Server rejects empty `store_id`s, though we intend to
change that in
lightningdevkit/vss-server#95, until that
lands we have to use non-empty `store_id`s.
@vincenzopalazzo @claude
event: expose PaidBolt12Invoice in PaymentSuccessful for proof of pay…
3c724a9 
…ment

Add the `bolt12_invoice: Option<PaidBolt12Invoice>` field to the
`PaymentSuccessful` event, enabling users to obtain proof of payment
for BOLT12 transactions.

For non-uniffi builds, import `PaidBolt12Invoice` directly from
`lightning::events` rather than re-exporting through `types.rs`.
For uniffi builds, use the custom `ffi::PaidBolt12Invoice` wrapper
that handles Arc-wrapping for enum variants.

The proof-of-payment assertion is integrated into the existing
`simple_bolt12_send_receive` test rather than a separate test.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@tnull
Merge pull request lightningdevkit#755 from TheBlueMatt/2026-01-vss-s…
e3d3318 
…igs-auth

Add support for the simple "sigs-based auth" VSS scheme
@tnull
Merge pull request lightningdevkit#805 from heyolaniran/refactoring_b…
cb99e84 
…inding_python_test

Refactoring binding python test
@tnull
Merge pull request lightningdevkit#733 from vincenzopalazzo/macros/bo…
fae2746 
…lt12-pop

[RFC] event: expose BOLT12 invoice in PaymentSuccessful for proof of payment
@enigbe
Implement tiered storage
48af9d3 
Introduces TierStore, a KVStore implementation that manages data
across three storage layers:

- Primary: Main data store for critical node data
- Ephemeral: Secondary store for non-critical, easily-rebuildable
data (e.g., network graph) with fast local access
- Backup: Tertiary store for disaster recovery with async/lazy
operations to avoid blocking primary store
- Unit tests for TierStore core functionality
@enigbe
Integrate TierStore into NodeBuilder
b3eee95 
Adds TierStoreConfig and two configuration methods to NodeBuilder:

- set_backup_store: Configure backup store for disaster recovery
- set_ephemeral_store: Configure ephemeral store for non-critical data

Modifies build_with_store to wrap the provided store in a TierStore,
as the primary store, applying any configured ephemeral and backup stores.

Note: Temporal dead code allowance will be removed in test commit.
@enigbe
Expose tier storage configuration across the FFI boundary
b1a0871 
Introduce FFI-safe abstractions and builder APIs to allow foreign
language targets to configure custom backup and ephemeral stores when
constructing nodes with a custom store.

Major changes include:

- Addition of FfiDynStoreTrait, an FFI-safe equivalent of DynStoreTrait,
  working around uniffi's lack of support for Pin<Box<T>>
- Addition of FfiDynStore, a concrete wrapper for foreign language store
  implementations
- Provision of FfiDynStoreTrait implementation for DynStoreWrapper to bridge
  native Rust stores to FFI layer (useful in testing)
- Extension of ArcedNodeBuilder with methods for configuring backup and
  ephemeral stores
- Exposure of build_with_store so foreign targets can build nodes with
  custom store implementations
- Addition of build_node_with_store test helper to abstract uniffi-gated
  store wrapping at build_with_store call sites
@enigbe
Add integration and FFI tests for tiered storage
4f374e5 
- Add Rust integration test verifying correct routing to storage tiers
- Add Python in-memory KV store and FFI test for tiered storage
@enigbe
Add per-key version-locked writes to FfiDynStore
2a8d9b9 
Introduce FfiDynStoreInner with per-key write version locks that
ensure write ordering and skip stale versions in both sync and async
code paths.

Test changes:
- Unify tier store test helpers to use TestSyncStore for all tiers,
  replacing mixed SqliteStore/FilesystemStore/TestStore usage that
  caused test hangs due to TestStore's async write blocking
- Split build_node_with_store into cfg-gated versions for uniffi
  vs non-uniffi feature flags
@enigbe
Add documentation for ForeignDynstoreTrait
8763af0
@enigbe
Remove unnecessary uniffi gating in tier store
23815f3
@enigbe
Drop DelayedStore and associated backup test
bd4e79d 
These were created to test that our backup store does
not impact the primary store writes but the boilerplate
appears too much for the functionality being tested.
@enigbe
Remove spammy logs
6fce438
@enigbe
DRY ephemeral key matching, fix visibility and appropriate kvstore usage
35f9ec2 
- Restrict `TierStoreInner` visibility from `pub` to `pub(crate)`
- Primary store can be either local or remote
- Extract repeated ephemeral key matching into a standalone
  `is_ephemerally_cached_key` helper to DRY up `read_internal`,
  `write_internal`, and `remove_internal`
- Replace `KVStoreSync::list` with async `KVStore::list` in
  `list_internal` to avoid blocking the async runtime
@enigbe enigbe force-pushed the 2025-10-tiered-data-storage branch from db1fe83 to 35f9ec2 Compare March 9, 2026 15:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

7 participants

@enigbe @benthecarman @tnull @Camillarhi @TheBlueMatt @heyolaniran @vincenzopalazzo