-
Notifications
You must be signed in to change notification settings - Fork 125
Enhance onchain transaction management #628
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Enhance onchain transaction management #628
Conversation
|
👋 Thanks for assigning @tnull as a reviewer! |
39922cc to
c228760
Compare
|
🔔 1st Reminder Hey @tnull! This PR has been waiting for your review. |
|
🔔 2nd Reminder Hey @tnull! This PR has been waiting for your review. |
|
🔔 3rd Reminder Hey @tnull! This PR has been waiting for your review. |
|
🔔 4th Reminder Hey @tnull! This PR has been waiting for your review. |
tnull
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Just took an initial look and added a few high-level comments. I have yet to review any of the actual RBF logic changes.
However, as noted elsewhere, please don't make all the changes as single huge commit, but rather break the PR up in logical commits that all have descriptive commit messages outlining what the change is, why it's necessary, etc. For guidance you can have a look at https://cbea.ms/git-commit/
It would also make sense to not include the changes for #452 in this initial PR directly, but do it in a separate PR to keep the diff more manageable and reviewable.
a52b1e5 to
3523954
Compare
3523954 to
609d0a0
Compare
|
Thanks for the PR. |
src/wallet/mod.rs
Outdated
| self.payment_store.remove(&payment_id)?; | ||
|
|
||
| self.payment_store.insert_or_update(payment_details)?; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm not fully sure about this flow: since with RBF we can’t guarantee which transaction will eventually confirm (usually the last one, but not always), removing the previous payment here might cause issues. What happens if the earlier tx ends up confirming instead of the latest — would the wallet balance be deducted while the store still shows the payment as pending? If so, could this lead to a confusing UX for the user?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Excellent catch. You're right to be concerned about the potential for state inconsistency.
However, the update_payment_store method acts as a reconciler that syncs the payment store with the actual state of the BDK wallet. Since the payment store is designed to be a reflection of the wallet's state.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We might potentially still end up with multiple entries for the same payment though, right, as we'd not drop the RBF entry once the original transaction confirms? There would also be no 'history' of all the bumps that happened.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, having multiple entries is a possibility. Regarding dropping the entry once the original confirms, that's what I plan on doing in this issue #452 .
As for the history of bumps, a solution could be introducing another status replaced with a replacement ID that indicates the transaction was bumped and replaced.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think one way to simplify tracking RBF transactions is to add two fields to each payment: is_rbf (a boolean indicating whether the transaction is part of an RBF sequence) and origin_payment_id (optional, pointing to the original payment in the sequence). The original transaction would have is_rbf = true and origin_payment_id = None. Each RBF attempt would also have is_rbf = true and set origin_payment_id to the original transaction’s ID. If a new RBF is created from an existing RBF, it can reuse the same origin_payment_id, linking it back to the original.
With this setup, handling confirmations becomes straightforward:
- If an RBF transaction confirms, find the original transaction via
origin_payment_idand clean up all other RBFs linked to it. - When a transaction with is_rbf = true and origin_payment_id = None (i.e., an original transaction) confirms, find all RBF transactions with origin_payment_id equal to its ID and remove them.
This keeps data consistent and ensures that extra lookups are only needed when is_rbf is true, making RBF handling explicit and efficient.
For example, with A as the original transaction and B, C, D as RBFs:
graph TD
A[A<br/>is_rbf=true<br/>origin=None]
B[B<br/>is_rbf=true<br/>origin=A.payment_id]
C[C<br/>is_rbf=true<br/>origin=A.payment_id]
D[D<br/>is_rbf=true<br/>origin=A.payment_id]
A --> B
A --> C
A --> D
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think there’s a design consideration for transactions that won’t confirm because an RBF replaced them. When an RBF transaction confirms, the original and any other RBFs in the sequence become invalid. In my opinion, a more user-friendly approach is to mark them as failed and reference the txid of the confirming transaction, which makes it clear why these transactions didn’t confirm while keeping the history intact.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you for this feedback. This aligns well with the introduction of a replacement id as mentioned above.
Regarding retaining the history of bumped transactions and marking them as failed when any confirms, this could lead to increasing clutter of failed transactions if RBF is used frequently.
My proposed approach during node sync and store updates is when a confirmed transaction is encountered, if it's an RBF transaction, we can confirm that transaction and clear/remove the history of bumps from the store. This prevents unnecessary accumulation of failed transaction records while maintaining proper transaction state tracking.
609d0a0 to
8b7cf78
Compare
Thanks for the review! I've split the PR into two separate commits as suggested |
|
🔔 1st Reminder Hey @tnull! This PR has been waiting for your review. |
|
🔔 2nd Reminder Hey @tnull! This PR has been waiting for your review. |
|
🔔 3rd Reminder Hey @tnull! This PR has been waiting for your review. |
tnull
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Did another pass and added some comments. Still have to take an even closer look at the RBF logic and think through edge cases.
src/wallet/mod.rs
Outdated
| self.payment_store.remove(&payment_id)?; | ||
|
|
||
| self.payment_store.insert_or_update(payment_details)?; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We might potentially still end up with multiple entries for the same payment though, right, as we'd not drop the RBF entry once the original transaction confirms? There would also be no 'history' of all the bumps that happened.
06e7ff3 to
6e6e2aa
Compare
|
🔔 1st Reminder Hey @tnull! This PR has been waiting for your review. |
|
🔔 2nd Reminder Hey @tnull! This PR has been waiting for your review. |
|
🔔 3rd Reminder Hey @tnull! This PR has been waiting for your review. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@Camillarhi Excuse the delay here - last few weeks have been super busy pushing towards an LDK 0.2 beta release. Will see to get back to review on this in a proper ASAP, it might take another week or so though.
In the meantime I want to note that recently there has been some movement towards #446 as BDK now added events (see bitcoindevkit/bdk_wallet#6 / bitcoindevkit/bdk_wallet#310) which will ship as part of the next BDK 2.2 release. So I do wonder if we should generally first move to update our payment store based on the events emitted from syncing (also allowing for #448 to land), and then basing the changes here on top. This might be in particular interesting as bitcoindevkit/bdk_wallet#310 included a TxReplaced event which might allow us to detect and track (all?) RBF transactions belonging to a 'payment' more easily.
Any thoughts on this?
Thanks for the update and for sharing those links. I've reviewed the BDK events implementation, and I agree that using the new events system would be a much cleaner approach for tracking on-chain payment states and handling RBF cases by updating the payment stores with the events emitted. It makes perfect sense to let #448 land first to establish the new events structure, and then rebase the changes from this PR on top of that foundation. On a related note, I saw that you opened #488. Would you like me to continue working on that, or would you prefer to handle it yourself? I'm happy to help. |
So, as mentioned over at #448 and elsewhere, we probably end up delaying #448 a bit more until we get lightningdevkit/rust-lightning#3566 with LDK 0.3. However, we now upgraded to BDK 2.2, i.e., could switch to make use of the events BDK now emits when applying wallet update. This is likely preferable to the current 'always-iterate-all-wallet-transactions' approach we currently use in Then, associating new RBF transactions with the pre-existing payment store entry in this PR could become considerably more easy/less error prone, given that the new |
Yeah, that makes a lot of sense! Switching to BDK 2.2’s sync events would clean up I’d be happy to take this on. I’ll open a PR to migrate |
8ec7391 to
aa1558a
Compare
|
🔔 1st Reminder Hey @tnull! This PR has been waiting for your review. |
tnull
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for the rebase, here are some updated comments.
aa1558a to
8631a4f
Compare
tnull
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Let me know when this is ready for the next round of review. Unfortunately seems to need yet-another rebase already. Sorry!
Sure, I will. |
8dd9382 to
764538a
Compare
|
🔔 1st Reminder Hey @tnull! This PR has been waiting for your review. |
|
🔔 2nd Reminder Hey @tnull! This PR has been waiting for your review. |
764538a to
00de553
Compare
|
🔔 3rd Reminder Hey @tnull! This PR has been waiting for your review. |
tnull
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Already looks pretty good, some comments, and it will need a rebase now that the broadcaster interface changed.
| Address, Amount, FeeRate, OutPoint, ScriptBuf, Transaction, TxOut, Txid, WPubkeyHash, Weight, | ||
| WitnessProgram, WitnessVersion, | ||
| }; | ||
|
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: Changes like this are unrelated and in this case will be reverted by the weekly cargo +nightly fmt job anyways.
| // Get all on-chain payments that are Pending | ||
| let pending_payments: Vec<PendingPaymentDetails> = | ||
| self.pending_payment_store.list_filter(|p| { | ||
| p.details.status == PaymentStatus::Pending |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hmm, wouldn't we expect all in the Pending store to be pending? In fact, should we add a debug assert in 1-2 places that ensures any entries are marked pending, and any other entries are removed from the pending-payment store?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, this is true. All records in PendingStore should be marked as pending. I will also add the debug assert
|
|
||
| if !txs_to_broadcast.is_empty() { | ||
| let tx_refs: Vec<&Transaction> = txs_to_broadcast.iter().collect(); | ||
| self.broadcaster.broadcast_transactions(&tx_refs); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This will need a rebase now as the BroadcasterInterface changed on main (note that the 'real' introduction of TransactionType etc will be in #79).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sure, I will rebase. Also, did you mean #791 ?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Also, did you mean #791 ?
Ah, yes, indeed!
| ConfirmationStatus::Unconfirmed, | ||
| ); | ||
| // We fetch payment details here since the replacement has updated the stored state | ||
| let payment = |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hmm, but that would mean we expect this get to always return Some at this point, right? Can we add a debug_assert checking that it does?
| }; | ||
|
|
||
| let old_tx = | ||
| locked_wallet.tx_details(txid).ok_or(Error::InvalidPaymentId)?.tx.deref().clone(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hmm, same here, we expect this to never fail - can we add a debug_assert?
| let min_required_fee_rate_sat_per_kwu = | ||
| old_fee_rate_sat_per_kwu + INCREMENTAL_RELAY_FEE_SAT_PER_1000_WEIGHT as u64; | ||
|
|
||
| let confirmation_target = ConfirmationTarget::OnchainPayment; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
See above, we might want to allow for an override target fee rate like we do for send_to_address etc?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, that is fine, I will go ahead and add that
src/payment/onchain.rs
Outdated
| /// higher fee, resulting in faster confirmation potential. | ||
| /// | ||
| /// Returns the Txid of the new replacement transaction if successful. | ||
| pub fn bump_fee_rbf(&self, payment_id: PaymentId) -> Result<Txid, Error> { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do we need an optional fee-rate override parameter here, similar to what we have in send_to_address?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, I will go ahead and include it
| Error::InvalidFeeRate | ||
| })?; | ||
|
|
||
| builder.fee_rate(required_fee_rate); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hmm, should we apply an upper-bound to the require_fee_rate, i.e., ensure it's still in some margin of what we expected? Maybe we even want to apply some upper-bound (relative to the original rate?) to our estimate to ensure we don't spend all our funds on fees if the chain source's fee estimation endpoint is lying to us?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah, I thought about this too, that's why I computed a min_required_fee_rate_sat_per_kwu above:
let min_required_fee_rate_sat_per_kwu =
old_fee_rate_sat_per_kwu + INCREMENTAL_RELAY_FEE_SAT_PER_1000_WEIGHT as u64;
Then take the max of that and the estimated_fee_rate. Based on the tests so far, the final_fee_rate is what ends up being used in most cases, the difference between it and required_fee_rate has been around 2-3 sat/kwu. I think it'd be pretty rare for the chain source estimate to actually win out, but an upper-bound sanity check on it is reasonable to add
Add automatic rebroadcasting of unconfirmed transactions triggered by the `ChainTipChanged` event from BDK. This ensures pending transactions remain in mempools.
Add `Replace-by-Fee` functionality to allow users to increase fees on pending outbound transactions, improving confirmation likelihood during network congestion. - Uses BDK's `build_fee_bump` for transaction replacement - Validates transaction eligibility: must be outbound and unconfirmed - Maintains payment history consistency across wallet updates - Includes integration tests for various RBF scenarios
00de553 to
b5b191b
Compare
This PR enhances on-chain transaction management:
Rebroadcast/bumping of on-chain wallet transactions
Handle RBF'd Pending payments
Changes
Related Issues