Steve Myers [Sat, 15 Feb 2025 04:55:06 +0000 (22:55 -0600)]
fix(sqlite): set connection journal_mode to WAL and busy_timeout to 5000 ms
This prevents: Error { code: DatabaseBusy, extended_code: 5 }, Some("database is locked")
which occured when syncing very large number of utxos.
See: electrum::test::test_electrum_large_num_utxos
For some wallets there exists a pathological case where we may try to fetch many thousands of transactions at once, which creates enormous memory pressure. By chunking the batch into more reasonably sized sub-queries, we allow time for memory to be freed.
For Bitkey, we have a user in which the bdk wallet tries to sync ~45000 transactions at once. See the below screenshots the impact of this change via basic memory profiling in our Android app:
### Notes to the reviewers
I am a Rust noob, so please feel free to suggest alternative implementations.
Note that https://github.com/bitcoindevkit/bdk/pull/1828 is also needed to successfully sync wallets of this size.
### Changelog notice
Fix out of memory issue caused by many batch fetching many txs
### Checklists
#### All Submissions:
* [x] I've signed all my commits
* [x] I followed the [contribution guidelines](https://github.com/bitcoindevkit/bdk/blob/master/CONTRIBUTING.md)
* [x] I ran `cargo fmt` and `cargo clippy` before committing
#### Bugfixes:
* [ ] This pull request breaks the existing API
* [ ] I've added tests to reproduce the issue which are now passing
* [x] I'm linking the issue being fixed by this PR
Joey Guinta [Thu, 13 Feb 2025 17:29:39 +0000 (12:29 -0500)]
Address out of memory issue caused by many txs.
For some wallets there exists a pathological case where we may try to fetch many thousands
of transactions at once, which creates enormous memory pressure. By chunking the batch
into more reasonably sized sub-queries, we allow time for memory to be freed.
This is needed to direct new and existing users to upgrade to the new `bdk_wallet`1.0 library.
I also used this PR to bump the version and changelog to v0.30.1 so we're ready to publish the fix in #1782.
### Checklists
#### All Submissions:
* [x] I've signed all my commits
* [x] I followed the [contribution guidelines](https://github.com/bitcoindevkit/bdk/blob/master/CONTRIBUTING.md)
* [x] I ran `cargo fmt` and `cargo clippy` before committing
<!-- You can erase any parts of this template not applicable to your Pull Request. -->
### Description
<!-- Describe the purpose of this PR, what's being adding and/or fixed -->
This PR fixes a bug introduced in https://github.com/bitcoindevkit/bdk/commit/04994e4f7f1073e7bef29528339369994d4b3666#diff-57bf66f87897e694c5ebfdfe0fd366774dda30b43eab93c1a0fdc802d0eb8c8dR171
In that commit `block_times.get(height).is_none()` was converted to `block_times.contains_key(height)`, inverting the code logic.
### Checklists
#### All Submissions:
* [x] I've signed all my commits
* [x] I followed the [contribution guidelines](https://github.com/bitcoindevkit/bdk/blob/master/CONTRIBUTING.md)
* [x] I ran `cargo fmt` and `cargo clippy` before committing
We would previously insert just one coinbase transaction in the database if we caught multiple in the same sync.
When we sync with electrum, before committing to the database, we remove from the update conflicting transactions, using the `make_txs_consistent` function. This function considers two txs to be conflicting if they spend from the same outpoint - but every coinbase transaction spends from the same outpoint!
Here we make sure to avoid filtering out coinbase transactions, by adding a check on the txid just before we do the filtering.
Fixes #1051
### Changelog notice
- Fix a bug when syncing coinbase utxos on electrum
### Checklists
#### All Submissions:
* [x] I've signed all my commits
* [x] I followed the [contribution guidelines](https://github.com/bitcoindevkit/bdk/blob/master/CONTRIBUTING.md)
* [x] I ran `cargo fmt` and `cargo clippy` before committing
#### Bugfixes:
* [x] I've added tests to reproduce the issue which are now passing
* [x] I'm linking the issue being fixed by this PR
Daniela Brozzoni [Tue, 22 Aug 2023 14:29:19 +0000 (16:29 +0200)]
fix(electrum): Don't ignore multiple coinbase txs
We would previously insert just one coinbase transaction in the database
if we caught multiple in the same sync.
When we sync with electrum, before committing to the database, we remove
from the update conflicting transactions, using the
`make_txs_consistent` function. This function considers two txs to be
conflicting if they spend from the same outpoint - but every coinbase
transaction spends from the same outpoint!
Here we make sure to avoid filtering out coinbase transactions, by
adding a check on the txid just before we do the filtering.
Although there is *some* code to handle multipath keys inside bdk,
it's all untested, and from a few quick tests it
seems that it's pretty easy to find buggy edge cases.
Better to deny multipath descs for now, and revisit the
decision once we work on supporting multidescriptor wallets.
Steve Myers [Thu, 3 Aug 2023 19:06:10 +0000 (14:06 -0500)]
release: bump version to 0.28.1
Summary
This patch release backports (from the BDK 1.0 dev branch) a fix for a bug in the policy condition calculation and adds a new taproot single key descriptor template (BIP-86). The policy condition calculation bug can cause issues when a policy subtree fails due to missing info even if it's not selected when creating a new transaction, errors on unused policy paths are now ignored.
Fixed
- Backported #932 fix for policy condition calculation #1008
Remove 1.57.0 MSRV dependency pinning from Cargo.toml and use `cargo update --precise` method instead. This is how bdk 1.0.0 and other rust projects are doing it now.
### Notes to the reviewers
This is needed to get the release/0.28 branch to build again. After it's merged I'll be able to start making the `0.28.1` release.
### Changelog notice
none.
### Checklists
#### All Submissions:
* [x] I've signed all my commits
* [x] I followed the [contribution guidelines](https://github.com/bitcoindevkit/bdk/blob/master/CONTRIBUTING.md)
* [x] I ran `cargo fmt` and `cargo clippy` before committing
This PR solves #836 for the release/0.28 branch. This PR adds a P2TR descriptor template and a BIP86 taproot descriptor template. With this, users can now create a taproot descriptor with templates.
### Notes to the reviewers
The commit from #840 is cherry-picked from the `master` branch to the `release/0.28` branch without any changes.
### Changelog notice
Add taproot descriptor template (BIP-86).
### Checklists
#### All Submissions:
* [x] I've signed all my commits
* [x] I followed the [contribution guidelines](https://github.com/bitcoindevkit/bdk/blob/master/CONTRIBUTING.md)
* [x] I ran `cargo fmt` and `cargo clippy` before committing
#### New Features:
* [x] I've added tests for the new feature
* [x] I've added docs for the new feature
Vladimir Fomene [Tue, 14 Feb 2023 22:16:02 +0000 (01:16 +0300)]
Create taproot descriptor template
This PR solves #836. This PR adds a P2TR
descriptor template and a BIP86 taproot
descriptor template. With this, users
can now create a taproot descriptor with templates.
9cffaad71f3d090196a3e35e84ad714118ea2846 Fix build errors (junderw) 3ccdb84523997d3a9582b1ebd2cf2fe373ddaa55 Fix policy condition calculation When constructing the `Condition` struct we recursively call `get_condition` on all the items in a threshold and short-circuit if there's an error somewhere (for example, because the policy-path hasn't been provided for a specific threshold). (Alekos Filini)
Pull request description:
Fixes #942
### Description
This backports "Fix policy condition calculation" (#932) onto release/0.28
### Notes to the reviewers
Currently kept the Author the same and the committer is myself. Let me know if this is not the desired outcome of the history. (I have made no modifications myself)
### Changelog notice
Backported "Fix policy condition calculation"
### Checklists
#### All Submissions:
* [x] I've signed all my commits
* [x] I followed the [contribution guidelines](https://github.com/bitcoindevkit/bdk/blob/master/CONTRIBUTING.md)
* [x] I ran `cargo fmt` and `cargo clippy` before committing
Alekos Filini [Fri, 16 Jun 2023 20:49:31 +0000 (13:49 -0700)]
Fix policy condition calculation
When constructing the `Condition` struct we recursively call
`get_condition` on all the items in a threshold and short-circuit if
there's an error somewhere (for example, because the policy-path hasn't
been provided for a specific threshold).
This can cause issues when the user doesn't care about a subtree, because
we still try to call `get_condition` on all the items and fail if
something is missing, even if the specific subtree isn't selected and
won't be used later on.
This commit changes the logic so that we first filter only the `selected`
items, and then unwrap the error using the question mark. If errors
happened somewhere else they will be ignored, as it should.
Add new `std` feature which enables `bitcoin/std` and `miniscript/std`, and enable `std` feature in `default` feature set.
### Notes to the reviewers
When using `bdk` in a wasm environment then `default` features should be disabled and `bitcoin/no-std` and `miniscript/no-std` must be enabled. This is a breaking change for anyone using `bdk` with `--no-default-features`.
I also updated the workflows to also run for `release/*` branches in addition to `master`.
### Changelog notice
Changed
- Added new `std` feature as part of `default` features, `std` must be enabled unless building for wasm.
### Checklists
#### All Submissions:
* [x] I've signed all my commits
* [x] I followed the [contribution guidelines](https://github.com/bitcoindevkit/bdk/blob/master/CONTRIBUTING.md)
* [x] I ran `cargo fmt` and `cargo clippy` before committing
#### New Features:
* [ ] I've added tests for the new feature
* [x] I've added docs for the new feature
<!-- You can erase any parts of this template not applicable to your Pull Request. -->
### Description
The updated version of `rust-bip39` was just released. It also [bumped the pinned version](https://github.com/rust-bitcoin/rust-bip39/pull/41) of `unicode-normalization`, which previously had lead to some incompatibilities.
### Notes to the reviewers
<!-- In this section you can include notes directed to the reviewers, like explaining why some parts
of the PR were done in a specific way -->
### Changelog notice
<!-- Notice the release manager should include in the release tag message changelog -->
<!-- See https://keepachangelog.com/en/1.0.0/ for examples -->
### Checklists
#### All Submissions:
* [x] I've signed all my commits
* [x] I followed the [contribution guidelines](https://github.com/bitcoindevkit/bdk/blob/master/CONTRIBUTING.md)
* [x] I ran `cargo fmt` and `cargo clippy` before committing
Update the `esplora-client` to the new `0.4` release.
### Notes to the reviewers
The `esplora-client` was updated in bitcoindevkit/rust-esplora-client#39 to not include default `rust-bitcoin` dependencies.
### Changelog notice
- Update the `esplora-client` optional dependency to version `0.4`.
### Checklists
#### All Submissions:
* [x] I've signed all my commits
* [x] I followed the [contribution guidelines](https://github.com/bitcoindevkit/bdk/blob/master/CONTRIBUTING.md)
* [x] I ran `cargo fmt` and `cargo clippy` before committing
A new [`rust-miniscript` release 9.0](https://github.com/rust-bitcoin/rust-miniscript/blob/master/CHANGELOG.md#900---november-5-2022) came out on Nov 14, updating to it to pickup the bug fixes. Also updating dependency`hwi` to new `0.5` version which used the `9.0` version of `rust-miniscript`.
### Notes to the reviewers
This new version of `rust-miniscript` uses the same version of `rust-bitcoin` we are on, 0.29.1.
### Changelog notice
Update rust-miniscript dependency to latest bug fix release 9.0.
### Checklists
#### All Submissions:
* [x] I've signed all my commits
* [x] I followed the [contribution guidelines](https://github.com/bitcoindevkit/bdk/blob/master/CONTRIBUTING.md)
* [x] I ran `cargo fmt` and `cargo clippy` before committing
### Description
Very small fixes to documentation:
1. I got a DM last week from a user who thought we had a bug with our timestamps. It turns out he was using the milliseconds version of the Unix timestamp in his project and didn't realize we were giving out a standard Unix timestamp. The docs now mention this explicitly.
2. I noticed some small inconsistencies in the documentation on the public templates while porting them over to Kotlin. This PR also fixes that so that all templates use a common documentation wording.
### Checklists
#### All Submissions:
* [x] I've signed all my commits
* [x] I followed the [contribution guidelines](https://github.com/bitcoindevkit/bdk/blob/master/CONTRIBUTING.md)
* [x] I ran `cargo fmt` and `cargo clippy` before committing
This enables e.g. `assert_eq!` comparisons of `SyncTime` in tests (we use this)
### Changelog notice
Derived `PartialEq`, `Eq` on `SyncTime`
#### All Submissions:
* [x] I've signed all my commits
* [x] I followed the [contribution guidelines](https://github.com/bitcoindevkit/bdk/blob/master/CONTRIBUTING.md)
* [x] I ran `cargo fmt` and `cargo clippy` before committing
We don't automatically want to make the interface `async` based on the used architecture, but now require the user to explicitly set the `async-interface` feature.
#### All Submissions:
* [x] I've signed all my commits
* [x] I followed the [contribution guidelines](https://github.com/bitcoindevkit/bdk/blob/master/CONTRIBUTING.md)
* [x] I ran `cargo fmt` and `cargo clippy` before committing
* [X] I've signed all my commits
* [X] I followed the [contribution guidelines](https://github.com/bitcoindevkit/bdk/blob/master/CONTRIBUTING.md)
* [X] I ran `cargo fmt` and `cargo clippy` before committing
#### Bugfixes:
* [ ] This pull request breaks the existing API
* [ ] I've added tests to reproduce the issue which are now passing
* [X] I'm linking the issue being fixed by this PR
Elias Rohrer [Tue, 10 Jan 2023 14:05:55 +0000 (15:05 +0100)]
Don't default to use `async`/`await` on `wasm32`
We don't automatically want to make the interface `async` based on the
used architecture, but now require the user to explicitly set the
`async-interface` feature.
Also bumped the `check-wasm` and `test_hardware_wallet` jobs rust STABLE version to 1.65.0 to match other jobs.
### Notes to the reviewers
The `rustls` crate changed their MSRV to 1.57 on 2023-01-12 with a patch release from 0.20.7 to 0.20.8, rustls/rustls#1152. This breaks our CI builds that use `explora-client` 0.3 because it depends on a version of `ureq` that uses the latest `rustls`.
* [x] I've signed all my commits
* [x] I followed the [contribution guidelines](https://github.com/bitcoindevkit/bdk/blob/master/CONTRIBUTING.md)
* [x] I ran `cargo fmt` and `cargo clippy` before committing
* Fix clippy for Rust 1.65.0
* Bump CI Rust stable version to 1.65.0
### Checklists
#### All Submissions:
* [X] I've signed all my commits
* [X] I followed the [contribution guidelines](https://github.com/bitcoindevkit/bdk/blob/master/CONTRIBUTING.md)
* [X] I ran `cargo fmt` and `cargo clippy` before committing
<!-- You can erase any parts of this template not applicable to your Pull Request. -->
### Description
Update esplora-client and electrsd.
Update: `electrsd v0.22` requires msrv at 1.57.. So removed from the PR for now..
### Notes to the reviewers
<!-- In this section you can include notes directed to the reviewers, like explaining why some parts
of the PR were done in a specific way -->
### Changelog notice
<!-- Notice the release manager should include in the release tag message changelog -->
<!-- See https://keepachangelog.com/en/1.0.0/ for examples -->
### Checklists
#### All Submissions:
* [x] I've signed all my commits
* [x] I followed the [contribution guidelines](https://github.com/bitcoindevkit/bdk/blob/master/CONTRIBUTING.md)
* [x] I ran `cargo fmt` and `cargo clippy` before committing
projects / bdk / log