Used regex "\[\^\w*[\s]+\w*\]" to find and remove spaces from footnote tags.
The [previous version] of the [Rusty Paper Wallet] followed the original paper wallet design: WIF[^WIF] as secret part with the option to generate a different kind of addresses (legacy, nested segwit, and segwit).
-There were plans to [support mnemonic](https://github.com/RCasatta/rusty-paper-wallet/issues/5) instead of WIF keys because it may[^WIF core] save the sweep transaction[^sweep] and there are more wallets capable of importing a mnemonic instead of a WIF.
+There were plans to [support mnemonic](https://github.com/RCasatta/rusty-paper-wallet/issues/5) instead of WIF keys because it may[^WIFcore] save the sweep transaction[^sweep] and there are more wallets capable of importing a mnemonic instead of a WIF.
However, choosing a single address type or having wallet support for a specific format is the kind of problem [descriptors] solve perfectly, so the latest [Rusty Paper Wallet] version now accepts a descriptor and the network as parameters.
The paper wallet must then be printed, and it is better to use a printer without wifi and also to be aware that some sensitive data may remain in the printer's cache.
-Then the paper wallet must be cut along the dotted lines, the secret part should be folded twice over the black zone[^black zone]. The black zone helps to avoid showing the secret parts in the presence of back-light. Once the folding is done the paper wallet should be plasticized to prevent being damaged by water.
+Then the paper wallet must be cut along the dotted lines, the secret part should be folded twice over the black zone[^blackzone]. The black zone helps to avoid showing the secret parts in the presence of back-light. Once the folding is done the paper wallet should be plasticized to prevent being damaged by water.
## BDK
[guarantees]: https://github.com/bitcoin/bitcoin/blob/master/doc/descriptors.md#checksums
[^WIF]: Wallet Input Format, a string encoding a ECDSA private key https://en.bitcoin.it/wiki/Wallet_import_format
-[^WIF core]: Unless the user import the WIF directly into bitcoin core
+[^WIFcore]: Unless the user import the WIF directly into bitcoin core
[^sweep]: Some wallets refers to sweep as the action to create a transaction taking all the funds from the paper wallet and sending those to the wallet itself.
-[^black zone]: Ideally, the black zone should be twice as long as the secret part to cover it back and front, long descriptor may leave a shorter black zone, ensure to have you printer set with vertical layout for best results.
+[^blackzone]: Ideally, the black zone should be twice as long as the secret part to cover it back and front, long descriptor may leave a shorter black zone, ensure to have you printer set with vertical layout for best results.
## Introduction: what is fee estimation?
-Fee estimation is the process of selecting the fee rate[^fee rate] for a bitcoin transaction being created, according to two main factors:
+Fee estimation is the process of selecting the fee rate[^feerate] for a bitcoin transaction being created, according to two main factors:
* The current congestion of the Bitcoin network.
* The urgency, or lack thereof, for the transaction confirmation, i.e, its inclusion in a block.
We need to register the moment in time when transactions enter in the node's mempool; to be efficient and precise we should not only call the RPC endpoints but listen to [ZMQ] events. Luckily, the just released bitcoin core 0.21.0 added a new [ZMQ] topic `zmqpubsequence` notifying mempool events (and block events). The logger is also listening to `zmqpubrawtx` and `zmqpubrawblock` topics, to make less RPC calls.
We are not only interested in the timestamp of the transaction entering the mempool, but also how many blocks it will take until the same transaction is confirmed.
-In the final dataset this field is called `confirms_in`[^blocks target]; if `confirms_in = 1` it means the transaction is confirmed in the first block created after it has been seen for the first time.
+In the final dataset this field is called `confirms_in`[^blockstarget]; if `confirms_in = 1` it means the transaction is confirmed in the first block created after it has been seen for the first time.
Another critical piece of information logged by the data logger is the `fee_rate` of the transaction, since the absolute fee value paid by a bitcoin transaction is not available nor derivable given only the transaction itself, as the inputs don't have explicit amounts.
In the following [Part 2] we are going to talk about the dataset.
-[^fee rate]: The transaction fee rate is the ratio between the absolute fee expressed in satoshi, over the weight of the transaction measured in virtual bytes. The weight of the transaction is similar to the byte size, however a part of the transaction (the segwit part) is discounted, their byte size is considered less because it creates less burden for the network.
+[^feerate]: The transaction fee rate is the ratio between the absolute fee expressed in satoshi, over the weight of the transaction measured in virtual bytes. The weight of the transaction is similar to the byte size, however a part of the transaction (the segwit part) is discounted, their byte size is considered less because it creates less burden for the network.
[^mempool]: mempool is the set of transactions that are valid by consensus rules (for example, they are spending existing bitcoin), broadcasted in the bitcoin peer to peer network, but they are not yet part of the blockchain.
[^disclaimer]: DISCLAIMER: I am not an expert data-scientist!
-[^blocks target]: Conceptually similar to bitcoin core `estimatesmartfee` parameter called "blocks target", however, `confirms_in` is the real value not the desired target.
+[^blockstarget]: Conceptually similar to bitcoin core `estimatesmartfee` parameter called "blocks target", however, `confirms_in` is the real value not the desired target.
[^fast]: 16GB of compressed raw logs are processed and a compressed CSV produced in about 5 minutes.
[Part 1]: /blog/2021/01/fee-estimation-for-light-clients-part-1/
The model is for light-client such as [neutrino] based ones. In these clients the mempool is already available (it's needed to check for received transactions) but we can't compute fee rates of this transactions because previous confirmed inputs are not in the mempool!
-Luckily, **thanks to temporal locality [^temporal locality], an important part of mempool transactions spend outputs created very recently**, for example in the last 6 blocks.
+Luckily, **thanks to temporal locality [^temporallocality], an important part of mempool transactions spend outputs created very recently**, for example in the last 6 blocks.
The blocks are available through the p2p network, and downloading the last 6 is considered a good compromise between resource consumption and accurate prediction. We need the model to be built with the same data available in the prediction phase, as a consequence *the mempool data in the dataset refers only to transactions having their inputs in the last 6 blocks*. However the `bitcoin-csv` tool inside the [data logger] allows to configure this parameter.
#### The outliers
In the following [Part 3] we are going to talk about the model.
-[^temporal locality]: In computer science temporal locality refers to the tendency to access recent data more often than older data.
+[^temporallocality]: In computer science temporal locality refers to the tendency to access recent data more often than older data.
[Part 1]: /blog/2021/01/fee-estimation-for-light-clients-part-1/
[Part 2]: /blog/2021/01/fee-estimation-for-light-clients-part-2/
A significant part of a ML model are the activation functions, `relu` (Rectified Linear Unit) is one of the most used lately, because it's simple and works well as we learned in this [introducing neural network video]. `relu` it's equal to zero for negative values and equal to the input for positive values. Being non-linear allows the whole model to be non-linear.
-For the last layer it is different: we want to enforce a minimum for the output, which is the minimum relay fee `1.0`[^minimum relay fee]. One could not simply cut the output of the model after prediction because all the training would not consider this constraint. So we need to build a custom activation function that the model training will be able to use for the [gradient descent] optimization step. Luckily this is very simple using tensorflow primitives:
+For the last layer it is different: we want to enforce a minimum for the output, which is the minimum relay fee `1.0`[^minimumrelayfee]. One could not simply cut the output of the model after prediction because all the training would not consider this constraint. So we need to build a custom activation function that the model training will be able to use for the [gradient descent] optimization step. Luckily this is very simple using tensorflow primitives:
```
def clip(x):
## The prediction phase
-A [prediction test tool] is available on github. At the moment it uses a bitcoin core node to provide network data for simplicity, but it queries it only for the mempool and the last 6 blocks, so it's something that also a light-client could do[^disabled bloom filter].
+A [prediction test tool] is available on github. At the moment it uses a bitcoin core node to provide network data for simplicity, but it queries it only for the mempool and the last 6 blocks, so it's something that also a light-client could do[^disabledbloomfilter].
The following chart is probably the best visualization to evaluate the model, on the x axis there is the real fee rate while on the y axis there is the prediction, the more the points are centered on the bisection, the more the model is good.
We can see the model is doing quite well, the MAE is 8 which is way lower than `estimatesmartfee`. However, there are big errors some times, in particular for prediction for fast confirmation (`confirms_in=1 or confirms_in=2`) as shown by the orange points. Creating a model only for blocks target greater than 2 instead of simply remove some observations may be an option.
[^MAE]: MAE is Mean Absolute Error, which is the average of the series built by the absolute difference between the real value and the estimation.
[^drift]: drift like MAE, but without the absolute value
-[^minimum relay fee]: Most node won't relay transactions with fee lower than the min relay fee, which has a default of `1.0`
-[^disabled bloom filter]: An important issue emerged after the article came out, a bitcoin core client with bloom filter disabled (default as of 0.21) doesn't serve the mempool via p2p.
+[^minimumrelayfee]: Most node won't relay transactions with fee lower than the min relay fee, which has a default of `1.0`
+[^disabledbloomfilter]: An important issue emerged after the article came out, a bitcoin core client with bloom filter disabled (default as of 0.21) doesn't serve the mempool via p2p.
[Part 1]: /blog/2021/01/fee-estimation-for-light-clients-part-1/
[Part 2]: /blog/2021/01/fee-estimation-for-light-clients-part-2/
projects / bitcoindevkit.org / commitdiff