title: "Fee estimation for light-clients (Part 1)"
description: ""
author: "Riccardo Casatta"
-date: "2021-01-21"
+date: "2021-01-25"
tags: ["fee", "machine learning"]
hidden: true
draft: false
where the `fee_rate` column is the output we want, also called the "*target*" or "*label*" in ML terminology, and the other columns are our inputs.
Can we build this table just by looking at the Bitcoin blockchain? Unfortunately, we can't:
-The main thing that's missing is an indication of when the node first saw a transaction that has been later confirmed in a block. With that knowledge we can say that the fee-rate of that transaction was the exact value required to confirm
+The main thing that's missing is an indication of when the node first saw a transaction that has been later confirmed in a block. With that knowledge we can say that the fee rate of that transaction was the exact value required to confirm
within the number of blocks it actually took to be confirmed. For instance, if we see transaction `t` when the blockchain is at height `1000` and then we notice that `t` has been included in block `1006`, we can deduce that the
-fee-rate paid by `t` was the exact value required to get confirmed within the next `6` blocks.
+fee rate paid by `t` was the exact value required to get confirmed within the next `6` blocks.
So to build our model, we first need to gather these data, and machine learning needs a *lot* of data to work well.
-My logger instance started collecting data on the 18th of December 2020, and as of today (18th January 2020), the raw logs are about 14GB.
+My logger instance started collecting data on the 18th of December 2020, and as of today (25th January 2020), the raw logs are about 16GB.
I expect (or at least hope) the raw logs, the CSV dataset, or the data logger will be useful also for other projects as well, like monitoring the propagation of transactions or other works involving raw mempool data. We will share raw logs data through torrent soon.
[^fee rate]: The transaction fee rate is the ratio between the absolute fee expressed in satoshi, over the weight of the transaction measured in virtual bytes. The weight of the transaction is similar to the byte size, however a part of the transaction (the segwit part) is discounted, their byte size is considered less because it creates less burden for the network.
[^mempool]: mempool is the set of transactions that are valid by consensus rules (for example, they are spending existing bitcoin), broadcasted in the bitcoin peer to peer network, but they are not yet part of the blockchain.
-[^temporal locality]: In computer science temporal locality refers to the tendency to access recent data more often than older data.
[^disclaimer]: DISCLAIMER: I am not an expert data-scientist!
-[^MAE]: MAE is Mean Absolute Error, which is the average of the series built by the absolute difference between the real value and the estimation.
-[^drift]: drift like MAE, but without the absolute value
-[^minimum relay fee]: Most node won't relay transactions with fee lower than the min relay fee, which has a default of `1.0`
[^blocks target]: Conceptually similar to bitcoin core `estimatesmartfee` parameter called "blocks target", however, `confirms_in` is the real value not the desired target.
-[^fast]: 14GB of compressed raw logs are processed and a compressed CSV produced in about 4 minutes.
+[^fast]: 16GB of compressed raw logs are processed and a compressed CSV produced in about 5 minutes.
[Part 1]: /blog/2021/01/fee-estimation-for-light-clients-part-1/
[Part 2]: /blog/2021/01/fee-estimation-for-light-clients-part-2/
[Part 3]: /blog/2021/01/fee-estimation-for-light-clients-part-3/
[`estimatesmartfee`]: https://bitcoincore.org/en/doc/0.20.0/rpc/util/estimatesmartfee/
-[core]: https://bitcoincore.org/
[bitmex withdrawals]: https://b10c.me/mempool-observations/2-bitmex-broadcast-13-utc/
[fee estimators]: https://b10c.me/blog/003-a-list-of-public-bitcoin-feerate-estimation-apis/
[neutrino]: https://github.com/bitcoin/bips/blob/master/bip-0157.mediawiki
[weekend]: https://www.blockchainresearchlab.org/2020/03/30/a-week-with-bitcoin-transaction-timing-and-transaction-fees/
[ZMQ]: https://github.com/bitcoin/bitcoin/blob/master/doc/zmq.md
[data logger]: https://github.com/RCasatta/bitcoin_logger
-[this one]: https://blockstream.info/tx/33291156ab79e9b4a1019b618b0acfa18cbdf8fa6b71c43a9eed62a849b86f9a
-[dataset]: https://storage.googleapis.com/bitcoin_log/dataset_17.csv.gz
-[google colab notebook]: https://colab.research.google.com/drive/1yamwh8nE4NhmGButep-pfUT-1uRKs49a?usp=sharing
-[plain python]: https://github.com/RCasatta/
-[example]: https://www.tensorflow.org/tutorials/keras/regression
-[tune hyperparameters]: https://www.tensorflow.org/tutorials/keras/keras_tuner
-[advice]: https://www.tensorflow.org/tutorials/keras/overfit_and_underfit#demonstrate_overfitting
-[introducing neural network video]: https://youtu.be/aircAruvnKk?t=1035
-[gradient descent]: https://en.wikipedia.org/wiki/Gradient_descent#:~:text=Gradient%20descent%20is%20a%20first,the%20direction%20of%20steepest%20descent.
-[latest trend]: https://towardsdatascience.com/adam-latest-trends-in-deep-learning-optimization-6be9a291375c
-[exponential decay]: https://www.tensorflow.org/api_docs/python/tf/compat/v1/train/exponential_decay
-[prediction test tool]: https://github.com/RCasatta/estimate_ml_fee
-[bdk]: https://github.com/bitcoindevkit/bdk
-[Square crypto]: https://squarecrypto.org/
-[get stuck]: https://github.com/RCasatta/bitcoin_logger/blob/master/notes.md
-[hashed feature columns]: https://www.tensorflow.org/tutorials/structured_data/feature_columns#hashed_feature_columns
-[tensorflow]: https://www.tensorflow.org/
-[TFRecord format]: https://www.tensorflow.org/tutorials/load_data/tfrecord
-[Leonardo Comandini]: https://twitter.com/LeoComandini
-[Domenico Gabriele]: https://twitter.com/domegabri
-[Alekos Filini]: https://twitter.com/afilini
-[Ferdinando Ametrano]: https://twitter.com/Ferdinando1970
-[I]: https://twitter.com/RCasatta
title: "Fee estimation for light-clients (Part 2)"
description: ""
author: "Riccardo Casatta"
-date: "2021-01-21"
+date: "2021-01-25"
tags: ["fee", "machine learning"]
hidden: true
draft: false
## The dataset
-The [dataset] is publicly available (~400MB gzip compressed, ~1.6GB as plain CSV).
+The [dataset] is publicly available (~500MB gzip compressed, ~2GB as plain CSV).
-The output of the model is the fee rate, expressed in `[satoshi/vbytes]` **TODO: missing link??**.
+The output of the model is the fee rate, expressed in `[satoshi/vbytes]`.
What about the inputs? Generally speaking, we have two main requirements for what can be included as input for our model:
* It must be correlated to the output, even with a non-linear relation.
* It must be available to a light client: for instance, assuming to have knowledge and an index of the last 1000 blocks is considered too much.
-To evaluate the approach we are taking, we also want to compare our model's results with another available estimation: for this reason the dataset includes data to compute the error agains Bitcoin Core's `estimatesmartfee` results, though we are not going to use it for this model.
+To evaluate the approach we are taking, we also want to compare our model's results with another available estimation: for this reason the dataset includes data to compute the error agains Bitcoin Core's `estimatesmartfee` results, even though we are not going to use it for this model.
The dataset will contain only transactions that spend already confirmed inputs. If we wanted to include transactions with unconfirmed inputs as well, the fee rate would have to be computed as a whole;
for example if transaction `t2` spends an unconfirmed input from `t1` (while `t1` only spends confirmed inputs, and all its other outputs are unspent), the aggregated fee rate would have to be used.
bucket | bucket min fee rate | bucket max fee rate
-|-|-
-a0|1.0|1.5 = min*(1+`percentage_increment`)
-a1|1.5 = previous max|2.25
-a2|2.25| 3.375
-...|...|...
-a15|437.89|inf
+a0| 1.0 = `min_relay_fee` | 1.5 = min*(1+`percentage_increment`)
+a1| 1.5 = previous max | 2.25
+a2| 2.25 | 3.375
+ai| ... | `min_relay_fee` * (1+`percentage_increment`)^(i+1)
+a15| 437.89 | inf
The array stops at `a15` because `a16` would have a bucket min greater than `array_max`.
-We previously stated this model is for light-client such as [neutrino] based ones. In these clients the mempool is already available (it's needed to check for received transactions) but the problem is we can't compute fee rates of this transactions because previous confirmed inputs are not in the mempool!
+The model is for light-client such as [neutrino] based ones. In these clients the mempool is already available (it's needed to check for received transactions) but we can't compute fee rates of this transactions because previous confirmed inputs are not in the mempool!
Luckily, **thanks to temporal locality [^temporal locality], an important part of mempool transactions spend outputs created very recently**, for example in the last 6 blocks.
-The blocks are available through the p2p network, and downloading the last 6 is considered a good compromise between resource consumption and accurate prediction. We need the model to be built with the same data available in the prediction phase, as a consequence the mempool data in the dataset refers only to transactions having their inputs in the last 6 blocks. However the `bitcoin-csv` tool inside the [data logger] allows to configure this parameter.
+The blocks are available through the p2p network, and downloading the last 6 is considered a good compromise between resource consumption and accurate prediction. We need the model to be built with the same data available in the prediction phase, as a consequence *the mempool data in the dataset refers only to transactions having their inputs in the last 6 blocks*. However the `bitcoin-csv` tool inside the [data logger] allows to configure this parameter.
#### The outliers
-Another information the dataset contain is the block percentile fee rate, considering `r_i` to be the rate of the `ith` transaction in a block, `q_k` is the fee rate value such that for each transaction in a block `r_i` < `q_k` returns the `k%` transactions in the block that are paying lower fees.
+The dataset also contains the block percentile fee rate `q_k`, considering `r_i` to be the rate of the `ith` transaction in a block, `q_k` is the fee rate value such that for each transaction in a block `r_i` < `q_k` returns the `k%` transactions in the block that are paying lower fees.
Percentiles are not used to feed the model but to filter some outliers tx.
Removing this observations is controversial at best and considered cheating at worse. However, it should be considered that Bitcoin Core `estimatesmartfee` doesn't even bother to give estimation for the next block, we think this is due to the fact that many transactions that are confirming in the next block are huge overestimation [^overestimation], or clearly errors like [this one] we found when we started logging data.
-These outliers are a lot for transactions confirming in the next block (`confirms_in=1`), less so for `confirms_in=2`, mostly disappeared for `confirms_in=3` or more. It's counterintuitive that overestimation exist for `confirms_in>1`, by definition an overestimation is a fee rate way higher than needed, so how is possible that an overestimation doesn't enter the very next block? There are a couple of reasons why a block is discovered without containing a transaction with high fee rate:
+These outliers are several for transactions confirming in the next block (`confirms_in=1`), less so for `confirms_in=2`, mostly disappeared for `confirms_in=3` or more. It's counterintuitive that overestimation exists for `confirms_in>1`, by definition an overestimation is a fee rate way higher than needed, so how is possible that an overestimation doesn't enter the very next block? There are a couple of reasons why a block is discovered without containing a transaction with high fee rate:
* network latency: my node saw the transaction but the miner didn't see that transaction yet,
* block building latency: the miner saw the transaction, but didn't finish to rebuild the block template or decided it's more efficient to finish a cycle on the older block template.
block_avg_fee | no | block average fee rate `[sat/vbytes]` of block `current_height+confirms_in`
core_econ | no | bitcoin `estimatesmartfee` result for `confirms_in` block target and in economic mode. Could be not available `?` when a block is connected more recently than the estimation has been requested, estimation are requested every 10 secs.
core_cons | no | Same as above but with conservative mode
-mempool_len | no | Sum of the mempool transaction with fee rate available (sum of every `a*` field)
+mempool_len | no | Sum of the mempool transactions with fee rate available (sum of every `a*` field)
parent_in_cpfp | no | It's 1 when the transaction has outputs that are spent in the same block in which the transaction is confirmed (they are parent in a CPFP relations).
q1-q30-... | no | Transaction confirming fast could be outliers, usually paying a lot more than required, this percentiles are used to filter those transactions,
a1-a2-... | yes | Contains the number of transaction in the mempool with known fee rate in the ith bucket.
[^drift]: drift like MAE, but without the absolute value
[^minimum relay fee]: Most node won't relay transactions with fee lower than the min relay fee, which has a default of `1.0`
[^blocks target]: Conceptually similar to Bitcoin Core `estimatesmartfee` parameter called "blocks target", however, `confirms_in` is the real value not the desired target.
-[^fast]: 14GB of compressed raw logs are processed and a compressed CSV is produced in about 4 minutes.
[Part 1]: /blog/2021/01/fee-estimation-for-light-clients-part-1/
[Part 2]: /blog/2021/01/fee-estimation-for-light-clients-part-2/
[ZMQ]: https://github.com/bitcoin/bitcoin/blob/master/doc/zmq.md
[data logger]: https://github.com/RCasatta/bitcoin_logger
[this one]: https://blockstream.info/tx/33291156ab79e9b4a1019b618b0acfa18cbdf8fa6b71c43a9eed62a849b86f9a
-[dataset]: https://storage.googleapis.com/bitcoin_log/dataset_17.csv.gz
+[dataset]: https://storage.googleapis.com/bitcoin_log/dataset_18.csv.gz
[google colab notebook]: https://colab.research.google.com/drive/1yamwh8nE4NhmGButep-pfUT-1uRKs49a?usp=sharing
-[plain python]: https://github.com/RCasatta/
[example]: https://www.tensorflow.org/tutorials/keras/regression
[tune hyperparameters]: https://www.tensorflow.org/tutorials/keras/keras_tuner
[advice]: https://www.tensorflow.org/tutorials/keras/overfit_and_underfit#demonstrate_overfitting
title: "Fee estimation for light-clients (Part 3)"
description: ""
author: "Riccardo Casatta"
-date: "2021-01-21"
+date: "2021-01-25"
tags: ["fee", "machine learning"]
hidden: true
draft: false
The code building and training the model with [tensorflow] is available in [google colab notebook] (jupyter notebook); you can also download the file as plain python and run it locally. About 30 minutes are needed to train the model, but it heavily depends on the hardware available.
-
+
<div align="center">Tired to read and want a short interesting evidence? In the last month a ~50 sat/vbyte transaction never took more than a day to confirm and a ~5 sat/vbyte never took more than a week</div><br/>
-As a reference, in the code we have a calculation of the bitcoin core `estimatesmartfee` MAE[^MAE] and drift[^drift], note this are `[satoshi/bytes]` (not virtual bytes).
-MAE is computed as `avg(abs(fee_rate_bytes - core_econ))` when `core_econ` is available (about 1.2M observations, sometime the value is not available when considered too old).
+As a reference, in the code we have a calculation of the bitcoin core `estimatesmartfee` MAE[^MAE] and drift[^drift].
+MAE is computed as `avg(abs(fee_rate - core_econ))` when `core_econ` is available (about 1.2M observations, sometime the value is not available when considered too old).
-estimatesmartfee mode | MAE [satoshi/bytes] | drift
+estimatesmartfee mode | MAE [satoshi/vbytes] | drift
-|-|-
-economic| 35.22 | 29.76
-conservative | 54.28 | 53.13
+economic| 28.77 | 20.79
+conservative | 46.49 | 44.73
As seen from the table, the error is quite high, but the positive drift suggests `estimatesmartfee` prefers to overestimate to avoid transactions not confirming.
-
As we said in the introduction, network traffic is correlated with time and we have the timestamp of when the transaction has been first seen, however a ML model doesn't like plain numbers too much, but it behaves better with "number that repeats", like categories, so we are converting the timestamp in `day_of_week` a number from 0 to 6, and `hours` a number from 0 to 24.
#### Splitting
metrics=['mae', 'mse'])
```
-
+
The model is fed with the `encoded_features` coming from the processing phase, then there are 2 layers with 64 neurons each followed by one neuron giving the `fee_rate` as output.
49*64+65*64+ = 7361
```
-Honestly, about the neural network parameters, they are mostly the one taken from this tensorflow [example], we even tried to [tune hyperparameters], however, we decided to follow this [advice]: *"The simplest way to prevent overfitting is to start with a small model:"*. We hope this work will attract other data scientists to this bitcoin problem, improving the model. We also think that a longer time for the data collection is needed to capture various situations.
+Honestly, neural network parameters are mostly the one taken from this tensorflow [example], we even tried to [tune hyperparameters], however, we decided to follow this [advice]: *"The simplest way to prevent overfitting is to start with a small model:"*. We hope this work will attract other data scientists to this bitcoin problem, improving the model. We also think that a longer time for the data collection is needed to capture various situations.
A significant part of a ML model are the activation functions, `relu` (Rectified Linear Unit) is one of the most used lately, because it's simple and works well as we learned in this [introducing neural network video]. `relu` it's equal to zero for negative values and equal to the input for positive values. Being non-linear allows the whole model to be non-linear.
```
Epoch 1/200
-5617/5617 [==============================] - 40s 6ms/step - loss: 546.4828 - mae: 16.7178 - mse: 546.4828 - val_loss: 297.0464 - val_mae: 11.4745 - val_mse: 297.0464
+7559/7559 [==============================] - 34s 3ms/step - loss: 547.8023 - mae: 16.9547 - mse: 547.8023 - val_loss: 300.5965 - val_ma
+e: 11.9202 - val_mse: 300.5965
...
-Epoch 198/200
-5617/5617 [==============================] - 38s 6ms/step - loss: 147.4738 - mae: 7.8188 - mse: 147.4738 - val_loss: 147.7298 - val_mae: 7.7628 - val_mse: 147.7298
+Epoch 158/200
+7559/7559 [==============================] - 31s 3ms/step - loss: 163.2548 - mae: 8.3126 - mse: 163.2548 - val_loss: 164.8296 - val_mae: 8.3402 - val_mse: 164.8296
```
Training is done in epochs, under every epoch all the training data is iterated and model parameters updated to minimize the loss function.
Our model doesn't look to suffer overfitting cause `loss` and `val_loss` doesn't diverge during training
-
+
While we told the training to do 200 epochs, the training stopped at 198 because we added an `early_stop` call back with `20` as `PATIENCE`, meaning that after 20 epoch and no improvement in `val_loss` the training is halted, saving time and potentially avoiding overfitting.
The following chart is probably the best visualization to evaluate the model, on the x axis there is the real fee rate while on the y axis there is the prediction, the more the points are centered on the bisection, the more the model is good.
We can see the model is doing quite well, the MAE is 8 which is way lower than `estimatesmartfee`. However, there are big errors some times, in particular for prediction for low blocks target as shown by the orange points. Creating a model only for blocks target greater than 2 instead of simply remove some observations may be an option.
-
+
The following chart is instead a distribution of the errors, which for good model should resemble the normal distribution centered in 0, and it loooks like the model is respecting that.
-
+
## Conclusion and future development
[^drift]: drift like MAE, but without the absolute value
[^minimum relay fee]: Most node won't relay transactions with fee lower than the min relay fee, which has a default of `1.0`
[^blocks target]: Conceptually similar to bitcoin core `estimatesmartfee` parameter called "blocks target", however, `confirms_in` is the real value not the desired target.
-[^fast]: 14GB of compressed raw logs are processed and a compressed CSV produced in about 4 minutes.
[Part 1]: /blog/2021/01/fee-estimation-for-light-clients-part-1/
[ZMQ]: https://github.com/bitcoin/bitcoin/blob/master/doc/zmq.md
[data logger]: https://github.com/RCasatta/bitcoin_logger
[this one]: https://blockstream.info/tx/33291156ab79e9b4a1019b618b0acfa18cbdf8fa6b71c43a9eed62a849b86f9a
-[dataset]: https://storage.googleapis.com/bitcoin_log/dataset_17.csv.gz
[google colab notebook]: https://colab.research.google.com/drive/1yamwh8nE4NhmGButep-pfUT-1uRKs49a?usp=sharing
-[plain python]: https://github.com/RCasatta/
[example]: https://www.tensorflow.org/tutorials/keras/regression
[tune hyperparameters]: https://www.tensorflow.org/tutorials/keras/keras_tuner
[advice]: https://www.tensorflow.org/tutorials/keras/overfit_and_underfit#demonstrate_overfitting
projects / bitcoindevkit.org / commitdiff
