From: Riccardo Casatta Date: Mon, 25 Jan 2021 13:48:21 +0000 (+0100) Subject: fix link, move assets to subfolder X-Git-Url: http://internal-gitweb-vhost/%22https:/parse/scripts/database/-script/struct.CommandStringError.html?a=commitdiff_plain;h=cc8c843c737c2c9938e3da8bd8122019ad0193d6;p=bitcoindevkit.org fix link, move assets to subfolder --- diff --git a/content/blog/2021/fee_estimation_for_light_clients_part_1.md b/content/blog/2021/fee_estimation_for_light_clients_part_1.md index b569818fc7..b5cd4a1339 100644 --- a/content/blog/2021/fee_estimation_for_light_clients_part_1.md +++ b/content/blog/2021/fee_estimation_for_light_clients_part_1.md @@ -97,7 +97,7 @@ For these reasons, the logger is split into two parts: a process listening to th Raw logs are self-contained: for example, they contain all the previous transaction output values for every relevant transaction. This causes some redundancy, but in this case it's better to trade some efficiency for more performance when recreating the dataset. -![High level graph](/images/high-level-graph.svg) +![High level graph](/images/fee-estimation-for-light-clients/high-level-graph.svg) My logger instance started collecting data on the 18th of December 2020, and as of today (25th January 2020), the raw logs are about 16GB. diff --git a/content/blog/2021/fee_estimation_for_light_clients_part_2.md b/content/blog/2021/fee_estimation_for_light_clients_part_2.md index 21f931d093..8dfefb9073 100644 --- a/content/blog/2021/fee_estimation_for_light_clients_part_2.md +++ b/content/blog/2021/fee_estimation_for_light_clients_part_2.md @@ -103,7 +103,7 @@ q1-q30-... | no | Transaction confirming fast could be outliers, usually paying a1-a2-... | yes | Contains the number of transaction in the mempool with known fee rate in the ith bucket. -![The good, the bad and the ugly](/images/the-good-the-bad-the-ugly.jpg) +![The good, the bad and the ugly](/images/fee-estimation-for-light-clients/the-good-the-bad-the-ugly.jpg)
My biological neural network fired this, I think it's because a lot of chapters start with "The"


diff --git a/content/blog/2021/fee_estimation_for_light_clients_part_3.md b/content/blog/2021/fee_estimation_for_light_clients_part_3.md index 579f3c8c42..f9dfdc8bd4 100644 --- a/content/blog/2021/fee_estimation_for_light_clients_part_3.md +++ b/content/blog/2021/fee_estimation_for_light_clients_part_3.md @@ -23,7 +23,7 @@ This post is part 2 of 3 of a series. ([Part 1], [Part 2]) The code building and training the model with [tensorflow] is available in [google colab notebook] (jupyter notebook); you can also download the file as plain python and run it locally. At least 1 hour is needed to train the full model, but it heavily depends on the hardware available. -![graph confirm_in blocks vs fee_rate](/images/20210125-091313-confirms_in-fee_rate.png) +![graph confirm_in blocks vs fee_rate](/images/fee-estimation-for-light-clients/20210125-091313-confirms_in-fee_rate.png)
Do you want to choose the fee without a model? In the last 5 weeks a ~50 sat/vbyte transaction never took more than a day to confirm and a ~10 sat/vbyte never took more than a week

As a reference, in the code we have a calculation of the bitcoin core `estimatesmartfee` MAE[^MAE] and drift[^drift]. @@ -72,7 +72,7 @@ model.compile(loss='mse', metrics=['mae', 'mse']) ``` -![model graph](/images/20210125-091313-model.png) +![model graph](/images/fee-estimation-for-light-clients/20210125-091313-model.png) The model is fed with the `encoded_features` coming from the processing phase, then there are 2 layers with 64 neurons each followed by one neuron giving the `fee_rate` as output. @@ -143,7 +143,7 @@ The value `loss` is the MSE on the training data while `val_loss` is the MSE val Our model doesn't look to suffer overfitting cause `loss` and `val_loss` doesn't diverge during training -![train history](/images/20210125-091313-train-history.png) +![train history](/images/fee-estimation-for-light-clients/20210125-091313-train-history.png) While we told the training to do 200 epochs, the training stopped at 158 because we added an `early_stop` call back with `20` as `PATIENCE`, meaning that after 20 epoch and no improvement in `val_loss` the training is halted, saving time and potentially avoiding overfitting. @@ -154,11 +154,11 @@ A [prediction test tool] is available on github. At the moment it uses a bitcoin The following chart is probably the best visualization to evaluate the model, on the x axis there is the real fee rate while on the y axis there is the prediction, the more the points are centered on the bisection, the more the model is good. We can see the model is doing quite well, the MAE is 8 which is way lower than `estimatesmartfee`. However, there are big errors some times, in particular for prediction for fast confirmation (`confirms_in=1 or confirms_in=2`) as shown by the orange points. Creating a model only for blocks target greater than 2 instead of simply remove some observations may be an option. -![prediction results](/images/20210125-091313-true-and-predictions.png) +![prediction results](/images/fee-estimation-for-light-clients/20210125-091313-true-and-predictions.png) The following chart is instead a distribution of the errors, which for good model should resemble the normal distribution centered in 0, and it loooks like the model is respecting that. -![error distribution](/images/20210125-091313-error-distribution.png) +![error distribution](/images/fee-estimation-for-light-clients/20210125-091313-error-distribution.png) ## Conclusion and future development @@ -192,7 +192,7 @@ And also this tweet that remembered me [I] had this work in my TODO list

I don't understand Machine Learning(ML), but is it horrible to use ML to predict bitcoin fees?

I have heard tales of this "Deep Learning" thing where you throw a bunch of data at it and it gives you good results with high accuracy.

— sanket1729 (@sanket1729) December 9, 2020
-This is the final part of the series. In the previous [Part 1] we talked about the problem and in [Part 3] we talked about the dataset. +This is the final part of the series. In the previous [Part 1] we talked about the problem and in [Part 2] we talked about the dataset. [^MAE]: MAE is Mean Absolute Error, which is the average of the series built by the absolute difference between the real value and the estimation. [^drift]: drift like MAE, but without the absolute value diff --git a/static/images/20210125-091313-confirms_in-fee_rate.png b/static/images/20210125-091313-confirms_in-fee_rate.png deleted file mode 100644 index e3d433683b..0000000000 Binary files a/static/images/20210125-091313-confirms_in-fee_rate.png and /dev/null differ diff --git a/static/images/20210125-091313-error-distribution.png b/static/images/20210125-091313-error-distribution.png deleted file mode 100644 index f889730994..0000000000 Binary files a/static/images/20210125-091313-error-distribution.png and /dev/null differ diff --git a/static/images/20210125-091313-model.png b/static/images/20210125-091313-model.png deleted file mode 100644 index 47098c458b..0000000000 Binary files a/static/images/20210125-091313-model.png and /dev/null differ diff --git a/static/images/20210125-091313-train-history.png b/static/images/20210125-091313-train-history.png deleted file mode 100644 index c2af8cb6bd..0000000000 Binary files a/static/images/20210125-091313-train-history.png and /dev/null differ diff --git a/static/images/20210125-091313-true-and-predictions.png b/static/images/20210125-091313-true-and-predictions.png deleted file mode 100644 index 49f351b9bc..0000000000 Binary files a/static/images/20210125-091313-true-and-predictions.png and /dev/null differ diff --git a/static/images/fee-estimation-for-light-clients/20210125-091313-confirms_in-fee_rate.png b/static/images/fee-estimation-for-light-clients/20210125-091313-confirms_in-fee_rate.png new file mode 100644 index 0000000000..e3d433683b Binary files /dev/null and b/static/images/fee-estimation-for-light-clients/20210125-091313-confirms_in-fee_rate.png differ diff --git a/static/images/fee-estimation-for-light-clients/20210125-091313-error-distribution.png b/static/images/fee-estimation-for-light-clients/20210125-091313-error-distribution.png new file mode 100644 index 0000000000..f889730994 Binary files /dev/null and b/static/images/fee-estimation-for-light-clients/20210125-091313-error-distribution.png differ diff --git a/static/images/fee-estimation-for-light-clients/20210125-091313-model.png b/static/images/fee-estimation-for-light-clients/20210125-091313-model.png new file mode 100644 index 0000000000..47098c458b Binary files /dev/null and b/static/images/fee-estimation-for-light-clients/20210125-091313-model.png differ diff --git a/static/images/fee-estimation-for-light-clients/20210125-091313-train-history.png b/static/images/fee-estimation-for-light-clients/20210125-091313-train-history.png new file mode 100644 index 0000000000..c2af8cb6bd Binary files /dev/null and b/static/images/fee-estimation-for-light-clients/20210125-091313-train-history.png differ diff --git a/static/images/fee-estimation-for-light-clients/20210125-091313-true-and-predictions.png b/static/images/fee-estimation-for-light-clients/20210125-091313-true-and-predictions.png new file mode 100644 index 0000000000..49f351b9bc Binary files /dev/null and b/static/images/fee-estimation-for-light-clients/20210125-091313-true-and-predictions.png differ diff --git a/static/images/fee-estimation-for-light-clients/high-level-graph.svg b/static/images/fee-estimation-for-light-clients/high-level-graph.svg new file mode 100644 index 0000000000..23e156a17c --- /dev/null +++ b/static/images/fee-estimation-for-light-clients/high-level-graph.svg @@ -0,0 +1,139 @@ + + +G + + +cluster_logger + +bitcoin-logger + + +cluster_bitcoind + +bitcoind + + +cluster_csv + +bitcoin-csv + + + +store + +store + + + +flush + +flush + + + +store->flush + + + + + +rpc + +rpc + + + +store->rpc + + +missing data + + + +zmq_reader + +zmq_reader + + + +zmq_reader->store + + + + + +raw_logs + +raw_logs + + + +flush->raw_logs + + + + + +rpc_call + +rpc_call + + + +rpc_call->store + + + + + +rpc_call->rpc + + +estimatesmartfee + + + +zmq + +zmq + + + +zmq->zmq_reader + + +rawtx, rawblock, sequence + + + +raw_logs_reader + +raw_logs_reader + + + +csv_writer + +csv_writer + + + +csv + +csv + + + +csv_writer->csv + + + + + +raw_logs->raw_logs_reader + + + + + \ No newline at end of file diff --git a/static/images/fee-estimation-for-light-clients/the-good-the-bad-the-ugly.jpg b/static/images/fee-estimation-for-light-clients/the-good-the-bad-the-ugly.jpg new file mode 100644 index 0000000000..86e0ba0708 Binary files /dev/null and b/static/images/fee-estimation-for-light-clients/the-good-the-bad-the-ugly.jpg differ diff --git a/static/images/high-level-graph.svg b/static/images/high-level-graph.svg deleted file mode 100644 index 23e156a17c..0000000000 --- a/static/images/high-level-graph.svg +++ /dev/null @@ -1,139 +0,0 @@ - - -G - - -cluster_logger - -bitcoin-logger - - -cluster_bitcoind - -bitcoind - - -cluster_csv - -bitcoin-csv - - - -store - -store - - - -flush - -flush - - - -store->flush - - - - - -rpc - -rpc - - - -store->rpc - - -missing data - - - -zmq_reader - -zmq_reader - - - -zmq_reader->store - - - - - -raw_logs - -raw_logs - - - -flush->raw_logs - - - - - -rpc_call - -rpc_call - - - -rpc_call->store - - - - - -rpc_call->rpc - - -estimatesmartfee - - - -zmq - -zmq - - - -zmq->zmq_reader - - -rawtx, rawblock, sequence - - - -raw_logs_reader - -raw_logs_reader - - - -csv_writer - -csv_writer - - - -csv - -csv - - - -csv_writer->csv - - - - - -raw_logs->raw_logs_reader - - - - - \ No newline at end of file diff --git a/static/images/the-good-the-bad-the-ugly.jpg b/static/images/the-good-the-bad-the-ugly.jpg deleted file mode 100644 index 86e0ba0708..0000000000 Binary files a/static/images/the-good-the-bad-the-ugly.jpg and /dev/null differ