From cc8c843c737c2c9938e3da8bd8122019ad0193d6 Mon Sep 17 00:00:00 2001 From: Riccardo Casatta Date: Mon, 25 Jan 2021 14:48:21 +0100 Subject: [PATCH] fix link, move assets to subfolder --- .../2021/fee_estimation_for_light_clients_part_1.md | 2 +- .../2021/fee_estimation_for_light_clients_part_2.md | 2 +- .../2021/fee_estimation_for_light_clients_part_3.md | 12 ++++++------ .../20210125-091313-confirms_in-fee_rate.png | Bin .../20210125-091313-error-distribution.png | Bin .../20210125-091313-model.png | Bin .../20210125-091313-train-history.png | Bin .../20210125-091313-true-and-predictions.png | Bin .../high-level-graph.svg | 0 .../the-good-the-bad-the-ugly.jpg | Bin 10 files changed, 8 insertions(+), 8 deletions(-) rename static/images/{ => fee-estimation-for-light-clients}/20210125-091313-confirms_in-fee_rate.png (100%) rename static/images/{ => fee-estimation-for-light-clients}/20210125-091313-error-distribution.png (100%) rename static/images/{ => fee-estimation-for-light-clients}/20210125-091313-model.png (100%) rename static/images/{ => fee-estimation-for-light-clients}/20210125-091313-train-history.png (100%) rename static/images/{ => fee-estimation-for-light-clients}/20210125-091313-true-and-predictions.png (100%) rename static/images/{ => fee-estimation-for-light-clients}/high-level-graph.svg (100%) rename static/images/{ => fee-estimation-for-light-clients}/the-good-the-bad-the-ugly.jpg (100%) diff --git a/content/blog/2021/fee_estimation_for_light_clients_part_1.md b/content/blog/2021/fee_estimation_for_light_clients_part_1.md index b569818fc7..b5cd4a1339 100644 --- a/content/blog/2021/fee_estimation_for_light_clients_part_1.md +++ b/content/blog/2021/fee_estimation_for_light_clients_part_1.md @@ -97,7 +97,7 @@ For these reasons, the logger is split into two parts: a process listening to th Raw logs are self-contained: for example, they contain all the previous transaction output values for every relevant transaction. This causes some redundancy, but in this case it's better to trade some efficiency for more performance when recreating the dataset. -![High level graph](/images/high-level-graph.svg) +![High level graph](/images/fee-estimation-for-light-clients/high-level-graph.svg) My logger instance started collecting data on the 18th of December 2020, and as of today (25th January 2020), the raw logs are about 16GB. diff --git a/content/blog/2021/fee_estimation_for_light_clients_part_2.md b/content/blog/2021/fee_estimation_for_light_clients_part_2.md index 21f931d093..8dfefb9073 100644 --- a/content/blog/2021/fee_estimation_for_light_clients_part_2.md +++ b/content/blog/2021/fee_estimation_for_light_clients_part_2.md @@ -103,7 +103,7 @@ q1-q30-... | no | Transaction confirming fast could be outliers, usually paying a1-a2-... | yes | Contains the number of transaction in the mempool with known fee rate in the ith bucket. -![The good, the bad and the ugly](/images/the-good-the-bad-the-ugly.jpg) +![The good, the bad and the ugly](/images/fee-estimation-for-light-clients/the-good-the-bad-the-ugly.jpg)
My biological neural network fired this, I think it's because a lot of chapters start with "The"


diff --git a/content/blog/2021/fee_estimation_for_light_clients_part_3.md b/content/blog/2021/fee_estimation_for_light_clients_part_3.md index 579f3c8c42..f9dfdc8bd4 100644 --- a/content/blog/2021/fee_estimation_for_light_clients_part_3.md +++ b/content/blog/2021/fee_estimation_for_light_clients_part_3.md @@ -23,7 +23,7 @@ This post is part 2 of 3 of a series. ([Part 1], [Part 2]) The code building and training the model with [tensorflow] is available in [google colab notebook] (jupyter notebook); you can also download the file as plain python and run it locally. At least 1 hour is needed to train the full model, but it heavily depends on the hardware available. -![graph confirm_in blocks vs fee_rate](/images/20210125-091313-confirms_in-fee_rate.png) +![graph confirm_in blocks vs fee_rate](/images/fee-estimation-for-light-clients/20210125-091313-confirms_in-fee_rate.png)
Do you want to choose the fee without a model? In the last 5 weeks a ~50 sat/vbyte transaction never took more than a day to confirm and a ~10 sat/vbyte never took more than a week

As a reference, in the code we have a calculation of the bitcoin core `estimatesmartfee` MAE[^MAE] and drift[^drift]. @@ -72,7 +72,7 @@ model.compile(loss='mse', metrics=['mae', 'mse']) ``` -![model graph](/images/20210125-091313-model.png) +![model graph](/images/fee-estimation-for-light-clients/20210125-091313-model.png) The model is fed with the `encoded_features` coming from the processing phase, then there are 2 layers with 64 neurons each followed by one neuron giving the `fee_rate` as output. @@ -143,7 +143,7 @@ The value `loss` is the MSE on the training data while `val_loss` is the MSE val Our model doesn't look to suffer overfitting cause `loss` and `val_loss` doesn't diverge during training -![train history](/images/20210125-091313-train-history.png) +![train history](/images/fee-estimation-for-light-clients/20210125-091313-train-history.png) While we told the training to do 200 epochs, the training stopped at 158 because we added an `early_stop` call back with `20` as `PATIENCE`, meaning that after 20 epoch and no improvement in `val_loss` the training is halted, saving time and potentially avoiding overfitting. @@ -154,11 +154,11 @@ A [prediction test tool] is available on github. At the moment it uses a bitcoin The following chart is probably the best visualization to evaluate the model, on the x axis there is the real fee rate while on the y axis there is the prediction, the more the points are centered on the bisection, the more the model is good. We can see the model is doing quite well, the MAE is 8 which is way lower than `estimatesmartfee`. However, there are big errors some times, in particular for prediction for fast confirmation (`confirms_in=1 or confirms_in=2`) as shown by the orange points. Creating a model only for blocks target greater than 2 instead of simply remove some observations may be an option. -![prediction results](/images/20210125-091313-true-and-predictions.png) +![prediction results](/images/fee-estimation-for-light-clients/20210125-091313-true-and-predictions.png) The following chart is instead a distribution of the errors, which for good model should resemble the normal distribution centered in 0, and it loooks like the model is respecting that. -![error distribution](/images/20210125-091313-error-distribution.png) +![error distribution](/images/fee-estimation-for-light-clients/20210125-091313-error-distribution.png) ## Conclusion and future development @@ -192,7 +192,7 @@ And also this tweet that remembered me [I] had this work in my TODO list

I don't understand Machine Learning(ML), but is it horrible to use ML to predict bitcoin fees?

I have heard tales of this "Deep Learning" thing where you throw a bunch of data at it and it gives you good results with high accuracy.

— sanket1729 (@sanket1729) December 9, 2020
-This is the final part of the series. In the previous [Part 1] we talked about the problem and in [Part 3] we talked about the dataset. +This is the final part of the series. In the previous [Part 1] we talked about the problem and in [Part 2] we talked about the dataset. [^MAE]: MAE is Mean Absolute Error, which is the average of the series built by the absolute difference between the real value and the estimation. [^drift]: drift like MAE, but without the absolute value diff --git a/static/images/20210125-091313-confirms_in-fee_rate.png b/static/images/fee-estimation-for-light-clients/20210125-091313-confirms_in-fee_rate.png similarity index 100% rename from static/images/20210125-091313-confirms_in-fee_rate.png rename to static/images/fee-estimation-for-light-clients/20210125-091313-confirms_in-fee_rate.png diff --git a/static/images/20210125-091313-error-distribution.png b/static/images/fee-estimation-for-light-clients/20210125-091313-error-distribution.png similarity index 100% rename from static/images/20210125-091313-error-distribution.png rename to static/images/fee-estimation-for-light-clients/20210125-091313-error-distribution.png diff --git a/static/images/20210125-091313-model.png b/static/images/fee-estimation-for-light-clients/20210125-091313-model.png similarity index 100% rename from static/images/20210125-091313-model.png rename to static/images/fee-estimation-for-light-clients/20210125-091313-model.png diff --git a/static/images/20210125-091313-train-history.png b/static/images/fee-estimation-for-light-clients/20210125-091313-train-history.png similarity index 100% rename from static/images/20210125-091313-train-history.png rename to static/images/fee-estimation-for-light-clients/20210125-091313-train-history.png diff --git a/static/images/20210125-091313-true-and-predictions.png b/static/images/fee-estimation-for-light-clients/20210125-091313-true-and-predictions.png similarity index 100% rename from static/images/20210125-091313-true-and-predictions.png rename to static/images/fee-estimation-for-light-clients/20210125-091313-true-and-predictions.png diff --git a/static/images/high-level-graph.svg b/static/images/fee-estimation-for-light-clients/high-level-graph.svg similarity index 100% rename from static/images/high-level-graph.svg rename to static/images/fee-estimation-for-light-clients/high-level-graph.svg diff --git a/static/images/the-good-the-bad-the-ugly.jpg b/static/images/fee-estimation-for-light-clients/the-good-the-bad-the-ugly.jpg similarity index 100% rename from static/images/the-good-the-bad-the-ugly.jpg rename to static/images/fee-estimation-for-light-clients/the-good-the-bad-the-ugly.jpg -- 2.49.0