create account

Building a Blockchain with Go - Part 7 - The UTXO Set and BadgerDB Iterators by tensor

View this thread on steemit.com
· @tensor ·
$49.61
Building a Blockchain with Go - Part 7 - The UTXO Set and BadgerDB Iterators
![golang.jpg](https://cdn.steemitimages.com/DQmdE7SfppXmLzLBWEn63VP1pdtREsGyxU3mzfvd3ybW5TF/golang.jpg)

#### Repository
https://github.com/golang/go

#### What Will I Learn?

- You will learn about the UTXO set
- You will learn how to iterate over BadgerDB keys
- You will learn how to add perfixes to BagerDB
- You will learn about Blockchain optimization
- You will learn how to add a chainstate persistence layer 
- You will learn how to batch delete keys from BadgerDB

#### Requirements
#### System Requirements:
**Operating System:** 
- FreeBSD 10.3 or later
- Linux 2.6.23 or later with glibc
- macOS 10.10 or later
- Windows 7, Server 2008R2 or later

#### Required Knowledge
- A little understanding of the Go programming language
- Go installed on your computer
- A text editor or IDE like Gogland (VS Code used)

#### Resources for Go and this Project:
- Awesome Go Github: https://github.com/avelino/awesome-go
- Golang Installation Page: https://golang.org/dl/
- Golang Home Page: https://golang.org/
- Golang Documentation Page: https://golang.org/doc/
- Documentation about the `Big` library: https://golang.org/pkg/math/big/
- Documentation about the `Bytes` library: https://golang.org/pkg/bytes/


- Information about Elliptic Curves: http://mathworld.wolfram.com/EllipticCurve.html
- Information about Base58: https://en.bitcoinwiki.org/wiki/Base58
- BadgerDB Documentation: https://godoc.org/github.com/dgraph-io/badger
- MermaidJS: https://mermaidjs.github.io/

#### Credits/Sources: 

- Go Logo:  https://golang.org/

#### Difficulty


- Advanced 


#### Description

Blockchains are by definition immutable and distributed databases.  As we move closer and closer towards a production level application, we need to consider how we can avoid some of the pitfalls that could potentially happen from a lack of scaling.  While our testing environment has only really dealt with four to five blocks at once; each with their own transaction sets, a real blockchain can scale out to hundreds of thousands of blocks which takes up a significant portion of disk space.  Before we add the peer to peer networking layer to our application, we need to ensure that each peer doesn't have to hold and read 100-200 GBs worth of data just to verify a transaction on the blockchain.  


##### UTXO Persistance

As we've seen in these tutorials, most of the transactions in a blockchain rely on what are called Unspent Transaction Outputs.    If you want to know the balance of an account, you count up the tokens inside of the set of unspent outputs which are locked by the user's public hash key.  If you want to send tokens from one account to another, you also need to use this set of unspent outputs to create that transaction.  The unspent transaction output set is important and it makes sense to separate this layer of data from the blockchain itself.   Not all users want to run a full node to access a blockchain.  Even full nodes should not have to read the entire blockchain to make a new transaction or find an account's balance.  This becomes even more important as the blockchain scales out to hundreds of thousands of blocks.   

![utxo.png](https://cdn.steemitimages.com/DQmcM8ygBczKpwMSGebqBfsFTKgqjp4JVadSGqjLTVti457/utxo.png)

The flowchart above shows a three block blockchain.  In this example, there are three unspent transaction outputs and eight total transactions in all three blocks combined.  The blockchain persistence layer stores all of the transactions and each block that contains those transactions.  The UTXO set on the other hand only stores the Unspent Transaction Outputs to minimize the cost of having to create new transactions.  In the case of Disk space and Performance, using the UTXO for most transactions is better and is no less secure then using the blockchain itself.  This persistence layer is called the chainstate persistence layer and it is one of the major ways of optimizing and shrinking the size of a blockchain node. 

##### Differentiating Data in BadgerDB

This UTXO set needs to be stored in the database along with our blockchain.  With BadgerDB being a low level Key-Value storage solution, the main way of doing this is by using prefixes.  Badger does not contain any tables nor does it have buckets but we are able to append prefixes to the keys of like data to properly work with and modify that data in bulk.  It might seem intuitive to create a separate database for the chainstate apart from the blockchain database, but this would require adding a second instance of the BadgerDB database client to the application.  Doing this is possible but it removes any guarantees that the application will not corrupt the data in the database.  

![badgerdb-delete.png](https://cdn.steemitimages.com/DQmVWkvAvXLtDNuQy7K5AHke1VXAT2LKSVAh7AdyuGsmkKf/badgerdb-delete.png) 

In our code, we built a `DeleteByPrefix` helper function.  The primary function of this method is to iterate through our database and delete keys that are associated with a specific prefix by bulk.  The function which is featured in the code above, opens a read-only transaction with the BadgerDB database.  It then iterates through the database without fetching the values associated with the database keys.  Once its collected up to one hundred thousand keys, it executes a bulk delete command to remove both the keys and values from the database.  If there are more than one hundred thousand keys in the database, then the command deletes the keys in iterations of one hundred thousand at a time.  This is a much more efficient process then deleting each key individually as they are being read.  This function is used to reindex the UTXO set in our database as it is created and as it grows.   




The Source Code for this video may be found here: https://github.com/tensor-programming/golang-blockchain/tree/part_7
   


#### Video Tutorial

<iframe width="560" height="315" src="https://www.youtube.com/embed/eA_xQuKJfrM" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>

### Curriculum 


- [Building a Blockchain with Go - Go Modules and a Basic Blockchain - Part 1](https://steemit.com/@tensor/building-a-blockchain-with-go---go-modules-and-a-basic-blockchain---part-1)
- [Building a Blockchain with Go - Refactor and Proof of Work - Part 2](https://steemit.com/@tensor/building-a-blockchain-with-go---refactor-and-proof-of-work---part-2)
- [Building a Blockchain with Go - Persistence and Command Line - Part 3](https://steemit.com/@tensor/building-a-blockchain-with-go---persistence-and-command-line---part-3)
- [Building a Blockchain with Go - Adding Primitive Transactions - Part 4](https://steemit.com/utopian-io/@tensor/building-a-blockchain-with-go---adding-primitive-transactions---part-4)
- [Building a Blockchain with Go - Part 5 - Building a Basic Wallet Module](https://steemit.com/utopian-io/@tensor/building-a-blockchain-in-golang---part-5---building-a-basic-wallet-module)
- [Building a Blockchain with Go - Part 6 - Adding Digital Signatures](https://steemit.com/utopian-io/@tensor/building-a-blockchain-with-go---part-6---adding-digital-signatures)

#### Proof of Work Done
https://github.com/tensor-programming
πŸ‘  , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , and 177 others
properties (37)
authortensor
permlinkbuilding-a-blockchain-with-go---part-7---the-utxo-set-and-badgerdb-iterators
categoryutopian-io
json_metadata{"tags":["utopian-io","video-tutorials","steemstem","programming","blockchain"],"app":"steem-plus-app"}
last_update2018-11-11 03:41:18
created2018-11-11 03:41:18
active2018-11-14 16:46:57
last_payout2018-11-18 03:41:18
depth0
children7
net_rshares0
vote_rshares0
children_abs_rshares0
cashout_time1969-12-31 23:59:59
max_cashout_time1969-12-31 23:59:59
total_vote_weight0
reward_weight10,000
total_payout_value37.194 SBD
curator_payout_value12.420 SBD
author_rewards63,366
net_votes235
root_authortensor
root_permlinkbuilding-a-blockchain-with-go---part-7---the-utxo-set-and-badgerdb-iterators
max_accepted_payout100,000.000 SBD
percent_steem_dollars10,000
allow_repliestrue
allow_votestrue
allow_curation_rewardstrue
beneficiaries
0.
accountsteemplus-pay
weight100
1.
accountutopian.pay
weight500
root_title"Building a Blockchain with Go - Part 7 - The UTXO Set and BadgerDB Iterators"
pending_payout_value0.000 SBD
total_pending_payout_value0.000 STEEM
author_reputation58,497,990,405,174
promoted0.000 SBD
body_length0
reblogged_by[]
author_curate_reward""
vote details (241)
@steem-plus ·
SteemPlus upvote
Hi, @tensor!

You just got a **7.41%** upvote from SteemPlus!
To get higher upvotes, earn more SteemPlus Points (SPP). On your Steemit wallet, check your SPP balance and click on "How to earn SPP?" to find out all the ways to earn.
If you're not using SteemPlus yet, please check our last posts in [here](https://steemit.com/@steem-plus) to see the many ways in which SteemPlus can improve your Steem experience on Steemit and Busy.
πŸ‘  
properties (37)
authorsteem-plus
permlinkbuilding-a-blockchain-with-go---part-7---the-utxo-set-and-badgerdb-iterators---vote-steemplus
categoryutopian-io
json_metadata{}
last_update2018-11-11 06:32:09
created2018-11-11 06:32:09
active2018-11-11 06:32:09
last_payout2018-11-18 06:32:09
depth1
children0
net_rshares0
vote_rshares0
children_abs_rshares0
cashout_time1969-12-31 23:59:59
max_cashout_time1969-12-31 23:59:59
total_vote_weight0
reward_weight10,000
total_payout_value0.000 SBD
curator_payout_value0.000 SBD
author_rewards0
net_votes1
root_authortensor
root_permlinkbuilding-a-blockchain-with-go---part-7---the-utxo-set-and-badgerdb-iterators
max_accepted_payout1,000,000.000 SBD
percent_steem_dollars10,000
allow_repliestrue
allow_votestrue
allow_curation_rewardstrue
beneficiaries[]
root_title"Building a Blockchain with Go - Part 7 - The UTXO Set and BadgerDB Iterators"
pending_payout_value0.000 SBD
total_pending_payout_value0.000 STEEM
author_reputation76,702,403,490,282
promoted0.000 SBD
body_length0
reblogged_by[]
author_curate_reward""
vote details (1)
@steemstem ·
re-tensor-building-a-blockchain-with-go---part-7---the-utxo-set-and-badgerdb-iterators-20181112t005044992z
<div class='text-justify'> <div class='pull-left'> <br /> <center> <img width='125' src='https://i.postimg.cc/9FwhnG3w/steemstem_curie.png'> </center>  <br/> </div> <br /> <br /> 

 This post has been voted on by the **SteemSTEM** curation team and voting trail in collaboration with **@curie**. <br /> 
 If you appreciate the work we are doing then consider [voting](https://www.steemit.com/~witnesses) both projects for witness by selecting [**stem.witness**](https://steemconnect.com/sign/account_witness_vote?approve=1&witness=stem.witness) and [**curie**](https://steemconnect.com/sign/account_witness_vote?approve=1&witness=curie)! <br /> 
For additional information please join us on the [**SteemSTEM discord**]( https://discord.gg/BPARaqn) and to get to know the rest of the community! </div>
properties (36)
authorsteemstem
permlinkre-tensor-building-a-blockchain-with-go---part-7---the-utxo-set-and-badgerdb-iterators-20181112t005044992z
categoryutopian-io
json_metadata{"app":"bloguable-bot"}
last_update2018-11-12 00:50:48
created2018-11-12 00:50:48
active2018-11-12 00:50:48
last_payout2018-11-19 00:50:48
depth1
children0
net_rshares0
vote_rshares0
children_abs_rshares0
cashout_time1969-12-31 23:59:59
max_cashout_time1969-12-31 23:59:59
total_vote_weight0
reward_weight10,000
total_payout_value0.000 SBD
curator_payout_value0.000 SBD
author_rewards0
net_votes0
root_authortensor
root_permlinkbuilding-a-blockchain-with-go---part-7---the-utxo-set-and-badgerdb-iterators
max_accepted_payout1,000,000.000 SBD
percent_steem_dollars10,000
allow_repliestrue
allow_votestrue
allow_curation_rewardstrue
beneficiaries[]
root_title"Building a Blockchain with Go - Part 7 - The UTXO Set and BadgerDB Iterators"
pending_payout_value0.000 SBD
total_pending_payout_value0.000 STEEM
author_reputation145,549,499,195,809
promoted0.000 SBD
body_length0
reblogged_by[]
@rosatravels ·
$6.66
Hi @tensor

Thank you for another great tutorial on building a blockchain with Go.  This is already Part 7 of the series.

I can see that your instruction is very detailed.

Learners should be able to follow all the steps that you show in your flow chart.    There are so many concepts that need to learn.

Your contribution has been evaluated according to [Utopian policies and guidelines](https://join.utopian.io/guidelines), as well as a predefined set of questions pertaining to the category.

To view those questions and the relevant answers related to your post, [click here](https://review.utopian.io/result/9/112211113).

---- 
Need help? Write a ticket on https://support.utopian.io/. 
Chat with us on [Discord](https://discord.gg/uTyJkNm). 
[[utopian-moderator]](https://join.utopian.io/)
πŸ‘  , , , , , , , , , , ,
properties (37)
authorrosatravels
permlinkre-tensor-building-a-blockchain-with-go---part-7---the-utxo-set-and-badgerdb-iterators-20181112t082140028z
categoryutopian-io
json_metadata{"tags":["utopian-io"],"users":["tensor"],"links":["https://join.utopian.io/guidelines","https://review.utopian.io/result/9/112211113","https://support.utopian.io/","https://discord.gg/uTyJkNm","https://join.utopian.io/"],"app":"steemit/0.1"}
last_update2018-11-12 08:21:42
created2018-11-12 08:21:42
active2018-11-14 16:46:57
last_payout2018-11-19 08:21:42
depth1
children2
net_rshares0
vote_rshares0
children_abs_rshares0
cashout_time1969-12-31 23:59:59
max_cashout_time1969-12-31 23:59:59
total_vote_weight0
reward_weight10,000
total_payout_value5.048 SBD
curator_payout_value1.613 SBD
author_rewards8,600
net_votes12
root_authortensor
root_permlinkbuilding-a-blockchain-with-go---part-7---the-utxo-set-and-badgerdb-iterators
max_accepted_payout1,000,000.000 SBD
percent_steem_dollars10,000
allow_repliestrue
allow_votestrue
allow_curation_rewardstrue
beneficiaries[]
root_title"Building a Blockchain with Go - Part 7 - The UTXO Set and BadgerDB Iterators"
pending_payout_value0.000 SBD
total_pending_payout_value0.000 STEEM
author_reputation120,949,895,676,629
promoted0.000 SBD
body_length0
reblogged_by[]
author_curate_reward""
vote details (12)
@tensor ·
As always, thank you for moderating my content.
properties (36)
authortensor
permlinkre-rosatravels-re-tensor-building-a-blockchain-with-go---part-7---the-utxo-set-and-badgerdb-iterators-20181112t212743453z
categoryutopian-io
json_metadata{"tags":["utopian-io"],"app":"steemit/0.1"}
last_update2018-11-12 21:27:42
created2018-11-12 21:27:42
active2018-11-12 21:27:42
last_payout2018-11-19 21:27:42
depth2
children0
net_rshares0
vote_rshares0
children_abs_rshares0
cashout_time1969-12-31 23:59:59
max_cashout_time1969-12-31 23:59:59
total_vote_weight0
reward_weight10,000
total_payout_value0.000 SBD
curator_payout_value0.000 SBD
author_rewards0
net_votes0
root_authortensor
root_permlinkbuilding-a-blockchain-with-go---part-7---the-utxo-set-and-badgerdb-iterators
max_accepted_payout1,000,000.000 SBD
percent_steem_dollars10,000
allow_repliestrue
allow_votestrue
allow_curation_rewardstrue
beneficiaries[]
root_title"Building a Blockchain with Go - Part 7 - The UTXO Set and BadgerDB Iterators"
pending_payout_value0.000 SBD
total_pending_payout_value0.000 STEEM
author_reputation58,497,990,405,174
promoted0.000 SBD
body_length0
reblogged_by[]
@utopian-io ·
Thank you for your review, @rosatravels! Keep up the good work!
properties (36)
authorutopian-io
permlinkre-re-tensor-building-a-blockchain-with-go---part-7---the-utxo-set-and-badgerdb-iterators-20181112t082140028z-20181114t164653z
categoryutopian-io
json_metadata"{"app": "beem/0.20.9"}"
last_update2018-11-14 16:46:57
created2018-11-14 16:46:57
active2018-11-14 16:46:57
last_payout2018-11-21 16:46:57
depth2
children0
net_rshares0
vote_rshares0
children_abs_rshares0
cashout_time1969-12-31 23:59:59
max_cashout_time1969-12-31 23:59:59
total_vote_weight0
reward_weight10,000
total_payout_value0.000 SBD
curator_payout_value0.000 SBD
author_rewards0
net_votes0
root_authortensor
root_permlinkbuilding-a-blockchain-with-go---part-7---the-utxo-set-and-badgerdb-iterators
max_accepted_payout1,000,000.000 SBD
percent_steem_dollars10,000
allow_repliestrue
allow_votestrue
allow_curation_rewardstrue
beneficiaries[]
root_title"Building a Blockchain with Go - Part 7 - The UTXO Set and BadgerDB Iterators"
pending_payout_value0.000 SBD
total_pending_payout_value0.000 STEEM
author_reputation114,059,639,292,732
promoted0.000 SBD
body_length0
reblogged_by[]
@steem-ua ·
#### Hi @tensor!

Your post was upvoted by @steem-ua, new Steem dApp, using UserAuthority for algorithmic post curation!
Your post is eligible for our upvote, thanks to our collaboration with @utopian-io!
**Feel free to join our [@steem-ua Discord server](https://discord.gg/KpBNYGz)**
properties (36)
authorsteem-ua
permlinkre-building-a-blockchain-with-go---part-7---the-utxo-set-and-badgerdb-iterators-20181112t084226z
categoryutopian-io
json_metadata"{"app": "beem/0.20.9"}"
last_update2018-11-12 08:42:27
created2018-11-12 08:42:27
active2018-11-12 08:42:27
last_payout2018-11-19 08:42:27
depth1
children0
net_rshares0
vote_rshares0
children_abs_rshares0
cashout_time1969-12-31 23:59:59
max_cashout_time1969-12-31 23:59:59
total_vote_weight0
reward_weight10,000
total_payout_value0.000 SBD
curator_payout_value0.000 SBD
author_rewards0
net_votes0
root_authortensor
root_permlinkbuilding-a-blockchain-with-go---part-7---the-utxo-set-and-badgerdb-iterators
max_accepted_payout1,000,000.000 SBD
percent_steem_dollars10,000
allow_repliestrue
allow_votestrue
allow_curation_rewardstrue
beneficiaries[]
root_title"Building a Blockchain with Go - Part 7 - The UTXO Set and BadgerDB Iterators"
pending_payout_value0.000 SBD
total_pending_payout_value0.000 STEEM
author_reputation18,855,903,458,602
promoted0.000 SBD
body_length0
reblogged_by[]
@utopian-io ·
Hey, @tensor!

**Thanks for contributing on Utopian**.
We’re already looking forward to your next contribution!

**Get higher incentives and support Utopian.io!**
 Simply set @utopian.pay as a 5% (or higher) payout beneficiary on your contribution post (via [SteemPlus](https://chrome.google.com/webstore/detail/steemplus/mjbkjgcplmaneajhcbegoffkedeankaj?hl=en) or [Steeditor](https://steeditor.app)).

**Want to chat? Join us on Discord https://discord.gg/h52nFrV.**

<a href='https://steemconnect.com/sign/account-witness-vote?witness=utopian-io&approve=1'>Vote for Utopian Witness!</a>
properties (36)
authorutopian-io
permlinkre-building-a-blockchain-with-go---part-7---the-utxo-set-and-badgerdb-iterators-20181112t150411z
categoryutopian-io
json_metadata"{"app": "beem/0.20.9"}"
last_update2018-11-12 15:04:12
created2018-11-12 15:04:12
active2018-11-12 15:04:12
last_payout2018-11-19 15:04:12
depth1
children0
net_rshares0
vote_rshares0
children_abs_rshares0
cashout_time1969-12-31 23:59:59
max_cashout_time1969-12-31 23:59:59
total_vote_weight0
reward_weight10,000
total_payout_value0.000 SBD
curator_payout_value0.000 SBD
author_rewards0
net_votes0
root_authortensor
root_permlinkbuilding-a-blockchain-with-go---part-7---the-utxo-set-and-badgerdb-iterators
max_accepted_payout1,000,000.000 SBD
percent_steem_dollars10,000
allow_repliestrue
allow_votestrue
allow_curation_rewardstrue
beneficiaries[]
root_title"Building a Blockchain with Go - Part 7 - The UTXO Set and BadgerDB Iterators"
pending_payout_value0.000 SBD
total_pending_payout_value0.000 STEEM
author_reputation114,059,639,292,732
promoted0.000 SBD
body_length0
reblogged_by[]