create account

Caution When Requesting Data From Steem RPC Nodes by steemchiller

View this thread on steemit.com
· @steemchiller ·
$41.33
Caution When Requesting Data From Steem RPC Nodes
<center><a href="https://pixabay.com/de/frau-programmieren-programmierung-3597095/"><img src="https://cdn.steemitimages.com/DQmYHDjBE78RArUadjW1bDFNsb5XoGzqxfSiFrBTWoEqBF3/woman-3597095_1920.jpg"></a></center>

https://steemitimages.com/DQmXA9RBqr2qRTbWpcnvGDv38v9v6gak6WotkLMMUZxeERk/hr_thin.png

If you are a developer and you are working with data from public RPC endpoints, this might be of interest for you. I've found the reason for the duplicate account operations and posts on SteemWorld and just want to inform you about the cause for the issues here.

## Don't use the field 'id'
---

When you make a `get_content` call or any other that returns posts, the returned post-id can differ from node to node. I recommend to use the fields `author` and `permlink` for identifying posts and never rely on a (global) unique ID in the field `id`, because otherwise you will get in trouble for sure.

The same is true for loading data from the account history with `get_account_history`. The returned `id` might not always be the same. Even when using just one endpoint like `api.steemit.com`, the returned operations must not always have the same `id` and, as I saw in my tests, they can even differ from minute to minute. The reason for this will be explained below.

## How to identify one unique account operation
---

A few months ago I changed my `getAccountHistory` method on SteemWorld, so that each operation contains an `unique_id`. The only way to achieve this is to build an own ID by combining the `block`, `trx_id` or `timestamp` with the operation's content. One can't rely on the transaction details like `op_in_trx` or `trx_in_block` solely, because they can differ with different node configurations.

I'm building the ID by combining the block-number with the op-type in `op[0]` and a 32-bit hash of the operation's data in `op[1]`:

---
~~~
buildUniqueId: function(blockNum, op)
{
    let i, key, hash,
    data = {},
    keys = Object.keys( op[1] ).sort();

    for (i=0; i<keys.length; i++) {
        key = keys[i];
        data[ key ] = op[1][ key ];
    }
    hash = fastHash( JSON.stringify(data) );

    return blockNum + "-" + op[0] + "-" + hash;
}
~~~
---
Sometimes there were still duplicate operations coming up in the past weeks and the reason for this was the field order in `op[1]`, which can also differ from node to node. Therefore I'm sorting the fields by name first (as shown in the code above) and it works without any issues now.

Of course, a fast 32-bit hash may not be the right choice when storing the operations in a database, but for my purpose on SteemWorld it is good enough and does the job very well.

## Keep in mind that your requests might be routed to different endpoints
---

Some RPC node maintainers do route incoming requests to other endpoints, when they make a `replay` or change configurations on their servers, therefore you can't say for sure that the received data always come from the same node. That is why you should never make use of the field `id` to get a (global) unique ID for the received data.

Hope this helps some of you.

https://steemitimages.com/DQmXA9RBqr2qRTbWpcnvGDv38v9v6gak6WotkLMMUZxeERk/hr_thin.png

https://steemitimages.com/DQmarYCHGSBms38CSwf8Be3n56RN16nz8MS6MTSV2jX3TfT/chiller-footer-sh-400x90.png
πŸ‘  , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , and 629 others
properties (23)
post_id68,724,415
authorsteemchiller
permlinkcaution-when-requesting-data-from-steem-rpc-nodes
categorysteemworld
json_metadata{"tags":["steemworld","steemdev","dev","steem","help"],"image":["https:\/\/cdn.steemitimages.com\/DQmYHDjBE78RArUadjW1bDFNsb5XoGzqxfSiFrBTWoEqBF3\/woman-3597095_1920.jpg"],"links":["https:\/\/pixabay.com\/de\/frau-programmieren-programmierung-3597095\/"],"app":"steemit\/0.1","format":"markdown"}
created2019-01-12 15:04:30
last_update2019-01-12 15:04:30
depth0
children8
net_rshares81,705,235,554,769
last_payout2019-01-19 15:04:30
cashout_time1969-12-31 23:59:59
total_payout_value31.873 SBD
curator_payout_value9.455 SBD
pending_payout_value0.000 SBD
promoted0.000 SBD
body_length3,288
author_reputation138,038,426,460,288
root_title"Caution When Requesting Data From Steem RPC Nodes"
beneficiaries[]
max_accepted_payout1,000,000.000 SBD
percent_steem_dollars10,000
author_curate_reward""
vote details (693)
@steemprojects ·
This post has been just added as new item to _[timeline of SteemWorld on Steem Projects](https://steemprojects.com/projects/p/steemworld/?utm_source=comment_timeline&utm_medium=steem&utm_campaign=new_event&utm_content=c1)_.

If you want to be notified about new updates from this project, register on Steem Projects and add SteemWorld to your favorite projects.
properties (22)
post_id68,727,100
authorsteemprojects
permlinkre-caution-when-requesting-data-from-steem-rpc-nodes-20190112t162023
categorysteemworld
json_metadata{}
created2019-01-12 16:20:24
last_update2019-01-12 16:20:24
depth1
children0
net_rshares0
last_payout2019-01-19 16:20:24
cashout_time1969-12-31 23:59:59
total_payout_value0.000 SBD
curator_payout_value0.000 SBD
pending_payout_value0.000 SBD
promoted0.000 SBD
body_length363
author_reputation28,988,264,959
root_title"Caution When Requesting Data From Steem RPC Nodes"
beneficiaries[]
max_accepted_payout1,000,000.000 SBD
percent_steem_dollars10,000
@ennosan ·
$0.05
Danke fΓΌr den Tipp!!!!
πŸ‘  ,
properties (23)
post_id68,729,152
authorennosan
permlinkre-steemchiller-caution-when-requesting-data-from-steem-rpc-nodes-20190112t172523887z
categorysteemworld
json_metadata{"tags":["steemworld"],"app":"steemit\/0.1"}
created2019-01-12 17:25:24
last_update2019-01-12 17:25:24
depth1
children0
net_rshares106,804,726,222
last_payout2019-01-19 17:25:24
cashout_time1969-12-31 23:59:59
total_payout_value0.041 SBD
curator_payout_value0.013 SBD
pending_payout_value0.000 SBD
promoted0.000 SBD
body_length22
author_reputation4,761,872,663,008
root_title"Caution When Requesting Data From Steem RPC Nodes"
beneficiaries[]
max_accepted_payout1,000,000.000 SBD
percent_steem_dollars10,000
author_curate_reward""
vote details (2)
@geekpowered ·
$0.04
That seems like a huge bug. If a field is useless because it cannot be relied upon, then it should not exist.
πŸ‘  
properties (23)
post_id68,739,216
authorgeekpowered
permlinkre-steemchiller-caution-when-requesting-data-from-steem-rpc-nodes-20190112t232748865z
categorysteemworld
json_metadata{"tags":["steemworld"],"community":"steempeak","app":"steempeak"}
created2019-01-12 23:27:48
last_update2019-01-12 23:27:48
depth1
children2
net_rshares82,556,147,508
last_payout2019-01-19 23:27:48
cashout_time1969-12-31 23:59:59
total_payout_value0.031 SBD
curator_payout_value0.010 SBD
pending_payout_value0.000 SBD
promoted0.000 SBD
body_length109
author_reputation84,031,950,538,161
root_title"Caution When Requesting Data From Steem RPC Nodes"
beneficiaries[]
max_accepted_payout1,000,000.000 SBD
percent_steem_dollars10,000
author_curate_reward""
vote details (1)
@steemchiller ·
$0.18
It's not really a bug, because the field just contains an index for the data and the nodes can be configured with different plugins and settings. When a node does not hold all types of operations for example, the id will of course not be the same as for other nodes.

The field can only be used for local/private environments and even then I would be cautious. It's a bit irritating for developers, but I'm glad that I found a solution now :)
πŸ‘  , , ,
properties (23)
post_id68,739,736
authorsteemchiller
permlinkre-geekpowered-re-steemchiller-caution-when-requesting-data-from-steem-rpc-nodes-20190112t235120502z
categorysteemworld
json_metadata{"tags":["steemworld"],"app":"steemit\/0.1"}
created2019-01-12 23:51:21
last_update2019-01-12 23:51:21
depth2
children1
net_rshares348,261,273,039
last_payout2019-01-19 23:51:21
cashout_time1969-12-31 23:59:59
total_payout_value0.134 SBD
curator_payout_value0.043 SBD
pending_payout_value0.000 SBD
promoted0.000 SBD
body_length442
author_reputation138,038,426,460,288
root_title"Caution When Requesting Data From Steem RPC Nodes"
beneficiaries[]
max_accepted_payout1,000,000.000 SBD
percent_steem_dollars10,000
author_curate_reward""
vote details (4)
@geekpowered ·
Yes. It's a common solution for certain things...just usually not when you're asking something of a database... Very good to know for developers though.
properties (22)
post_id68,739,896
authorgeekpowered
permlinkre-steemchiller-re-geekpowered-re-steemchiller-caution-when-requesting-data-from-steem-rpc-nodes-20190112t235819190z
categorysteemworld
json_metadata{"tags":["steemworld"],"app":"steemit\/0.1"}
created2019-01-12 23:58:18
last_update2019-01-12 23:58:18
depth3
children0
net_rshares0
last_payout2019-01-19 23:58:18
cashout_time1969-12-31 23:59:59
total_payout_value0.000 SBD
curator_payout_value0.000 SBD
pending_payout_value0.000 SBD
promoted0.000 SBD
body_length152
author_reputation84,031,950,538,161
root_title"Caution When Requesting Data From Steem RPC Nodes"
beneficiaries[]
max_accepted_payout1,000,000.000 SBD
percent_steem_dollars10,000
@pennsif ·
This post has been included in the latest edition of  [**SOS Daily News**](https://steemit.com/steem/@pennsif/sosdailynewsnewsaboutthestateofsteem12january2019-65z2x3oowz) - a digest of all you need to know about the State of Steem.

***

* *Editor of the [**The State of Steem SoS Daily News**](https://steemit.com/steem/@pennsif/sosdailynewsnewsaboutthestateofsteem12january2019-65z2x3oowz).*

* *Promoter of [**The State of Steem SoS Weekly Forums**](https://steemit.com/dtube/@pennsif/k8811wa6).*

* *Editor of the [**weekly listing of steem radio shows, podcasts & social broadcasts**](https://steemit.com/mspwaves/@pennsif/schedule-of-radio-shows-podcasts-and-social-broadcasts-supported-by-dsound-week-beginning-7-january-2019).*

* *Founder of the [**A Dollar A Day**](https://steemit.com/adollaraday/@adollaraday/a-dollar-a-day-charitable-giving-project-ususd-5000-donated-in-8-months-we-made-it) charitable giving project.*

***
properties (22)
post_id68,758,600
authorpennsif
permlinkre-steemchiller-caution-when-requesting-data-from-steem-rpc-nodes-20190113t134442735z
categorysteemworld
json_metadata{"tags":["steemworld"],"links":["https:\/\/steemit.com\/steem\/@pennsif\/sosdailynewsnewsaboutthestateofsteem12january2019-65z2x3oowz","https:\/\/steemit.com\/dtube\/@pennsif\/k8811wa6","https:\/\/steemit.com\/mspwaves\/@pennsif\/schedule-of-radio-shows-podcasts-and-social-broadcasts-supported-by-dsound-week-beginning-7-january-2019","https:\/\/steemit.com\/adollaraday\/@adollaraday\/a-dollar-a-day-charitable-giving-project-ususd-5000-donated-in-8-months-we-made-it"],"app":"steemit\/0.1"}
created2019-01-13 13:44:45
last_update2019-01-13 13:44:45
depth1
children0
net_rshares0
last_payout2019-01-20 13:44:45
cashout_time1969-12-31 23:59:59
total_payout_value0.000 SBD
curator_payout_value0.000 SBD
pending_payout_value0.000 SBD
promoted0.000 SBD
body_length938
author_reputation244,218,060,434,120
root_title"Caution When Requesting Data From Steem RPC Nodes"
beneficiaries[]
max_accepted_payout1,000,000.000 SBD
percent_steem_dollars10,000
@steemdetective ·
# Hy steemchiller
 Did you know that you have 70493 transactions on steem blockchain! 
 You can check and download all your transactions, votes, comments and more with @steemdetective! 
 Start here [steemdetective.com](https://steemdetective.com)
properties (22)
post_id68,872,083
authorsteemdetective
permlinkre-caution-when-requesting-data-from-steem-rpc-nodes-20190116t074723
categorysteemworld
json_metadata{}
created2019-01-16 07:47:24
last_update2019-01-16 07:47:24
depth1
children0
net_rshares0
last_payout2019-01-23 07:47:24
cashout_time1969-12-31 23:59:59
total_payout_value0.000 SBD
curator_payout_value0.000 SBD
pending_payout_value0.000 SBD
promoted0.000 SBD
body_length246
author_reputation173,780,082,874
root_title"Caution When Requesting Data From Steem RPC Nodes"
beneficiaries[]
max_accepted_payout1,000,000.000 SBD
percent_steem_dollars10,000
@steem-ua ·
#### Hi @steemchiller!

Your post was upvoted by @steem-ua, new Steem dApp, using UserAuthority for algorithmic post curation!
Your **UA** account score is currently 6.145 which ranks you at **#263** across all Steem accounts.
Your rank has improved 3 places in the last three days (old rank 266).

In our last Algorithmic Curation Round, consisting of 249 contributions, your post is ranked at **#14**.
##### Evaluation of your UA score:

* You've built up a nice network.
* The readers appreciate your great work!
* Good user engagement!


**Feel free to join our [@steem-ua Discord server](https://discord.gg/KpBNYGz)**
properties (22)
post_id68,947,321
authorsteem-ua
permlinkre-caution-when-requesting-data-from-steem-rpc-nodes-20190118t032916z
categorysteemworld
json_metadata{"app":"beem\/0.20.14"}
created2019-01-18 03:29:18
last_update2019-01-18 03:29:18
depth1
children0
net_rshares0
last_payout2019-01-25 03:29:18
cashout_time1969-12-31 23:59:59
total_payout_value0.000 SBD
curator_payout_value0.000 SBD
pending_payout_value0.000 SBD
promoted0.000 SBD
body_length622
author_reputation21,821,715,475,664
root_title"Caution When Requesting Data From Steem RPC Nodes"
beneficiaries[]
max_accepted_payout1,000,000.000 SBD
percent_steem_dollars10,000