#### Repository
[https://github.com/pibara-utopian/asyncsteem](https://github.com/pibara-utopian/asyncsteem)
#### What Will I Learn?
* How to integrate micro-transaction based steemit user authentication into your **asyncsteem**/Twisted-based web app.
* How to chain multiple STEEM JSON-RPC operations asynchronously with **asyncsteem**
* Why and how to avoid synchronous operations when interacting with a database or template engine from a twisted web app.
#### Requirements
* Experience with Python
* A base understanding of the STEEM blockchain
* A base understanding of the HTTP protocol and cookies.
* Having read and understood [Writing an asynchronous STEEM web-app in Python with asyncsteem](https://steemit.com/utopian-io/@mattockfs/writing-an-asynchronous-steem-web-app-in-python-with-asyncsteem)
#### Difficulty Vs
* Advanced
#### Tutorial Contents
Where in the [previous asyncsteem tutorial](https://steemit.com/utopian-io/@mattockfs/writing-an-asynchronous-steem-web-app-in-python-with-asyncsteem) we learned how to build a web interface for visualizing what is happening on the STEEM blockchain in real time using asyncsteem, Twisted-web and VisJs, in this tutorial we are going to create a slightly more real-life web application with Twisted-web and Asyncsteem. We will use both the high-level blockchain event streaming API and the lower level rpcclient API that asyncsteem offers, and we are going to add in some more asynchonous operations by adding a bit of Redis to the mix. The application we will build in this tutorial is based on the ideas behind [this script](https://steemit.com/steem/@tcpolymath/a-simple-python-script-to-find-inactive-follows) by @tcpolymath that finds inactive follows, combined with the core concept described in [this post](https://steemit.com/authentication/@mattockfs/concepts-for-steemconnect-alternative-for-authentication-but-not-authorization) of using a cookie and a micro transaction as alternative for using SteemConnect for authentication of your steemit account with third-party services.
#### SteemConnect vs microtransactions
There is a diference between third-party services that require to know who you are and services that require to be able to act with part of your authority. The [SteemConnect](https://steemconnect.com/) service provides facilities for the later. Think of steemconnect as a trusted third party between the user and the service provider when it comes to delegating part of the authority your account comes with. This can be very useful if for example if you write a service that votes or posts on behalf of the user. If there are twenty people like you running such services, the user doesn't need to trust his or her posting key to each of these services, but could simply delegate them to steemconnect, to then, using a delegate-by-proxy pattern, delegate a revocable and possibly attenuated part of the authority of his/her posting key to your service. Do this times ten, and it makes a lot of sense the user would rather delegate once to steemconnect than to ten different services.
This is all nice and dandy if the service you are running does indeed need delegated authority from its users. But what if it doesn't? What if the user has zero interest in services that require him or her to delegate authority to it, but would like to use services that merely wont to know for sure who he or she is?
In such a scenario, steemconnect is, I would argue, a horrible solution. The user must delegate authority to steemconnect that no service actually requires. To any security conscience steemit user, this should come as a complete horror. There are two relatively recent examples where users were asked to use steemconnect in situations where it would seem all the service needed was for the user to proof his.her identity yet ended up using requiring the user to delegate authority to steemconnect.
* The ByteBall airdrop
* The @steem-ua *View your UA* functionality.
Both services as it seems only need(ed) to know for sure that you as a user are in fact the owner of a given account and both services thought it justified to for that reason demand you as a user should delegate a large chunk of your authority to steemconnect.
Some people may think steemconnect, in fact, is the only option third party service providers have to authenticate steemit users, but that is incorrect. There is a relatively simple way to authenticate you as the owner of your account, and that is using a microtransaction. You are the only one who can initiate a payment from your account, and with some clever use of the memo field, it is almost trivially easy to set up authentication for a third party service using a 0.001 SBD or 0.001 STEEM microtransaction.
#### The four parts of the puzzle.
There are five parts to using microtransactions as a way to authenticate the users of your service:
1) Generating an unguessable random session id.
2) Using a private secret and a keyed hashing algorithm to create a session-id signature
3) Using HTTP Cookies to bind a signed session-id to a browser session
4) Using a microtransaction to bind the unsigned session-id to a steemit account.
Let's look at the first two steps, generating a session id:
```python
#Blake2 keyed hashing for cookie signing
from pyblake2 import blake2b
#Random numbers for unique cookie id
from random import randint
class CookieUtil:
def __init__(self,secret):
#We use a secret for BLAKE2 signing of our cookies
self.secret = secret.encode("ascii")
def new_cookie(self):
#Take a large random number
random_buffer = hex(randint(0,1000000000000000000000000000000000000000000))[2:-1]
#Create a signature for that number using our secret
signature = blake2b(data=random_buffer,key=self.secret,digest_size=48).hexdigest()
#Make a cookie string from the random number and its signature. Noone can make a valid cookie but us.
random_string = str(random_buffer).encode("ascii")
cookie = random_string + "-" + signature
return cookie
def valid_cookie(self,cookie):
if cookie == None:
return False
#Split up the cookie into the original random number and its signature.
[cookie_id,signature1] = cookie.split("-");
cookie_buffer = cookie_id.encode("ascii");
#Calculate the signuture of the random bit again.
signature2 = blake2b(data=cookie_buffer,key=self.secret,digest_size=48).hexdigest()
#The existing signature should be the same as the new one.
ok = (signature1 == signature2)
return ok
```
We use the randint function from the Python random module to generate a large random integer that we then convert to a hex string. As keyed hashing function we use BLAKE2B, the 64 bit variant of the fast and secure [BLAKE2]( https://blake2.net/) hashing algoritm. We could alternatively have used HMAC with SHA2, but as BLAKE2 komes with keyed usage as a build in option, BLAKE2 seems like the perfect option for our goal. We than concattenate the random string and the signature together with a hyphen character as splitter.
In the **valid_cookie** method, we simply chop up our cookie value into its two parts again and calculate the keyes BLAKE2 hash once more. If the calculated hash equates the one we found in the cookie, then our cookie string is OK.
Now we move on to step three, the actual HTTP Cookie. To do this we make a trivial Twisted Web based mini web server.
#### A simple Twisted Web web server
```python
#Jinja2 templates for HTML output.
from jinja2 import Environment, FileSystemLoader
#Some base twisted imports
from twisted.internet import reactor, endpoints
from twisted.logger import Logger, textFileLogObserver
from twisted.web import server,resource
from twisted.web.static import File
from twisted.web.util import redirectTo
#some basic imports
from os.path import join, dirname, realpath
#Simple twisted web resource that sets cookie if needed and redirects to functionality page.
class SetCookieIfNeeded(resource.Resource):
isLeaf = True
def __init__(self,cu):
self.cu = cu
def render_GET(self, request):
oldcookie = request.getCookie("steemauth")
#Only create a fresh new cookie if non was already set.
if oldcookie == None:
newcookie = self.cu.new_cookie()
request.addCookie("steemauth",newcookie)
#Redirect to service.
return redirectTo("/steemauth",request)
#Our base resource and authentication check.
class SteemAuth(resource.Resource):
isLeaf = True
def __init__(self,cu, account,templates,reactor,log):
self.cu = cu
self.account = account
self.templates = templates
self.reactor = reactor
self.log = log
def render_GET(self, request):
#If we are here, there should be a cookie
oldcookie = request.getCookie("steemauth")
#Check if the cookie is actually a valid one we created ourselv
if cu.valid_cookie(oldcookie):
#Take just the cookie id from the cookie.
cookie_id = oldcookie.split("-")[0]
rv = self.templates["askauth"].render(session=cookie_id)
return rv.encode("ascii");
else:
newcookie = self.cu.new_cookie()
request.addCookie("steemauth",newcookie)
return redirectTo("/steemauth",request)
#Our mini authentication server with some basic helo world functionality.
class MiniAuthWebServer(resource.Resource):
def __init__(self,cu, account,reactor,log):
self.cu = cu
self.children = ["/","/steemauth"]
self.account = account
self.reactor = reactor
self.log = log
thisdir = os.path.dirname(os.path.abspath(__file__))
#We do our synchonous IO for loading Jinja2 templates at server start.
j2_env = Environment(loader=FileSystemLoader(thisdir),trim_blocks=True)
self.templates = dict();
self.templates["askauth"] = j2_env.get_template('authenticate.tmpl')
self.templates["helloworld"] = j2_env.get_template('hello-world.tmpl')
self.templates["error"] = j2_env.get_template('error.tmpl')
def getChild(self, name, request):
if request.uri == '/steemauth':
return SteemAuth(cu, self.account,self.templates,self.reactor, self.log)
else:
if request.uri == '/':
return SetCookieIfNeeded(self.cu);
else:
return resource.NoResource()
#Get config with unique unguessable secret
mypath = os.path.dirname(os.path.abspath(__file__))
with open(mypath + "/mini-auth.json") as configjsonfile:
conf = json.load(configjsonfile)
secret = conf["secret"]
#Set up logging
observer = textFileLogObserver(io.open(join(mypath,"mini-auth.log"), "a"))
logger = Logger(observer=observer,namespace="asyncsteem")
#Our CookieUtil is the main hub of our program
cu = CookieUtil(secret)
#Run our HTTP server on port 5080
root = MiniAuthWebServer(cu,account,reactor,logger)
factory = server.Site(root)
endpoint = endpoints.TCP4ServerEndpoint(reactor, 5080)
endpoint.listen(factory)
#Start twisted reactor.
reactor.run()
```
Most of this should look familiar from our [previous tutorial](https://steemit.com/utopian-io/@mattockfs/writing-an-asynchronous-steem-web-app-in-python-with-asyncsteem), but let us focus on the new parts. We make some changes to what we have now to get things up and running with a simple service and we'll discuss the **jinja2** template stuff afterwards, so lets focus on the actual cookie stuff for now. There are two resources we serve in order to make sure our user has a cookie:
* /: SetCookieIfNeeded, this is our root resource. This resource set the cookie if none is found and then redirects to the /steemauth resource. If there is no cookie, a new cookie is created and set.
* /steemauth : SteemAuth, this resource is assumed to be used with the cookie set. If there is no **valid** cookie, a new cookie is created overwriting any old cookie that may be there.
#### Adding Redis, asynchronously
Redis is an in-memory database that is very suitable for our current purpose of storing info on cookie usage and authenticated users. Even blocking synchronous operations in Redis are quite fast, it is important to resist our tendency to just intermix synchronous operations in our asynchronous event processing code. Remember, we don't know how many concurrent clients might be knocking on our door at any given time. It is important to know when you can call synchronous operations, for example at server construction time as we see with our Jinja2 template loading in the code above, and when not to, for example, while processing an asynchronous HTTP request. Fortunately, many databases can be approached asynchronously using Twisted (compatible) libraries in Python. For old-school relational databases, there is *twisted.enterprise.adbapi* and many other like Redis a bit of searching will usually yield an option to asynchronously interact with the database of your choice. For Redis the **cyclone** library provides us with the nonblocking asynchronous primitives we need.
```python
#Asynchonous Redis library for storing some persistent data with.
from cyclone import redis
#This is our hub class that is used both from the blockchain streamer and from the web server.
class CookieUtil:
def __init__(self,secret,logger):
...
#Callback for setting the redis connection.
def setDb(db):
self.db = db
self.logger.info("Connected to Redis")
self.logger = logger
#The local Redis is used to keep a persistent store of cookie to authenticated user mappings
self.db = None
d = redis.Connection()
d.addCallback(setDb)
#Try to keep a record of how far behind we are.
self.last_blocktime = datetime.utcfromtimestamp(0)
def new_cookie(self):
...
def valid_cookie(self,cookie):
...
if ok and self.db != None:
#If the cookie is OK, keep a record of when this cookie was last used in our Redis server.
k = cookie_id + "-lastvisit"
v = str(time.time())
d=self.db.set(k,v)
return ok
def authenticated_user(self,cookie_id):
if self.db == None:
return None
#Look up if the cookie is linked to an authenticated user.
k = cookie_id + "-steemauth"
return self.db.get(k)
def blocktime(self,tm):
#Remember the last blocktime
self.last_blocktime = tm
def process_transfer(self,frm, memo):
if self.db != None:
#If we register a potentially authenticating transfer in the blochchain,
#look in Redis to see if we gave the memo out as cookie id.
k1 = memo[:40] + "-lastvisit"
ltp = self.db.get(k1)
def process_last_time(lasttime):
#If the last time isn't None, this is a cookie id we know
if lasttime != None:
self.logger.info("Known cookie id in transaction.")
#Check if this cookie id has already been registered to a steemit account
k2 = memo[:40] + "-steemauth"
acp = self.db.get(k2)
def process_account(account):
#If the memo holds a valid cookie id that has never been registered to any steem account before
if account == None:
#Store the cookie steem-account mapping.
d=self.db.set(k2,frm)
self.logger.info("Cookie has now been linked to a steemit user.")
else:
self.logger.info("This particular cookie id was already registered to a steem user before.")
#Above is asynchounous callback for result from Redis
acp.addCallback(process_account)
else:
self.logger.info("Transaction without valid cookie id")
#Above is asynchounous callback for result from Redis
ltp.addCallback(process_last_time)
else:
self.logger.error("No Redis Db connection yet while processing transfer.")
def behind_string(self):
#Return how many time has passed since last block we saw.
behind = datetime.utcnow() - self.last_blocktime
return str(behind)
```
We start off making some changes to our CookieUtil class. Note that the set method is basically fire and forget so it is easy to overlook the fact these methods are asynchronous. Note also the authenticated_user method looks deceptively like something that might return a string. We will come back to that one below, now we look at the **process_transfer** method that we will bind to an asynchsteem bot class soon.
If we detect a transfer, we look up in our Redis database to see if the memo field matches a session id we know, and if it does, we check if the session id has not previously been claimed. Then if it hasn't, we can safely bind the user doing the transfer to the session id we found in the memo field.
Notice we have created a chain of callbacks. We do a get, an other get and then a set, all asynchonously. The last set is fire and forget as we saw before, but the gets all register a callback that gets called with the results.
#### asyncsteem transfers bot code
```python
#Asyncsteem import for talking to JSON-RPC steem API nodes.
from asyncsteem import ActiveBlockChain, RpcClient
#This is the basic asyncsteem bot.
class TransferStream:
def __init__(self,cu,account):
self.cu = cu
self.account = account
#We listen only to transfers
def transfer(self,tm,transfer_event,client):
frm = transfer_event["from"]
to = transfer_event["to"]
memo = transfer_event["memo"]
#For keeping track of if we are behind on the blockchain.
self.cu.blocktime(tm)
#Only process transfers to one single account, our owner.
if to == self.account:
self.cu.process_transfer(frm, memo)
...
account = conf["account"]
...
#Asyncsteem blochchain streamer
blockchain = ActiveBlockChain(reactor,log=logger,nodelist="default")
#Our CookieUtil is the main hub of our program
cu = CookieUtil(secret,logger)
#Instantiate our bot and link it to our blockchain
steembot = TransferStream(cu,account)
blockchain.register_bot(steembot,"flag_stream")
...
```
So now it is time to link in our main asyncsteem stuff, We define a blockchain streaming bot that processes transfer events and call the method we just discussed at every transfer to our configured owner account.
So now we have our code for authenticating users with microtransactions, let us add some hello-world code to show we mean business.
#### Some authentication using service.
We discussed @steem-ua before when talking about services that should benefit from using micro-transaction based authentication. Now, however, we look at @steem-ua from a more up note angle. In [this post](https://steemit.com/steem/@tcpolymath/a-simple-python-script-to-find-inactive-follows) @tcpolymath described how, for @steem-ua users, it might be useful to find and unfollow inactive follows. In his script @tcpolymath used a 40-day threshold for the newest posts for finding inactive follows. We change our web server code and add in some extra asyncsteem code to get our authenticated service to do something useful. List the stale follows of our authenticated user. Again we do this in a way that chains a number of callbacks.
```python
#Main function this server provides to authenticated users. List stale followed accounts.
#This functionality was borowed from a script by @tcpolymath.
def report_on_followers(request,steemaccount,cu,reactor,log,template,etemplate):
def process_following(event,client):
#Process the followed accounts response from the API node.
def process_accounts(event2,client2):
#Make a list of all accounts that didn't post for more than 40 days.
stale_accounts = []
for account in event2:
name = account["name"]
inactive = (datetime.utcnow() - dateutil.parser.parse(account["last_post"])).days
if inactive > 40:
ias = str(inactive)
stale_accounts.append({"name": name , "inactive" : ias})
#Use jinja2 to return a simple report
html = template.render(authenticated = steemaccount, results=stale_accounts, behind=cu.behind_string())
request.write(html.encode("ascii"))
request.finish()
def process_accounts_err(errno,msg,client):
html = etemplate.render(message=msg)
request.write(html.encode("ascii"))
request.finish()
#Make a list with the accounts that we follow.
accountlist = [];
for entry in event:
accountlist.append(entry["following"])
#Request account info on all followed accounts.
opp2 = client.get_accounts(accountlist)
opp2.on_result(process_accounts)
opp2.on_error(process_accounts_err)
def process_following_err(errno,msg,client):
html = etemplate.render(message=msg)
request.write(html.encode("ascii"))
request.finish()
#STEEM JSON-RPC client.
client = RpcClient(reactor,log,stop_when_empty=False,rpc_timeout=40)
#Start by querying up to 1000 folowers of the authenticared account.
opp = client.get_following(steemaccount,"","blog",1000)
opp.on_result(process_following)
opp.on_error(process_following_err)
client()
#Our base resource and authentication check.
class SteemAuth(resource.Resource):
isLeaf = True
def __init__(self,cu, account,templates,reactor,log):
self.cu = cu
self.account = account
self.templates = templates
self.reactor = reactor
self.log = log
def render_GET(self, request):
#If we are here, there should be a cookie
oldcookie = request.getCookie("steemauth")
#Check if the cookie is actually a valid one we created ourselv
if cu.valid_cookie(oldcookie):
#Take just the cookie id from the cookie.
cookie_id = oldcookie.split("-")[0]
#Get defered for a possible authenticated user from this cookie id.
sa = self.cu.authenticated_user(cookie_id)
def get_steemacount_from_redis(steemaccount):
if steemaccount == None:
#If the user isn't authenticated yet, return authentication page.
self.log.info("Cookie not assigned to any steemit user")
rv = self.templates["askauth"].render(account=account,
session=cookie_id,
behind=cu.behind_string()
)
request.write(rv.encode("ascii"))
request.finish()
else:
#If the user is authenticated, run core business logic as borowed from @tcpolymaths script.
self.log.info("Cookie matches authenticated steemit user")
req = request
template = self.templates["following"]
etemplate = self.templates["error"]
report_on_followers(req,steemaccount,cu,self.reactor,self.log,template,etemplate)
return
#If there is a Redis problem, sa will ne None
if sa == None :
html = self.templates["error"].render(message="No connection to local Redis database")
return html.encode("ascii")
else :
#Let the defered callback above handle the result from the Redis lookup
sa.addCallback(get_steemacount_from_redis)
return server.NOT_DONE_YET
else:
newcookie = self.cu.new_cookie()
request.addCookie("steemauth",newcookie)
return redirectTo("/steemauth",request)
#Our mini authentication server with some basic helo world functionality.
class MiniAuthWebServer(resource.Resource):
def __init__(self,cu, account,reactor,log):
...
self.templates["following"] = j2_env.get_template('following.tmpl')
...
def getChild(self, name, request):
...
```
As we mentioned before, the return value from authenticated_user isn't a string. It is a *deferred* that we need to add a callback to. A callback that processes the actual return string, or None value if no authenticated user is found. If in our case an authenticated user is found, the report_on_followers function gets called that does the same thing @tcpolymath's script did.
This function uses asyncsteem in a different way. It spawns a new asynchonous RPC client and creates a chain of two asynchonous JSON-RPC calls to the STEEM API nodes:
* get_following : Getting a list of accounts the authenticated user is following.
* get_accounts: Get meta info, including last post date for the list of users returned from get_following.
Notice also how this chain is handled by the web resource. Instead of returning content from render-GET, the render_GET returns the special value NOT_DONE_YET. This leaves the responsibility for the HTTP response to the end of the callback chain using the write and finish methods on the request object.
#### jinja2
There is still one loose end to our code. Jinja2. Jinja2 is a HTML template engine that allows us to easily map data structures onto HTML content. Our stale follows teomate liiks something like this.
```html
<!DOCTYPE html>
<html>
<head>
<title>Mini-Auth demo asyncsteem</title>
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<link href="http://netdna.bootstrapcdn.com/bootstrap/3.0.0/css/bootstrap.min.css" rel="stylesheet" media="screen">
<style type="text/css">
.container {
max-width: 500px;
padding-top: 150px;
}
</style>
<meta http-equiv="refresh" content="90">
</head>
<body>
<div class="container">
<p>
<h3>Inactive accounts followed by you ({{authenticated}})</h3>
<ol>
{% for result in results %}
<li><A HREF="https://steemit.com/@{{ result.name }}">@{{ result.name }}</A> {{ result.inactive }} days since last post.
{% endfor %}
</ol>
</p>
<p>Currently running <b>{{behind}}</b> behind on the blockchain.</p>
</div>
<script src="http://code.jquery.com/jquery-1.10.2.min.js"></script>
<script src="http://netdna.bootstrapcdn.com/bootstrap/3.0.0/js/bootstrap.min.js"></script>
</body>
</html>
```
A detailed discussion of jinja2 falls outside of the context of this tutorial, but as the for loop in this example shows us, jinja2 allows us to map nested data structures to html structures.
The result of our script will now look something like this:
#### Curriculum
* [Writing a simple STEEM statistics-script in Python with asyncsteem](https://steemit.com/utopian-io/@mattockfs/writing-a-simple-steem-statistics-script-in-python-with-asyncsteem)
* [Writing an asynchronous STEEM web-app in Python with asyncsteem](https://steemit.com/utopian-io/@mattockfs/writing-an-asynchronous-steem-web-app-in-python-with-asyncsteem)
#### Proof of Work Done
The [mini-auth example script](https://github.com/pibara-utopian/asyncsteem/tree/master/sample_code/mini-auth) , that is part of the sample scripts in the asyncsteem repo, was written for this tutorial. Want to see this code in action, check [this](http://steem-auth.timelord.ninja/steemauth) demo site.