Building an iota tangle from scratch in python and flask
Introduction
Before i start i want to talk about 2 very useful resources that was very influential on this project. first of all a post on hackernoon.com by an author Daniel van Flymen. He implements a blockchain similar to the one implemented by blockchain. This post can be found here https://hackernoon.com/learn-blockchains-by-building-one-117428612f46. I will not go into the definitions of what a blockchain is because this post gives all of the necessary details about what a blockchain is to how it works. He gives very clean code that is very easy to follow. A great post you should check it out.
Second is a video by siraj reval
He gives a very clear and easy to follow explanation of how the iota tangle works. His explanation is the way in which i have chosen to implement my code. If the implementation, or my understanding of how the tangle works is incorrect please leave a comment and i will update my code and repost it. For this reason i will not be going into depth about what the tangle is too much.What is a tangle
Well now that the formalities are out of the way what is a tangle. Based on the video above a tangle is basically a directed acyclic graph, (DAG), which uses a proof of work algorithm. However instead of having miners to do the proof of work algorithm the act of making a transaction requires the client to validate 2 other random transactions which were made by some other random client. This DAG is then shared among every client in the network using a consensus algorithm
What will be implemented
This implementation is a basic implementation. It will focus on building the tangle and being able to share it with other clients. It will not be a way to share money reliably nodes will not have unique identifiers like wallet id’s etc. these may come later in follow up posts if people like it.
Who is this guide aimed at? You should be comfy reading and writing some basic Python, as well as have some understanding of how HTTP requests work, since we’ll be talking to our Blockchain over HTTP.
What do I need? Make sure that Python 3.6+ (along with pip) is installed. You’ll also need to install Flask and the Requests library. Also you will need an http client like postman or curl. I will be using postman as I simply like the gui. But you can use anything. Postman can be found here https://www.getpostman.com.
The final code can be found here https://github.com/ljlabs/tangle-pow
Step 1 building the tangle
Open a text editor and create a new files called tangle.py, api.py, and properties.py we will be using 3 files to help make the code easier to understand each part.
The file properties.py will be our basic configuration for the network and thus will have no logic only constants
RequiredProofs = 2 # the number of times a transaction needs to be processed
# befor it is seen as valid
numberOfValidationNodesNeeded = 2 # the number of nodes needed to connect to when
# addeing a new node to the network for the purpose
# of validating them when making a transaction
The skeleton for tangle.py will be
import properties as prop
import json
from time import time
import hashlib
import requests
from urllib.parse import urlparse
class Tangle(object):
def init(self):
pass # initialize the network
def valid_proof(self):
pass # used to make sure that the pow was correct
def proof_of_work(self):
pass # calculate the proof of work
def hash():
pass # compute the hash
def createNode(self):
pass # create a new node
def send_transaction(self, data):
pass # send a transaction
###########################################################################
# this is the consensus algorithm
###########################################################################
def valid_tangle(self, tangle):
pass # we will use this to ensure that the tangles we get from peers are correct
def register_peer(self, address):
pass # we will use this a add new peers to our p2p network
And finally the skeleton for the api.py
import hashlib
import json
from textwrap import dedent
from time import time
from uuid import uuid4
from tangle import Tangle
from urllib.parse import urlparse
import requests
from flask import Flask, jsonify, request
Instantiate our Node
app = Flask(name)
Instantiate the Blockchain
tangle = Tangle()
def new_transaction():
pass # make a new transaction on the network
def full_chain():
pass # returns the current tangle
Consensus
def register_nodes():
pass # add new peers to the network
def consensus():
pass # check for other maybe newer tangles on the network
def list_peers():
pass # return a list of peers
if name == 'main':
app.run(host='0.0.0.0', port=5001)
What do the nodes look like
Each of the nodes in our tangle will look like this
Node = {
'index': "int value",
'timestamp': time(),
'data': "this is the transaction data that is being stored",
'proof': "proof of work",
'previous_hashs': "the hash of the previous 2 nodes that it is connected to",
'previous_nodes': 'index values',
'next_nodes': 'indexes of the nodes pointing to it',
'validity': the number of times the node has been proven
}
At this point the way the tangle works should be easy to see. Each node has its data and identifying characteristics as well as some information about how it fits into the tangle and how many times it has been validated by the network
Creating transactions
This is by far the most work that will have to be implemented as making a transaction also validates other transactions making it the most important step in the creation of our tangle
class Tangle(object):
def init(self):
self.nodes = []
self.peers = set()
for i in range(prop.numberOfValidationNodesNeeded):
# Create the genesis block
self.nodes.append(self.createNode(None, [], len(self.nodes), validity = prop.RequiredProofs))
def valid_proof(last_proof, last_hash, proof):
# ensures that the node has the correct number of zeros
guess = (str(last_proof) + str(last_hash) + str(proof)).encode()
guess_hash = hashlib.sha256(guess).hexdigest()
return guess_hash[:4] == "0000"
def proof_of_work(self, last_proof, last_hash):
# computes the proof of work
proof = 0
while self.valid_proof(last_proof, last_hash, proof) is False:
proof += 1
return proof
def hash(node):
# make a hash of the block
# We must make sure that the Dictionary is Ordered, or we'll have inconsistent hashes
node_string = json.dumps(node, sort_keys=True).encode()
return hashlib.sha256(node_string).hexdigest()
def validate_node(self, node):
if self.nodes[node['index']]['validity'] < prop.RequiredProofs:
last_proof = self.nodes[node['index']]['proof'] # this nodes proof
last_hash = ""
for prevHash in self.nodes[node['index']]['previous_hashs']:
last_hash += prevHash # the hashes of the nodes this node connects
self.nodes[node['index']]['proof'] = self.proof_of_work(last_proof, last_hash)
self.nodes[node['index']]['validity'] += 1
def createNode(self, data, prevNodes, newIndex, validity=0):# these prevNodes are the indexes in the dag that points to the previous nodes
prevHashes = []
'''
may need to update every node that points to this when sending transaction
'''
for i in prevNodes:
prevHashes.append(self.hash(self.nodes[i]))
# now we tell the nodes that we are pointing to that we are poiinting to them
self.nodes[i]['next_nodes'].append(newIndex)
Node = {
'index': newIndex,
'timestamp': time(),
'data': data,
'proof': 0,
'previous_hashs': prevHashes, # hashes of the nodes we are connecting to
'previous_nodes': prevNodes, # indexes of the nodes we are connecting to
'next_nodes': [], # indexes of future nodes that will connect to us
'validity': validity,
}
return Node
def send_transaction(self, data):
# find 2 nodes in the network that are un proven
nodesToattach = []
nodesIndexes = []
newIndex = len(self.nodes)
worstCaseScinario = []
worstCaseScinarioindexes = []
'''
this function should be changed to search randomly
'''
for i in range(len(self.nodes)-1, -1, -1):
node=self.nodes[i]
if node['validity'] < prop.RequiredProofs:
nodesToattach.append(node)
nodesIndexes.append(node['index'])
else:
if worstCaseScinario == [] or len(worstCaseScinario) < prop.numberOfValidationNodesNeeded:
worstCaseScinario.append(node)
worstCaseScinarioindexes.append(node['index'])
if len(nodesToattach) == prop.numberOfValidationNodesNeeded:
break
# if there are not enough un varified transactions then use verified transactions
while len(nodesToattach) < prop.numberOfValidationNodesNeeded:
nodesToattach.append(worstCaseScinario.pop())
nodesIndexes.append(worstCaseScinarioindexes.pop())
# process nodes to attatch
for node in nodesToattach:
self.validate_node(node)
# now that those nodes are proven
# we can now attatch our node to the dag
self.nodes.append(self.createNode(data, nodesIndexes, newIndex))
return newIndex
To break down the steps
First we have to find 2 nodes in the network which are unverified. If not enough unverified transactions are found then verified transactions will have to be used. A transaction is considered valid if its validity level is >= the property RequiredProofs set in the properties.py file earlier This should be done at random. If a transaction that has been previously verified is taken then we will not validate it again.
Then we need to verify those 2 chosen nodes:
We compute its proof of work using an algorithm very similar to hashcash described here https://en.wikipedia.org/wiki/Hashcash. We increment its validity of each of the nodes
We then create the new transaction node:
We set its previous hashes to a list of the hashes of the nodes that we just verified
We set its previous nodes to the index of the nodes that we had previously created.
We timestamp the new node.
We set its index to the index at the end of the list which holds all of the lists
Its next nodes are set to [] as there are no nodes pointing to this node yet
And the validity is set to zero as no node has validated it yet
And finally we tell the 2 nodes that we have just verified that the new node that we have created is pointing to it.
Building the api
To have other peers in our p2p network know about us, and to be able to make transaction we will need to have a network facing interface this is where flask comes in.
def new_transaction():
# update tangle
tangle.resolve_conflicts()
# begin transaction
values = request.get_json()
# Check that the required fields are in the POST'ed data
required = ['sender', 'recipient', 'amount']
if not all(k in values for k in required):
return 'Missing values', 400
# Create a new Transaction
index = tangle.send_transaction(values)
response = {'message': 'Transaction will be added to Block ' + str(index)}
# tell peers to update tangle
for peer in tangle.peers:
requests.get("http://"+str(peer) + "/peers/resolve")
return jsonify(response), 201
def full_chain():
response = {
'tangle': tangle.nodes,
'length': len(tangle.nodes),
}
return jsonify(response), 200
We create 2 end point
/transactions/new
Allowing us to create new transactions
/tangle
Allowing us to view our current tangle graph
Consensus
The final portion of the project
This is the more interesting portion. Till this point we have a tangle that accepts new transactions as well as allows us to view our transaction history now comes the part to make that network decentralised.
We begin by updating our api.py file to is final form
import hashlib
import json
from textwrap import dedent
from time import time
from uuid import uuid4
from tangle import Tangle
from urllib.parse import urlparse
import requests
from flask import Flask, jsonify, request
Instantiate our Node
app = Flask(name)
Generate a globally unique address for this node
node_identifier = str(uuid4()).replace('-', '')
Instantiate the Blockchain
tangle = Tangle()
def new_transaction():
# update tangle
tangle.resolve_conflicts()
# begin transaction
values = request.get_json()
# Check that the required fields are in the POST'ed data
required = ['sender', 'recipient', 'amount']
if not all(k in values for k in required):
return 'Missing values', 400
# Create a new Transaction
index = tangle.send_transaction(values)
response = {'message': 'Transaction will be added to Block ' + str(index)}
# tell peers to update tangle
for peer in tangle.peers:
requests.get("http://"+str(peer) + "/peers/resolve")
return jsonify(response), 201
def full_chain():
response = {
'tangle': tangle.nodes,
'length': len(tangle.nodes),
}
return jsonify(response), 200
Consensus
def register_nodes():
values = request.get_json()
print(values)
print("asdugdag")
peers = None
if type("") == type(values):
print(json.loads(values))
peers = json.loads(values)['peers']
else:
peers = values.get('peers')
if peers is None:
return "Error: Please supply a valid list of nodes", 400
for peer in peers:
tangle.register_peer(peer)
response = {
'message': 'New peers have been added',
'total_nodes': list(tangle.peers),
}
return jsonify(response), 201
def consensus():
replaced = tangle.resolve_conflicts()
if replaced:
response = {
'message': 'Our chain was replaced',
'new_chain': tangle.nodes
}
else:
response = {
'message': 'Our chain is authoritative',
'chain': tangle.nodes
}
return jsonify(response), 200
def list_peers():
response = {
'known_peers': list(tangle.peers),
}
return jsonify(response), 201
if name == 'main':
app.run(host='0.0.0.0', port=5001)
As well as our tangle.py
import properties as prop
import json
from time import time
import hashlib
import requests
from urllib.parse import urlparse
"""
nodes in the dag are represented in json as folows
they can be transmitted across the network in this form and then can be
initialized in this form
Node = {
'index': "int value",
'timestamp': time(),
'data': "this is the transaction data that is being stored",
'proof': "proof of work",
'previous_hashs': "the hash of the previous 2 nodes that it is connected to",
'previous_nodes': 'index values',
'next_nodes': 'indexes of the nodes pointing to it',
'validity': the number of times the node has been proven
}
"""
class Tangle(object):
def init(self):
self.nodes = []
self.peers = set()
for i in range(prop.numberOfValidationNodesNeeded):
# Create the genesis block
self.nodes.append(self.createNode(None, [], len(self.nodes), validity = prop.RequiredProofs))
def valid_proof(last_proof, last_hash, proof):
# ensures that the node has the correct number of zeros
guess = (str(last_proof) + str(last_hash) + str(proof)).encode()
guess_hash = hashlib.sha256(guess).hexdigest()
return guess_hash[:4] == "0000"
def proof_of_work(self, last_proof, last_hash):
# computes the proof of work
proof = 0
while self.valid_proof(last_proof, last_hash, proof) is False:
proof += 1
return proof
def hash(node):
# make a hash of the block
# We must make sure that the Dictionary is Ordered, or we'll have inconsistent hashes
node_string = json.dumps(node, sort_keys=True).encode()
return hashlib.sha256(node_string).hexdigest()
def validate_node(self, node):
if self.nodes[node['index']]['validity'] < prop.RequiredProofs:
last_proof = self.nodes[node['index']]['proof'] # this nodes proof
last_hash = ""
for prevHash in self.nodes[node['index']]['previous_hashs']:
last_hash += prevHash # the hashes of the nodes this node connects
self.nodes[node['index']]['proof'] = self.proof_of_work(last_proof, last_hash)
self.nodes[node['index']]['validity'] += 1
def createNode(self, data, prevNodes, newIndex, validity=0):# these prevNodes are the indexes in the dag that points to the previous nodes
prevHashes = []
'''
may need to update every node that points to this when sending transaction
'''
for i in prevNodes:
prevHashes.append(self.hash(self.nodes[i]))
# now we tell the nodes that we are pointing to that we are poiinting to them
self.nodes[i]['next_nodes'].append(newIndex)
Node = {
'index': newIndex,
'timestamp': time(),
'data': data,
'proof': 0,
'previous_hashs': prevHashes, # hashes of the nodes we are connecting to
'previous_nodes': prevNodes, # indexes of the nodes we are connecting to
'next_nodes': [], # indexes of future nodes that will connect to us
'validity': validity,
}
return Node
def send_transaction(self, data):
# find 2 nodes in the network that are un proven
nodesToattach = []
nodesIndexes = []
newIndex = len(self.nodes)
worstCaseScinario = []
worstCaseScinarioindexes = []
'''
this function should be changed to search randomly
'''
for i in range(len(self.nodes)-1, -1, -1):
node=self.nodes[i]
if node['validity'] < prop.RequiredProofs:
nodesToattach.append(node)
nodesIndexes.append(node['index'])
else:
if worstCaseScinario == [] or len(worstCaseScinario) < prop.numberOfValidationNodesNeeded:
worstCaseScinario.append(node)
worstCaseScinarioindexes.append(node['index'])
if len(nodesToattach) == prop.numberOfValidationNodesNeeded:
break
# if there are not enough un varified transactions then use verified transactions
while len(nodesToattach) < prop.numberOfValidationNodesNeeded:
nodesToattach.append(worstCaseScinario.pop())
nodesIndexes.append(worstCaseScinarioindexes.pop())
# process nodes to attatch
for node in nodesToattach:
self.validate_node(node)
# now that those nodes are proven
# we can now attatch our node to the dag
self.nodes.append(self.createNode(data, nodesIndexes, newIndex))
return newIndex
###########################################################################
# this is the consensus algorithm
###########################################################################
def valid_tangle(self, tangle):
for node in tangle:
if node['index'] >= prop.numberOfValidationNodesNeeded: # dont test genesis nodes
validitiyOfNode = node['validity']
# make sure that the same number of nodes saying that they have
# validated him as his validity level
nextNodes = node['next_nodes']
print(nextNodes)
if validitiyOfNode < len(nextNodes):
return False
# make sure these nodes are pointing to him
for Nnode in nextNodes:
print(tangle[Nnode])
if node['index'] not in tangle[Nnode]['previous_nodes']:
return False
return True
def register_peer(self, address):
parsed_url = urlparse(address)
self.peers.add(parsed_url.netloc)
def resolve_conflicts(self):
neighbours = self.peers
new_tangle = None
# We're only looking for chains longer than ours
max_length = len(self.nodes)
# Grab and verify the chains from all the peers in our network
for peer in neighbours:
response = requests.get("http://"+str(peer) + "/tangle")
if response.status_code == 200:
length = response.json()['length']
tangle = response.json()['tangle']
# Check if the length is longer and the chain is valid
if length > max_length and self.valid_tangle(tangle):
max_length = length
new_tangle = tangle
# Replace our chain if we discovered a new, valid chain longer than ours
if new_tangle:
self.nodes = new_tangle
return True
return False
This now allows us to make calls to our network register new peers to that network and check if our peers have newer versions of our tangle it does this by checking if their tangle is longer than ours
Finishing up
At this point we can spin up the api by calling python api.py on 2 computers or on 2 different port numbers which is what i had done one on port 5000 and the other on port 5001
Firstly we should make a call to our api at http://127.0.0.1:5000/peers/register
{
"peers": ["http://X.X.X.X:5000"]
}
Where X.X.X.X is the url of the second instance of the application on your lan that you had spin up
Using an http client make a post request to http://127.0.0.1:5000/transactions/new
With the json object
{
"sender": "this is such fucking bull shit",
"recipient": "anne",
"amount": 5
}
And we should see a response
{
"message": "Transaction will be added to Block 2"
}
Now we can either make a call to http://127.0.0.1:5000/tangle or http://X.X.X.X:5000/tangle and both will give us the same tangle
And this concludes the tangle. If this was helpful please tell me in the comments. This is my first post so please go easy on me. It is probably not up to the scratch of other posts i am a software developer and writing is new to me. Please feel free to take my code on github and edit it in any project. Please tell me if this was helpful and if i should keep making new posts about building blockchain technology