As a quant trader, data is the life-blood of my strategies. Data allows each of my trading bots to form their own unique opinions about their environments in much the same way our own experiences shape our view of the world.
One of the things I love about trading cryptocurrencies in particularly is the democratic access to data; exchanges make it very easy to access their data, especially via computer programs.
This stands in stark contrast to the traditional financial markets, like the US stock market or futures markets, where you have to pay out the ass to get access to any sort of market data. They even make you pay for delayed data, in certain situations. Its a fucking racket.
Why is this so in traditional markets?
Well, its obvious why the exchanges love it because you've got to pay them to play the game.
The market participants, who pay the market data toll, tolerate this arrangement because they know it keeps out the little guys and is thus a barrier to entry and new competition...
But in crypto things are different. Almost every exchange of any size has something called an API which we can all use to get the data we need for free
Today we're gonna look at how to download historical data from Binance using the Python programming language. I picked Python because:
- of its ubiquitous use in the financial industry (better know Python or R for data analysis)
- the amount of work other people have already done building useful packages
- i know it very well (not gonna bullshit you! LOL)
In particular I used Python 3.6 for this tutorial. This assumed you already have Python installed, there are plenty of tutorials online but i would recommend just going to https://www.python.org/ and going to their downloads section.
The structure of this tutorial will begin by going over the packages we need to import to accomplish our task. Then we'll build a function to download any symbol's historical data. Then we'll use the function to visualize the price of STEEM/USDT over the last month.
First we're gonna import the packages we need:
import requests # for making http requests to binance
import json # for parsing what binance sends back to us
import pandas as pd # for storing and manipulating the data we get back
import numpy as np # numerical python, i usually need this somewhere
# and so i import by habit nowadays
import matplotlib.pyplot as plt # for charts and such
import datetime as dt # for dealing with times
If you don't have these installed already you can always use pip to download them without much fuss.
A web API like Binance's is really just a specially crafted url which instead of returning a human readable webpage, returns a machine readable chunk of data instead.
For instance, on the Binance API everytime we want to get the historical price bars for a symbol, we use this root url:
root_url = 'https://api.binance.com/api/v1/klines'
Then to get the bars for a specific symbol and timeframe, we craft a special url to get the data we want:
symbol = 'STEEMETH'
interval = '1h'
url = root_url + '?symbol=' + symbol + '&interval=' + interval
print(url)
This will print the url:
https://api.binance.com/api/v1/klines?symbol=STEEMETH&interval=1h
You can click on that link and see the data that is returned. Its a bunch of numbers and brackets. Really its price data and its defined in a format called JSON. Really JSON is just a multi-level dictionary.
You dont really need to know the low level shit, because we're going to use the built-in Python package json to parse everything for us like magic:
data = json.loads(requests.get(url).text)
This is one of the reasons i love Python, easy one liners like this. In this one line we make a HTTP GET request to Binance's API to grab the raw bars data, then we use the json package to parse that into a Python dictionary. Phew!
Now we're going to take that dictionary and turn it into something VERY USEFUL called a dataframe
We do this with an open source Python package named pandas
Despite the cute name, it was originally developed by AQR, a huge quant hedge fund
df = pd.DataFrame(data)
df.columns = ['open_time',
'o', 'h', 'l', 'c', 'v',
'close_time', 'qav', 'num_trades',
'taker_base_vol', 'taker_quote_vol', 'ignore']
df.index = [dt.datetime.fromtimestamp(x/1000.0) for x in df.close_time]
These lines convert the dictionary into a dataframe. We also properly label the columns according to Binance's API docs and then we convert the raw millisecond timestamps into python datetime objects.
The dataframe will allow us to perform analysis and do data manipulation if we so chose...
Now let's wrap this all up in a convenient function we can use to download the data for whatever symbol our hearts desire:
def get_bars(symbol, interval = '1h'):
url = root_url + '?symbol=' + symbol + '&interval=' + interval
data = json.loads(requests.get(url).text)
df = pd.DataFrame(data)
df.columns = ['open_time',
'o', 'h', 'l', 'c', 'v',
'close_time', 'qav', 'num_trades',
'taker_base_vol', 'taker_quote_vol', 'ignore']
df.index = [dt.datetime.fromtimestamp(x/1000.0) for x in df.close_time]
return df
Now we can do cool stuff like download the data for STEEM/ETH and ETH/USDT:
steemeth = get_bars('STEEMETH')
ethusdt = get_bars('ETHUSDT')
Now we can multiply the price of STEEM/ETH by ETH/USDT to get STEEM/USDT
steemusdt = steemeth['c'].astype('float') * ethusdt['c'].astype('float')
And plot the result:
steemusdt.plot()
Now with the get_bars function you can download data for whatever symbols Binance supports. You can also change the interval parameter, which is set to '1h' or 1 hour by default.
So just to put the whole thing together:
import requests # for making http requests to binance
import json # for parsing what binance sends back to us
import pandas as pd # for storing and manipulating the data we get back
import numpy as np # numerical python, i usually need this somewhere
# and so i import by habit nowadays
import matplotlib.pyplot as plt # for charts and such
import datetime as dt # for dealing with times
def get_bars(symbol, interval = '1h'):
root_url = 'https://api.binance.com/api/v1/klines'
url = root_url + '?symbol=' + symbol + '&interval=' + interval
data = json.loads(requests.get(url).text)
df = pd.DataFrame(data)
df.columns = ['open_time',
'o', 'h', 'l', 'c', 'v',
'close_time', 'qav', 'num_trades',
'taker_base_vol', 'taker_quote_vol', 'ignore']
df.index = [dt.datetime.fromtimestamp(x/1000.0) for x in df.close_time]
return df
steemeth = get_bars('STEEMETH')
ethusdt = get_bars('ETHUSDT')
steemusdt = steemeth['c'].astype('float') * ethusdt['c'].astype('float')
steemusdt.plot(figsize=(16,9))
This is obviously just the start of the usefulness of this function. Subsequent tutorials will make use of the Binance API to get historical data for more advanced analysis like correlation filtering and network analysis. Check back for more quant finance and trading tutorials in the future!
thanks for the info
Wow, this is so cool! I love your quant finance and trading content, it is very original and I can tell you are very experienced in the field!
Because of your background in the industry, I was wondering if I could get your opinion on a project that I recently found? It is called Dropil. It is still a relatively new company, and they just did an ICO a couple weeks ago for their tokens called Drops. The company uses algo trading bots, and users can use the drop tokens to invest with the bots. It seems very promising, but given the industries history with scams like Bitconnect I am skeptical to join. To me, Dropil seems very legitimate though, and it is much different than bitconnect. Still, I would like a second opinion on it. Thanks, looking forward to your response! :)
not familiar w dropil, will take a looksee altho that might take a sec to form opinion
Good tip! I'm used in checking cryptomarketcap for history but I guess this can be helpful too.
i wish they had an api
Oke. Help me
프로그래밍 기술이 필요하네요 :)
just gotta start doing it... we all started somewhere... i remember when i was coding vba in excel thinking i was the motherfucking man
Great, just as I was thinking of trading bots I stumble to your posts! I will follow your tutorials with great interest. Thank you
just curious, have you built any bots in the past? would you be interested in tutorials that go into building simple bots? i'm always interested in feedback as what to write about next
as for me, i've been automated for years, as trading w bots provides my strategies with a level discipline i unfortunately lack in real life ;-)
I totally get you, I also lack discipline and trading is waiting most of the time. That's why I've always wanted to build a trading bot but just haven't got started.
So I got 0 experience about building trading bot and I'm interested of any information I can get, from simple to complicated.
I am very interested. I have made a few python programs with raspberry pi tutorials and I am very familiar with TA and trading concepts. Great content by the way!
Holy mother of..... I would have never known I could do something like that on binance if it weren't for this post. Thank you
lol! awesome
and this is just the tip of the coolness iceberg when it comes to trading + coding
Thank you for the information! You're right. I need to learn Python language too. Then I can make more money.
python is great for data analysis and general computing
theres also a strong market for javascript, both on the front end and the back end
i think those are the languages i would learn right now if i were just starting out
Superb content sharing bro
this is going to help many,
thanks for sharing.
a very nice sharing. Thank you for sharing this. I downloaded the data in a nice way. Thank you.
Thank you for sharing this way of doing business exchanges, i will learn after understanding all your explanation, i am happy to learn in your story. Have a nice day
Congratulation marketstack! Your post has appeared on the hot page after 156min with 47 votes.
This is very informative, you did a great job. Thanks very much I am swinging into action now.
Nice to know :) Now everybody can load data from binance :P
Your blog posts are really awesome. I enjoy watching your posts
Thanks for sharing very informative !!!
Thanks for sharing this info.
Hi @marketstack, I just upvoted you!
I see that you know what you're talking about when it comes to crypto, so we're inviting you to an invite only Altcoin Fantasy contest. You can find the username and password for the contest on our Steemit post here.
Here is the link for the contest: https://altcoinfantasy.com/p/steemit
Thanks and we hope to see you there!