My journey to find intraday cryptocurrency trading data

[galea]

TLDR; I wrote a Python bot to render JS and scrape live coin prices because I couldn't find a free API. Then, a few days later, I found a free API for historical intraday trading data.

Ok great, just show me the API code.

Part 1 - The Idea

After seeing some cool blog posts predicting stocks (US equity market) with neural networks, I wanted to try it myself. Turns out yahoo has an API that works great for daily data but it's a no-go on the intraday. So the hunt began.

I didn't have much luck, but in hunting I discovered crypto. And then I watched a couple videos of traders having success with the cryptocurrency space (it's hard NOT to have trading success during a bull market as strong as crypto has been) and realized it would be far more timely to look here.

Part 2 - The JavaScript Scape

I scraped this CryptoCurrency.com table.

Here's what some of the Python code looked like:

The data scrape module looks like this (number 1, below)

2 - Decorator for error handling.
3 - Module for getting coin name e.g. Ethereum.
4 - Module for getting the symbol e.g. ETH.

The live prices are scraped in a similar way.

The bot is controlled via a chromedriver and the selenium python module, as seen in the following screen-capture GIF of the script running in an IPython notebook.

When the script is run, the webdriver on the left automatically renders the JavaScript as new URLs are requested. The resulting HTML is ingested into python and data is gathered.

I was so excited to have finished the scraper! I wanted my intraday trading data. I had EARNED it.

Then, before I had time to set it up on my RaspberryPI, I made the discovery.

I found...

Part 3 - The API

Now that you've seen the hard way, you will appreciate the easy way. Maybe not as much as I did though, seeing as I spend a week of free time on the scraper as seen above.

I looked back into my google history and actually found something marking the moment of discovery:

1 - Ingesting crypto news and info lead to a question
2 - I asked google the question
3 - I found the API

The API is easy to use, detailed, well documented, and best off all FREE.

Here is an example API call with python

import requests
url = 'https://min-api.cryptocompare.com/data/histominute' +\
        '?fsym=ETH' +\
        '&tsym=USD' +\
        '&limit=2000' +\
        '&aggregate=1'
response = requests.get(url)
data = response.json()['Data']

import pandas as pd
df = pd.DataFrame(data)
print(df)

I copy and pasted that code ^ into ipython just now and got this:

Check out this quick-start guide if you want to see a curated list of their API functions and examples.

Here is a sample from that ^ post.

Thanks for reading!

Drop me a comment if you have any questions.

Find me here on twitter

Comments

[marinagarra]

Hey, thanks so much for that, amazing work

[galea]

Thank you!

[steemitboard]

Congratulations @galea! You have completed some achievement on Steemit and have been rewarded with new badge(s) :

You published your First Post
You got a First Vote
You made your First Comment
You made your First Vote

Click on any badge to view your own Board of Honor on SteemitBoard.
For more information about SteemitBoard, click here

If you no longer want to receive notifications, reply to this comment with the word STOP

By upvoting this notification, you can help all Steemit users. Learn how here!