DIY CryptoCurrency Analysis Platform: 5 Min

in #cryptocurrency • 7 years ago (edited)

TL;DR

I wanted to share with you all how to begin building an open-source platform for analyzing cryptocurrency market data. If you aren't a fan of getting your trading advice from platforms derived from traditional markets and are curious about building your own tools for forecasting and investing, I hope you'll read further and comment.

All feedback is appreciated.

Free Datahz?

There are many APIs to choose from with more detailed options ... often with a price tag.

I chose this API for a few reasons:

Setup

Get VBox

https://www.virtualbox.org/wiki/Downloads

Get ELK VM

https://bitnami.com/redirect/to/151919/bitnami-elk-5.4.1-0-linux-debian-8-x86_64.ova

Config ELK VM

Change Adapter 1 to Host-Only

1.png

Enable Adapter 2 to NAT

2.png

For Improved Performance, Adjust RAM/Processors/Etc.

3.png

Login with Defaults
bitnami/bitnami
(Change password)

Enable ssh (this can be skipped but, I preferred ssh for this)

sudo rm -f /etc/ssh/sshd_not_to_be_run
sudo systemctl enable ssh
sudo systemctl start ssh

Install python + pip + elasticsearch.py

sudo apt-get upgrade -y python-dev libssl-dev git-dev
sudo python < <(curl -s https://bootstrap.pypa.io/get-pip.py)
sudo pip install elastic search

Check IP Address

sudo ifconfig
eth0      Link encap:Ethernet  HWaddr de:ad:be:ef:ca:fe
          inet addr:192.168.100.101  Bcast:192.168.100.255  Mask:255.255.255.0

Now we are ready to get going ...


The basic workflow is like so:

  1. Fetch JSON data from API
  2. Parse JSON data
  3. Configure elasticsearch index mapping to use the JSON object timestamp
  4. Send result of parsed JSON data to elasticsearch

This script can be saved to to something like chistory.py:

#Imports
import sys
import datetime
import requests
import json
from elasticsearch import Elasticsearch

#Variables
server = 'localhost'
arg1 = sys.argv[1]
arg2 = sys.argv[2]
doctype = arg2.lower()
id = arg1

#Send data to ELK function
def sendtoelastic(data):
    try:
        es = Elasticsearch(server)
        es.index(index=id, doc_type=doctype, body=data)
    except Exception as e:
        print 'Error sending to elasticsearch: {0}'.format(str(e))
        pass

#Send settings to ELK function
def settingselastic(data):
    try:
        es = Elasticsearch(server)
        es.indices.create(index=id, ignore=400, body=data)
    except Exception as e:
        print 'Error sending to elasticsearch: {0}'.format(str(e))
        pass

#Build request, download and load JSON data
payload = {'coin': arg2, 'period': arg1}
r = requests.get('http://coinmarketcap.northpole.ro/history.json?', params=payload)
h = r.text
d = json.loads(h)['history']

#Send settings
settings = '''{"mappings": {''' + '"' + arg1 + '"' + ''': {"properties": {"timestamp": {"type": "date","format": "epoch_second"}}}}}'''
settingselastic(settings)

#Send data
for i in d.items():
    for l in i:
        if type(l) is dict:
            sendtoelastic(json.dumps(l))

Usage:

python chistory.py 2017 BTC
python chistory.py 2017 BTC;python chistory.py 2017 ETH;python chistory.py 2017 DOGE
python chistory.py 2016 BTC;python chistory.py 2016 ETH;python chistory.py 2016 DOGE

Coins:

A current list of coins can be found here

Dumping Index:

If for any reason we aren't happy with the results, we can dump elastic like this:

curl -XDELETE 'http://localhost:9200/_all'

Haz Datahs ... Now What?

Here's a quick how-to visualize your data in Kibana with a few line graphs which may be of interest ...

Create Visualization

Change.1h x Date

  1. Click the "Visualize" Button
  2. Click "Create New Visualization"
  3. Click "Basic Charts > Line"
  4. Select Index "201*"
  5. Click Drop Down Button for Y-Axis
  6. Select Aggregation > "Sum"
  7. Select Field > "change1h"
    5.png
  8. Click "Select Buckets Type" > X-Axis
  9. Select Aggregation > "Date Histogram"
  10. Select Field > "timestamp"
  11. Select Interval > "Daily"
    6.png
  12. Click "Add Sub-Buckets" > "Split Series"
  13. Sub-Aggregation > "Filters"
  14. Filter 1 > "btc"
  15. Filter 2 > "eth"
  16. Filter 3 > "doge"
    7.png
  17. Save

Change.24h x Date

  1. Change Y-Axis Field > "change24h"
  2. Save
  3. Change name Change.24h
  4. Check "Save as a New Visualization" Option
  5. Save

Volume.EUR x Date

  1. Change Y-Axis Field > "volume24.eur"
  2. Save
  3. Change name Volume.EUR
  4. Check "Save as a New Visualization" Option
  5. Save

Create a Dashboard

  1. Select Dashboard Button
  2. Create New Dashboard
  3. Add Visualizations to Dashboard

4.png

On APIs

This is just a basic example of leveraging an API for analytics.

The basic mechanics ought be able to be applied to other APIs of interest.

If you'd like to see more API analytics, follow and up vote. I hope this helps someone!

If you made it this far ... Thanks!

#trading #bitcoin #money #beyondbitcoin
7 years ago in #cryptocurrency by
$9.54
  • Promotion Cost $1.00
  • Past Payouts $9.54
  • - Author $7.24
  • - Curators $2.30
9 votes
Reply 3
Sort:  
  • Trending
    • [-]
     7 years ago  

    Will this api be able to provide the bid spread in a baseline 0 graph as opposed to what is currently displayed by the crypto exchanges. May be able to get a better read on momentary momentum.

    [-]
     7 years ago  

    I'm not really clear on all those buzz words but, I'll try to explain further ...

    This is collecting raw data from the API:

    • Date
    • Position
    • Name
    • Symbol
    • Category
    • MarketCap x (usd,btc,eur,cny,gbp,cad,rub,hkd,jpy,aud)
    • Price x (usd,btc,eur,cny,gbp,cad,rub,hkd,jpy,aud)
    • AvailableSupply
    • Volume24h x (usd,btc,eur,cny,gbp,cad,rub,hkd,jpy,aud)
    • Change1h
    • Change24h
    • Change7d
    • Timestamp

    From this point, users/traders/gamblers can begin performing quantitative analysis as we see fit.

    $0.00
      Reply
      [-]
       7 years ago  

      This post received a 3.6% upvote from @randowhale thanks to @cayce! For more information, click here!

      $0.00
        Reply