Have you ever did something completely different that you initially wanted to do? For me this is frequently the case. But sometimes time is not wasted and I am happy to achieve something or learn something new.
Yesterday I logged in to my laptop for half an hour to check steemit posts and ended up with evening session that took place until 2 AM local time writing SSE client with real time feed for all available fiat currencies, CFDs and derivatives from FPForex.ru. This sounds weird for me even today as yesterday I did not know that SSE exist.
I had been working before on witness ICO price feed for Golos blockchain and noticed that solution provided by @someguy123 in Golosfeed-JS uses cheerio js (https://github.com/cheeriojs/cheerio) to extract data from web page tables that contain different instruments' prices.
I was curious why @someguy123 had to extract those data from tables so I logged in to the website and MY EYES HURT. The design looks sooo 90's and indeed in the footer we see "1995 - 2016" the website has been there for around 20 years ;) I soon noticed it includes real time data at http://www.forexpf.ru/quote_show.php. This looks fast. Almost like Poloniex ;) but there is no rest API.
I suspected that this must be some ancient AJAX code I started the debugger and whoops: inside the php page sits an iframe (another web page) generated server side using "JSP" (Java server pages). This thing contains some args after url. Digging deeeeper I found out that there are javascript modules that capture an access token from iframe url address and subscribe to SSE server to fetch data and update the tables.
Then I thought..."What the heck are Server-Sent Events (SSEs)", it is not AJAX, not websockets and found an article here https://www.html5rocks.com/en/tutorials/eventsource/basics/ with a quite good introduction to the subject.
And I came up with a crazy idea to catch this token and subscribe to a (sub)set of available instruments to get live ticker for fiat currencies, CFDs and futures directly from the server. Of course similar data are provided by brokers like IB, OANDA or other ones but 1) this was rather to solve a challenge 2) the website has a few Russian based instruments and currency pairs not easily available elsewhere.
And guess what - I succeeded. The outcome of this reverse-engineering exercise is an experimental tool https://github.com/cryptomental/social-chains-experiments/blob/master/fpticker.py that:
- Opens a keep-alive session and grabs SID token
- Subscribes to any instrument, subset, or all of them
- Streams ticker data in raw SSE format or dict/JSON
The available instruments are:
* INDEXES_CFD = ["DJIA", "SP500", "NASD_COMP", "NASD100", "FTSE100",
"DAX", "AEX" "CAC40", "SMI", "RTST", "USD_INDEX"]
* STOCKS = ["GAZ_ADR", "GMK_ADR", "LUK_ADR", "ROS_ADR", "RTK_ADR",
"SNG_ADR", "TAT_ADR", "VTB_ADR"]
* METALS = ["GOLD", "SILVER", "PLATINUM", "PALLADIUM", "ALUM",
"COPPER", "NICKEL", "BRENT", "LIGHT"]
* RUB = ["USDRUB", "EURRUB"]
* MINI = ["NASDAQ", "SnP500"],
* CURRENCY_CFD = ["AUD/JPY", "AUD/USD", "CAD/JPY", "CHF/JPY",
"EUR/AUD", "EUR/CAD", "EUR/CHF", "EUR/GBP",
"EUR/JPY", "EUR/USD", "GBP/CHF", "GBP/JPY",
"GBP/USD", "USD/CAD", "USD/CHF", "USD/JPY"]
* CURRENCY_FUTURES = ["AUD_FUT", "CAD_FUT", "CHF_FUT", "EUR_FUT",
"GBP_FUT", "JPY_FUT"]
* INDEX_FUTURES = ["DJIA_FUT", "SP500_FUT", "NASD100_FUT", "NIK_FUT"]
Raw stream example for currencies:
1;S=USDRUB;T=14:33:46;B=+64.398;NCH=0.627;NCHP=0.98;A=+64.408
1;S=EURRUB;T=14:33:57;B=-68.458;NCH=0.685;NCHP=1.01;A=-68.478
1;S=EURRUB;T=14:33:57;B=-68.457;NCH=0.684;NCHP=1.01;A=-68.477
1;S=EURRUB;T=14:33:58;B=+68.458;NCH=0.685;NCHP=1.01;A=+68.478
1;S=EURRUB;T=14:33:59;B=-68.458;NCH=0.685;NCHP=1.01;A=-68.478
Metals, stocks:
1;S=ROS_ADR;T=14:34:17;B=+5.330;NCH=0.030;NCHP=0.57;A=5.335
1;S=RTK_ADR;T=14:09:55;B=7.70;NCH=-0.11;NCHP=-1.41;A=+7.74
1;S=SNG_ADR;T=14:34:30;B=4.549;NCH=-0.041;NCHP=-0.89;A=+4.553
1;S=TAT_ADR;T=14:35:22;B=36.46;NCH=0.28;NCHP=0.77;A=-36.49
1;S=VTB_ADR;T=14:35:29;B=2.083;NCH=0.000;NCHP=0.00;A=-2.085
1;S=GOLD;T=14:35:29;B=+1211.71;NCH=-0.52;NCHP=-0.04;A=+1212.01
1;S=SILVER;T=14:27:26;B=+16.64;NCH=0.00;NCHP=0.00;A=+16.67
1;S=PLATINUM;T=13:54:22;B=+936.00;NCH=0.00;NCHP=0.00;A=+946.00
1;S=PALLADIUM;T=14:35:23;B=-743.30;NCH=1.70;NCHP=0.23;A=-746.30
1;S=ALUM;T=14:25:00;B=-1760.00;NCH=0.00;NCHP=0.00;A=-1763.00
1;S=COPPER;T=14:33:08;B=-5592.00;NCH=-20.00;NCHP=-0.36;A=-5597.00
1;S=NICKEL;T=14:18:30;B=+11302.00;NCH=4.00;NCHP=0.04;A=+11402.00
1;S=BRENT;T=14:35:16;B=-49.30;NCH=0.27;NCHP=0.55;A=-49.33
1;S=LIGHT;T=14:34:49;B=+48.20;NCH=0.30;NCHP=0.63;A=+48.23
Dict/JSON:
{'A': '+1212.44',
'B': '+1212.14',
'NCH': '-0.09',
'NCHP': '-0.01',
'S': 'GOLD',
'T': '14:42:52'}
{'A': '+16.67',
'B': '+16.64',
'NCH': '0.00',
'NCHP': '0.00',
'S': 'SILVER',
'T': '14:40:06'}
{'A': '+946.00',
'B': '+936.00',
'NCH': '0.00',
'NCHP': '0.00',
'S': 'PLATINUM',
'T': '13:54:22'}
These are just static snapshots, normally data is streamed real time. Looks pretty cool.
Treat this implementation as an experiment, or a toy, but you can access real time data with it.
This post might be too technical, but well, I am writing about some original discoveries here and actually I am writing this not to forget afterwards. I think I found another application for steemit being a diary ;)
A few references:
- Server-Sent events specification
https://html.spec.whatwg.org/multipage/comms.html#server-sent-events
- What the heck are Server-Sent Events (SSEs)
Very cool man.
Thanks!
Great work on deciphering this! I was never able to see any of this, so I just did some dirty HTML scanning. It works, it's not great, but it does definitely work well.
I should definitely work on porting your Python SSE to NodeJS (I don't actually like Node, but Piston has been iffy in the past few months, while steemjs-lib has been reliable) rather than using my HTML hacks.
The most important is that your solution is stable, therefore there is no need for changing it. But feel free to port it to NodeJS. It seems I need to check steemjs-lib as I did not pay much attention about it yet.