lightproject cross-posted this post in Hive Projects 5 years ago


hive2csv Major update

in #hivedevs5 years ago (edited)

Hello Hive Community,

I am super excited to announce a major update on hive2csv public tool. Thanks to some improvements, hive2csv will start the download right away, it will perform much faster and most importantly, queries now have virtually no time limits.

Demo #3


Feel free to try it out and post your feedback if you find any issues or have any suggestion.

Detailed Change log (for nerds):

  • Created delay constant for delays between requests to rpcnode
  • Changed depth (aka rpc request item limit) back to 1k items as per suggestion of public rpc node owner and maintainer (@anyx)
  • CombinedStream starts streaming to API response right away allowing the client download to start immediately and overall greatly improving hive2csv performance
  • Removed unnecessary intermediate step of streaming to local disk
  • Removed batch recovery feature
  • Abstracted away JSON formatting as a router function
  • Moved HTTP response headers formatting to beginning of router block
  • Added constants.js module for better project structure
  • Improved error handling
  • Set CombinedStreampauseStream option to false => Alleviates backpressure on rpcnode delegating buffering on my server side

If you use the service or find it useful, please reHive!

Cheers,
lightproject

Sort:  

Is this tool discontinued?