In order to encourage more community involvement with Steemit.com development, I have worked with the Steemit development team and @pharesim (one of the top 19 witnesses) to organize a bounty for adding an "Advanced Search" feature to Steemit.com. This bounty is to develop the back-end search engine (using ElasticSearch) that will support the addition of this feature to the website.
The deliverable for this bounty is a daemon service, which will continuously monitor the blockchain and keep an ElasticSearch database up-to-date.
Technical Requirements:
- Import all blocks from steemd (except the most recent N blocks) into ElasticSearch using nodejs.
- The block import should be a single command that will start from scratch or catch up if restarted.
- Any third party libraries will need to be approved in advance by the official Steemit development team.
- The application should match the coding style of Steemit.com as much as possible.
- The service must be compatible with the Steemit.com environment, and be able to be installed on the Steemit.com servers.
- The updates to the ElasticSearch database should be done in a reasonably efficient manner.
- It should be resilient and easy to deploy.
The ElasticSearch database must support the following queries:
- Full-text search queries of post text
- Query by author
- Query by tag
- Query by date/time range that the post was created
These are not 'required' but should be supported if possible:
- Query by number of upvotes (range)
- Query by number of comments (range)
- Query by the number of followers an author has (range)
- Query by the reputation of the author (range)
- Query by pending payout amount (range)
- Custom sort orders (newest vs. oldest, relevance, highest payout vs. lowest payout, etc.)
- The ability to query the ElasticSearch database for all the data elements needed to render a "preview post" - Title, thumbnail, preview text, author name, author reputation, category, post time, etc.
- If payouts are to be indexed, that part of the data does not necessarily need to be kept up-to-date in real-time. The active 24 hour posts could be re-indexed every 15-30 minutes, and the 30 day posts could be re-indexed every 1-2 hours.
Bounty Requirements:
- In order to receive the bounty, the developer must implement the ElasticSearch database described above, which meets all of the requirements listed in this post.
- The application must be accepted by the official Steemit, Inc. development team and pass all internal tests and code reviews.
- The bounty must be claimed by January 31, 2017. If the application is not completed by this time, the bounty will be voided. An extension may be provided if reasonable progress can be demonstrated, although this must be discussed with me and approved by me ahead of time if it is needed.
Technicalities:
- The bounty payment will be $300 SBD + the STEEM/SBD portion of the author payout from this post.
- After the payment period for this post closes, I will be exchanging any STEEM from the payout to SBD, using the internal exchange. I will make note of the final bounty amount in the comments after the exchange is complete.
- If any other users in the community add to the bounty, I cannot guarantee that part of the payment unless they provide me with the SBD to hold and distribute as part of the official bounty reward.
- It is the developer's responsibility to ensure that the code meets all of the Steemit dev team's requirements, and make whatever changes are necessary in order for the application to be accepted and published on the live site. I have no control over what the Steemit development team will accept.
- It is the developer's responsibility to ensure that all the requirements in this post are met.
- If you are unsure about anything, please ask.
Final Suggestions:
- It is highly recommended that if you are planning to pursue this bounty, you let me know in advance.
- If you have any questions, you can reply to this post or contact me in Steemit.chat.
- You should seek reviews from the Steemit dev team frequently throughout the process.
If anyone would like to add to the bounty, please indicate so in the comments below. I can hold and distribute any bounty funds as part of the official award, or provide you with a notification if/when the bounty is satisfied. Any funds sent to me would be returned if the bounty is not met.
A huge thank you to @pharesim for contributing the $300 SBD to make this bounty possible! Also thank you to the Steemit developers who took the time to help me put all this together so that we can make this bounty a success!
Remember to vote for witnesses here:
https://steemit.com/~witnesses
Might talk to @kaptainkrayola - I think he did most of this already at steemshovel.com or could do so pretty easily with what he already has. I know as funds dried up he likely stopped working on the project.
I'd happily share any code, elasticSearch Queries, whatever. I'm not all that interested in working on the Steemit.com codebase though after having gone down that road. I'll sign into steemit.chat though and we can talk more.
Ok, cool! I'll reply once I get home tonight.
I will try to track him down on Steemit chat. Thanks!
I have a slack that is used by him and some other friends. I added a link to this post in a #steemit channel on our slack too... so hopefully he'll see it.
Cool, thanks :)
could be like an extensions to @arcange 's steemsql ? :) He already has the ingestion in place...
Using the JDBC plugin for logstash -> https://www.elastic.co/guide/en/logstash/5.0/plugins-inputs-jdbc.html#_usage to hookup his mysql storage of the blockchain?
You are right, all the injection work is done.
SteemSQL is already a full text search enabled database.
It also has language tags on all post and comments
Hey @roelandp - thanks for the suggestion. I'll review it with the Steemit devs and see what they have to say.
Thrilled to see this bounty posted. I've just sent you 100 STEEM to add to the reward in whatever way makes the most sense.
Awesome, thanks @mada!! Do you mind if I convert it to SBD on the exchange? The bounty will likely be open for a while, so that will be the best way to protect against inflation + price fluctuation.
Sure thing - I hereby officially defer to your judgement on how best to use this bounty reward contribution: )
Done :) I have converted it to 11.422 SBD, which will be added to the official bounty. Thanks!!
The Total Bounty Amount is: $329.532
+18.110 (author reward)300.00 (@pharesim) +11.422 (@mada)
Great idea for a bounty, I can't stand the current google search setup - more flexibility in search + having better integration into the site would be great.
I have no experience with elasticsearch though, fingers crossed we've got some guru's on the job now :)
I remember there was @kaptainkrayola with steemshovel and @jabbasteem made steemle , but I'm not sure if they're still with us ..
I'm still here (mostly) and Steemshovel is still live and indexing posts. you can always find me on Skype - kaptainkrayola in case you can't reach me here or steemit.chat
I love this idea. Which we had a bounty platform to manage all of this
Yeah, me too! I tried pushing for it a few weeks ago, but it doesn't seem to be going anywhere.
https://steemit.com/bounties/@timcliff/proposal-for-a-bounty-management-forum
I suggest that using ElasticSearch is overkill for the moment considering the fact that the blockchain is only ~4GB. Even at 10x the size, that easily fits in system RAM on a modest computer.
I'm working on a project right now that takes the blockchain contents and jams it into a standard vanilla SQL database; this will easily provide full-text search for the foreseeable future (until the blockchain gets too big to fit into memory). A distributed search cluster is overkill for the moment.
Hi @sneak - Thanks for the feedback! Based on what you are saying, it sounds like you do not need any assistance from the community on this item. Is that correct?
I think that's probably true, yeah. We'll be open-sourcing all of our services.
This post has been linked to from another place on Steem.
Learn more about and upvote to support linkback bot v0.5. Flag this comment if you don't want the bot to continue posting linkbacks for your posts.
Built by @ontofractal
This post has been ranked within the top 25 most undervalued posts in the first half of Nov 22. We estimate that this post is undervalued by $8.64 as compared to a scenario in which every voter had an equal say.
See the full rankings and details in The Daily Tribune: Nov 22 - Part I. You can also read about some of our methodology, data analysis and technical details in our initial post.
If you are the author and would prefer not to receive these comments, simply reply "Stop" to this comment.
I hope we will see a successful fundraising here with upvotes, too.
This would be an awesome feature without depending a 3rd party (Google).
Agreed :)
I think it is well organized. See you.
Upvoted to add to the bounty!!!
I have a question unrelated to your post
Do you know if there are statistics available to check the percentage ratio of power down vs power up?@timcliff
Hi Snowflake,
On https://steemd.com/ there is a current_supply and total_vesting_fund_steem, which should be the info you need. current_supply is the total amount of STEEM in existance, and total_vesting_fund_steem is how much is being held as SP.
Solid blog. Interesting to see I'm not the only one that is thinking about this. The price of a coin should depend on the quality of the product, the team behind it, if proffesional investors believe in it, and a lot more facets. I found this amazing platform: https://www.coincheckup.com An amazing crypto research site. They researched every single crypto out there. This is truly great.