SteemShovel.com
Disclaimer : This is pre-alpha, in-development software so expect it to crash and/or be unavailable from time to time while I continue to work on it. Thanks for understanding :)
I'm excited to announce the very first rough draft of my Steem Search engine - Steem Shovel. I was frustrated with not being able to find older posts on Steemit so I set out to build my own search engine. I have some experience with ElasticSearch and it seemed like a great fit for indexing Steem posts and comments.
What's it do?
SteemShovel runs continuously indexing new content from new blocks. It will index both comments and posts and make them full-text searchable. Moving forward I plan to allow you to limit your search to specific authors and also restrict searches to certain tags. Right now it's just a full on mega search against all of the fields/text.
What powers SteemShovel?
The underlying application code is written in Node.JS while the search engine is powered by ElasticSearch. I use the SteemJS module (https://github.com/adcpm/steem) to talk to the Steem process.
The application is load balanced using AWS EC2 servers behind an Elastic LoadBalancer. This gives me the ability to add more servers / horsepower at will if anyone decides to use this app.
Why can't I find the post I'm looking for?
The indexing process is currently making its way through all of the Steem blocks. A second process is indexing new posts as they appear but there is a big gap between the two. Because of this it might be a while before you are able to find every single post you're looking for. Hang tight, they're coming!
What are some improvements coming to SteemShovel?
Well, I'm glad you asked! I have a pretty ambitious set of goals for this project so we'll see how it goes. The main focus now is to work on the indexing and build some more filters that you can use to search. Beyond that I want to improve the look of the site and make sure it's working well on mobile.
Once the basics are covered I have tons of features I want to work on - log in, voting, UI improvements to show threads of comments like Steemit does.
But wait, there's more!
I have also built this to be an API that anyone can talk to. All you have to do is direct an http client to https://steemshovel.com/search/?search={your search term} and you'll get some JSON to do with as you please.
If you have any questions or suggestions as to what you'd like to see this thing do, please comment and let me know.
@kaptainkrayola thanks for the heads-up! This is one of many steem tools.
@roelandp is making a collection of them at http://steemtools.com/
Keep it up!
I'll get the form filled out there to get listed directly. Thanks!
Great tool dude!
I've listed it on SteemTools!
It's fantastic that you have added a JSON api, this is really helpful!
Supercool that you are indexing...
Feature request, undoubtly already on the list:
Hats off! Applause!
This is much needed! Thank you!
Alright - you totally deserve the upvote and this comment.
Great job providing for the community. I have no doubt you will make:
You rock man!
Great!
Thanks for the support everyone. I think I'll use the money from this post to get a proper web designer to work on the site as well as some contracting hours from another node.js dev to help build out some more of the features.
It already looks pretty great IMO
Just W O W
Thank you so much for your hard work. Hope you're appropriately rewarded. Come on people, get behind @kaptainkrayola!
I do love your enthusiasm. Thanks for the support!
The built-in Steemit search is diabolical! You are providing a much needed service. Huge respect and gratitude.
Very nice idea, thank you for sharing
I love how fast it is, and the results seem to be pretty good. Kudos on the implementation so far.
I only have 1 minor feature request, which is to have the ability to show only results from posts in certain time period (ie. last 24hr, past week, past month, past year...).
I'll add some of those filters to the to-do list for sure. Thanks for the input
way to go! good job! just what i need!
Wow well done! I hadn't even considered this angle yet. We may have to have a chat at some point, I could see some of my projects pointing to steemshovel as their search provider :)
I'd love to collaborate. I'm on skype - same username if you want to chat.
ROCK ON!!! Way to go! Can't wait to try it out!
This thing is lightning fast. Great job.
Thank you. I'd love to take all the credit for that but a lot of it is just ElasticSearch is a fantastic tool.
Damn, really impressed with the speed and seeming thoroughness of the results. I keep throwing odd search terms at it to see what weird stuff it digs up...sorry.
Nice. Thanks :D
This is a fantastic example of giving back to the community kaptain. You definitely deserve some upvotes for all of the hard work and expertise displayed here. Great job, we all hope to see more soon!
Hello! I am @upvotes and I am here to help you get upvotes!
Just upvote this comment and your post automatically becomes an @upvotes candidate for: @people, @places, and @headlines
Several times a day the most upvoted candidates become the Best of Category post
Very good job!! I suggest you to add ordering by votes on list.
https://steemit.com/steemit/@minion/youtube-vs-steemit-enemies-or-siblings
[IMPORTANT] This post covers the Similarity between YouTube and Steemit as well on How to use Steemit and It's
OFFICIAL CONTENT RULES.
This is a great idea. Nice work!
This seem very cool ,
I support the idea, it will be a useful tool for sure.
that logo however looks like a face:
The logo seems pre-alpha as well...
I'm open to suggestions on the logo so if you have something better, send it over.
So...are there any list ranking? Meaning, does it place the most recent post on the top of the list, or are there other factors involved to get to the top..like with Google
oh men. what are you try to carry with this steemit shovel? raking all the monies in the communty alone. lol
why today is down ?
Good job guy! Exactly what we need!
http://www.google.com
Good initiative
Woop woop
Thanks
@bullionstackers new post
The growth of this Network starts to scare me :D !
Amazing Job, Kaptainkrayola !
I wish you the best of success!
Nice initiative. Very needed indeed.
IMO the most badly needed feature is the ability to browse user names and posts' tags.
Right now, it's impossible to find a post you know it exists, if you don't remember the exact spelling for the name of the author or the post's tag name.
That's on the list of things I want to add as well. I'll be posting more updates as I add new features so keep an eye out
Good news! we really need it. Google search is awkward.
That is cool you will be able to find my post about the machine gun festival I went to.
https://steemit.com/guns/@steve-mcclair/the-big-sandy-shoot-festival-for-machine-guns-heavy-artillery-and-gun-enthusiasts
Somebody better call Mike Mulligan!
This is wonderful! How do you guys develop apps on Steemit? Where are the developer resources or did you figure this all out on your own?
i think this will be one of most usefull tools around for an average user! gj
Wow, this is so cool. We are running our own blockchain project called ElectriCChain and this means that if we make a heap of categories and articles on steemit then we can leverage your API tools for our own project?
Sure, I don't see why not. i'll be adding more granular search options to both the site and the api so there's no reason you couldn't use it for that.
Thank you for using @upvotes! I have submitted your post to be a Best of Category candidate in @people, @places and @headlines
Feel free to upvote and add comments to your own candidate but more importantly please upvote exactly one other candidate to help improve quality, your ranking wont be lowered by this.
Just reply to this with a comment starting with a specific @category to become a candidate