Hello everyone, today I’m releasing v1.2 of both the AskSteem API and AskSteem Web UI which consists of many backend improvements to the AskSteem Core.
What is AskSteem?
AskSteem is the steem search engine, you can find more info about it here.
API Updates
Adjustable Ranking Weights
There are two new parameters available for the AskSteem Search API which are quality_weight and relevancy_weight. These new parameters allow API users to customize the AskSteem ranking algorithm to better suit their needs.
To understand what these new options do I will briefly summarize how our ranking algorithm works. When users search on AskSteem we try to find the results that are most relevant but still high quality. To do this we utilize many different ranking factors that fall into two categories: quality or relevancy. By default we give each of these categories a weight of 1, however, with these new API options, you can choose to weigh one higher than the other. For example, if you wanted to ensure the highest quality results make it to the top you could set relevancy_weight=1&quality_weight=3, or if you wanted more relevant results you could set relevancy_weight=3&quality_weight=1.
While most use cases will not require you to change these values we are making them available for special cases. Additionally, please note that relying too heavily on relevancy will make your search results susceptible to keyword stuffing.
Related Posts API
There is also a new API endpoint available that allows developers to easily find posts related to an existing post by providing its author and permlink.
https://api.asksteem.com/related?author=thekyle&permlink=introducing-asksteem-a-steem-search-engine
It also works for users
https://api.asksteem.com/related?user=thekyle
Blockchain Crawler Updates
In an update to AskSteem Core (the core technology that is utilized by all AskSteem services), we have updated our blockchain crawler. This is the software that syncs our search index with that actual data on the blockchain.
The previous version of the AskSteem crawler worked by periodically scanning old posts to check if the content had changed in any way such as new upvotes or comments. While this worked well when AskSteem was launched the blockchain has grown significantly and this is no longer a viable strategy.
The new version of the AskSteem crawler uses a technique called block linking. Where it scans the head block for activities that link back to previous blocks in the blockchain in order to update the data, such as posts, created in those previous blocks. This is a much more efficient method for keeping our index in sync with the blockchain and will result in more accurate information in search results. We are also working on applying this crawling technique to our user crawler.
Introducing AuthorRank
AuthorRank is a blockchain implementation of Google’s PageRank algorithm. Currently, we are testing it only on user search results but in the future, we will likely use this as a quality factor in post rankings too.
Right now, AuthorRank works by counting each follower of any steem user as a vote recommending that user. However, not all followers are given the same voting weight. Although our actual algorithm differs a good way to estimate the AuthorRank voting power of any given follower is to divide their user reputation by the number of accounts they are following. This means that followers with low reputations or who are following lots of accounts will have a smaller individual vote compared to accounts with high reputation or who are following few accounts.
AskSteem Web UI Updates
Over the past few days, I have completely rewritten the AskSteem Web UI from the ground up. Originally when AskSteem launched there was no API available so the Web UI needed to house all of the components required to run AskSteem Search including the ranking algorithm, logic, formatting, suggestions, etc.
Later I decided to fork large portions of the AskSteem Web UI codebase to create the AskSteem Search API, however, this meant I had a lot of duplicate code between the two services. I’ve had plans to migrate the AskSteem Web UI to using the AskSteem Search API for a few months but needed the API to reach feature parody before that could be done. In the AskSteem Search API v1.1 feature parody was achieved between the two services and I began purging the AskSteem Web UI’s codebase of duplicate code and replacing it with calls to the AskSteem Search API. In AskSteem Web UI v1.2 there is none of the original code used for searching and everything gets passed to the AskSteem API.
Version Codenames
I've also decided to make referencing AskSteem versions easier I will be adding a codename for each major version release which will be a type of fish. For this release, the codename is "Angelfish".
Hey, @thekyle. Upvoted! and resteemed!
Thank you very much for such a great tool. @biophil introduced me to asksteem.com a week ago, and since then, I use it a lot.
Steemit lacks a good search engine. Google search is not good enough. It has many shortcomings. AskSteem is helping a lot to fill this gap. Moreover, AskSteem can serve as an archive for steem. Steemit does not have history. So, you can use AskSteem for that.
However, I think it is not getting the support it deserves from Steemit and the community. I am sure that running such a service is costing a lot. So, instead of being rewarded for your efforts, you are probably spending some money.
AskSteem is great. Yet, it is still not perfect. I have some suggestions based on my experience with it:
1- We need a GUI for advanced search. most steemians are not technical, and they would not be able to use many of the advanced features. So, a GUI would be very much appreciated.
2- Until that is ready, please put a cheat sheet on the site.
3- We need to specify if the search is in a post or in a comment. Sometimes, I want to search only in comments.
4- We need to be able to use (>= and <=) with dates. currently you can specify a fixed day or a range.
5- We need to be able to search by reward amount.
6- We need sorting options. Sort by: relevance, date, votes, author_reputation, ... etc.
Again, thanks a lot for such a great tool.
Thanks for the feedback. An advanced search page has been a commonly requested feature so its definitely on the to-do list. The AskSteem API already supports sorting by specific fields so it should not be too difficult to add that functionality to the Web UI.
Great work you're doing there! Thank you very much! I just wonder how up-to-date the data is. How often does the crawler run and can I use asksteem to implement mention notifications in my project?
As was mentioned in the post we've just deployed a brand new crawler for updating posts. New posts are indexed within 5 minutes of being posted, old posts are updated whenever new activity such as comments, upvotes, etc. occur. Feel free to use our API in your service you can contact me on steemit.chat if you have any questions.
Thanks to provide Great New and Useful Information
Thanks for building this tool. Earlier today I tried a search 'monero' I was expecting to find some interesting analysis of Monero. Instead the results seem to heavily favor the single word Monero titles.
https://www.asksteem.com/search?q=monero
I have developed search before and understand how hard it is, I will hang in there while you perfect it. You might benefit from more user testing.
Thanks for the feedback, in order to implement some of the new API features we had to change our ranking algorithm which is likely why those results are such poor quality, even compared to previous versions of AskSteem. For the next update, I'll focus more on improving that.
Ok thanks, Ill keep it in mind when I think of something to search.
Just a heads up, I've pushed out a quick update which should make results better for one term queries, see here: https://www.asksteem.com/search?q=Monero
Wow, that is much better thanks :)