Being part of Utopian's development moderation team and to help the team be aware of the baseline of the current review time - one of our Key Performance Indicators (KPI) - this author think it best to take a closer look into the moderation (review) time for the development category.
Outline
- Monthly Contributions and Rejection Rate
- Reject Rate per Moderator
- Monthly Average Review Time
- January
- February
- March
- Were There Moderation Abuse?
- Target KPI and Conclusion
Scope of Analysis
The data extracted was from January 19, 2018 to March 2018.
1. Monthly Contributions and Rejection Rate
 |  |
Based on the left chart, from Jan. 19 to Mar 8, 2018, there were a total of 839 contributions reviewed by 22 moderators.
701 (83.55%) of these were accepted while the remaining 138 (16.45%) were rejected.
2. Reject Rate per Moderator
There were 5 moderators who flagged at 100%. While there were 8 moderators who accepted at 100%. In between, the moderator who rejected the most was @mkt rejecting at 53%.
It's easy to accept, but requires guts to reject.
How about sorting this by the number of reviewed contributions?
@ms10398, the top reviewer, flagged at 22%, while @codingdefined flagged at 21%.
Let's look into the details of how long it took to review these contributions.
3. Monthly Average Review Time
3.1. January
For January, the average review time was around 8 hours. This came from 12 moderators reviewing 236 contributions.
The quickest of these reviewers were @stoodkev and @flauwy reviewing contributions within the first 2 hours, while the moderators who reviewed problematic perhaps trivial contributions was @mkt taking an average of 14 hours.
Initially, I thought that the longer review times was bad, but when I started to become a moderator, I understood why some contributions took time to review - it was coming up with the decision whether to accept a contribution or not.
3.2. February
For February, the average review time increased to 32 hours. This came from 21 moderators reviewing 475 contributions.
The quickest of these reviewers were @jestemkioskiem who approved one contribution within an hour, while the moderators who took on the oldest contributions was @scipio approving/rejecting a contribution after 3 days.
3.3. March
For March, the average review time again increased to 40 hours. This came from 13 moderators reviewing 128 contributions.
The quickest of these reviewers were @flauwy who approved one contribution within 5 hours after its creation, while the moderator who took on the challenge to review the oldest contribution was @onos approving/rejecting a contribution after 3.5 days.
From January to March, there was an increase in review time maxing this month with 40 hours.
4. Were There Moderation Abuse?
One of the things moderators or supervisors have to check is moderation abuse - I am familiar or somehow know the people behind the development moderators and see them as moderators with character and I hope they don't take offense in me looking into this data.
Because I had difficulty in choosing and studying a visualization tool that can help detect abuse (a single moderator approving the same account multiple times), I chose the top 3 contributors and see which moderators approved those author's contributions and how often.
The top 3 contributors subjected to analysis were @justyy, @magicmonk, and @stoodkev.
Based on the chart above, there were 6 moderators who approved @justyy's 47 contributions.
For @magicmonk, there were 4 moderators who approved his 27 contributions.
While for @stoodkev, there were 8 moderators who approved his 27 contributions.
Perhaps this is a good indicator of NO abuse - an account being approved by multiple moderators - not just a single moderator
5. Conclusion and Target KPI
Based on the data presented above, we saw that from January there was an increase of the review time of the development category. From 8 hours in January, it increased to 32 hours in February, and is now at 40 hours for March. We also saw that the oldest contribution that was reviewed was past 3 days. This is still above the 2 days (48 hours) review time indicated in Utopian when a contribution is submitted for review.
We also learned that the approval % of the development category currently stands at around 84% the same figure with @abh12345's analysis.
@abh12345 and I may have different sources, but we came out with the same figures - the beauty of analysis done correctly.
And lastly, we've also looked at a potential visual tool to detect moderation abuse. The abuse check done here was meant to be a trial in order to use for succeeding analysis of potential moderation abuse. We also discovered a potential indicator of NO abuse - when multiple moderators approve an author's contributions.
Tools
- Power BI for charts
- Utopian.io local DB (MongoDB) for the data
- Studio 3T MongoDB for exporting data from local DB
{ "json_metadata.type": "development", $and: [ { "created": { $gte: "2018-01-19" } } ] }
I am part of a Blockchain Business Intelligence community. We all post under the tag #BlockchainBI. If you have an analysis you would like carried out on Steemit or Blockchain data, please do contact me or any of the #BlockchainBI team and we will do our best to help you...
You can find #blockchainbi on discord https://discordapp.com/invite/JN7Yv7j
Posted on Utopian.io - Rewarding Open Source Contributors
.
Thank you crokkon. Data analysis is a home i come back to. :)
Yes, I plan to cover that next (other contribution types) this was just a primer to look into possible challenges i may encounter before tackling the bigger challenge - multiple/other contribution types.
Hey @crokkon, I just gave you a tip for your hard work on moderation. Upvote this comment to support the utopian moderators and increase your future rewards!
Statistics and Numbers, It's hard for me to understand, but with your thoughts in it, you made it easier for someone with no background like me, thanks for this post @eastmael
Thanks bro! Feedback like this are very much appreciated. Sometimes if it's jusy us who read what we write, it seems clear and understandable to us but not for others. With this type of feedback we get to know that we're doing something right.
Hey @eastmael I am @utopian-io. I have just upvoted you!
Achievements
Community-Driven Witness!
I am the first and only Steem Community-Driven Witness. Participate on Discord. Lets GROW TOGETHER!
Up-vote this comment to grow my power and help Open Source contributions like this one. Want to chat? Join me on Discord https://discord.gg/Pc8HG9x
Phew!
I'm glad our numbers match up!
Also it's good to see the longest review time around 3.5 days. I delay my weekly report until 4/5 vote cycles have passed to try to give all the contributions in that week a chance to be fully processed.
Cheers!
Hello Asher, yes I do understand the work you put forth in creating your approve/reject analysis. Looking into the approve/reject was not in my original outline, but when I started to transition from the number of contributions to the amount of review time, it (approve/reject %) was the bridge that I had to cross to connect the two points I wanted to tackle. Having a separate (yours) independent report validated our figures. :)
Yes, and I'm glad you crossed the bridge and found the other side in good shape!
Thanks for the mention, your analysis is clear and looks great!
I wasn't able to post this comment. It was just saved as a draft. Lol.
Clarity and greatness - two characteristics i always try to live up to.
heh :)
That happens at times, and yes, two good characteristics to bide by!
Wow! Just.... WOW! Nice job!
Thanks chief! Glad you liked it. :)
such an amazing work,is it took days for you,amazing work,thanks a lot for sharing with us,hope you had a great time.
Thank you saikr. Yes I had a great time doing it.
The perfect analysis, hopefully you can be the best moderator in utopian-io @eastmael,,
Thank you zallina. I'm not aiming to be the best moderator in utopian-io, i'm just aiming to be the best moderator that i can be. :)
Yes @eastmael,, it's a very good wish, I hope you succeed in various things ,,
Awesome analysis @eastmael :)
Thank you codingdefined. Now we have an idea of where we are as whole development moderator team, it's now time to define where we want to go.
Review has been on another level now lots of developers on board
This is the starting point for the whole development moderation team. We'll see where we will go as a team from here.
Great work dear,you have done an amazing job, not an easy one,thanks a lot for sharing with us,have a nice day.
Thank you aaeesha. Yes indeed it's not an easy one but i felt relieved when i finished it.
It's so informative post and great review ....I learned many things from ur post thanks dear for your precious post @eastmael
Thank you masuma. Glad you learned something feom my post. Hope you'll be able to apply what you learned.
Sure dear