[Utopian Stats] Analyzing Utopian Moderation Time - Blockchain Business Intelligence

eastmael(68)

Published in

#utopian-io

Words

759

Reading

4 min

Listen

Play

This analysis aims to give an overview of the average review time of each of the Utopian categories.

This also aims to inform Utopian moderators of the average review time of each of the categories to act as basis for them should they choose to set Key Performance Indicators (KPIs) for their teams or to the project owner to set a KPI for the whole Utopian moderation team.

Due to the amount of data that need to be analyzed and the time required to come up with these reports, this author decided to split this into multiple reports. With this report providing an overview.

This report analyzed all categories as a whole - the number of contributions, the % rejection, and the average review time.

Outline

Contributions Overview
1. Monthly Contributions
2. Accepted vs Rejected
Rejection Rate
1. Per Category
2. Per Moderator
Monthly Average Review Time
Conclusion

Scope of Analysis

The data extracted was from January 19, 2018 to March 12, 2018.

1. Contributions Overview

Monthly Contributions	Accepted vs Rejected

Based on the left chart, from Jan. 19 to Mar 12, 2018, there were more than 24000 contributions reviewed by 121 moderators.

Around 40% (9679) of these were rejected while the remaining 60% (14602) were accepted.

2. Rejection Rate

2.1. Per Category

The table on the left shows the reject rate per category sorted by number of contributions.

translations, the top-most contributed category, before it was temporarily paused, was at 23% rejection
ideas (or suggestions), the second most contributed, was at 57%
While graphics, the third most contributed, was at 53%

Referring to the data on the previous section, the rejection rate of 23% for translations was below the overall average rejection rate of 40%.

This can either mean that translation contributions were high-quality that 80% were accepted or a possible leniency in moderation

2.2. Per Moderator

Based on the left table, @onos, @tykee, and @andrejcibik led all moderators with 1708, 1435, and 1031 contributions reviewed, respectively. Their rejection rate were 50%, 66%, and 54%, respectively.

The contributions they reviewed mainly came from the ideas and graphics categories.

3. Monthly Average Review Time

Based on the chart above, the average review time across all categories was 33 hours. For January it was 33 hours, for February 34 hours, and for March 27 hours.

The fastest review time was 26 seconds for this contribution while the longest review time was 545 hours for this contribution .

545 hours (23 days) was a bogus review time which prompted me to look into these contributions that took this long to review. Here were examples:

Contribution	Category	Review Time (Hours)	Last Moderator	Status
Behavior Analysis: #cebu for Cebu community(January 2018)	Analysis	545 (23 days)	@knowledges	Rejected
How To Build AJAX Based Navigation list in ASP.NET	Tutorial	507 (21 days)	@espoem	Rejected
NIGERIANS EARN FROM MUSIC THROUGH DSOUND	Visibility	488 (20 days)	@knowledges	Rejected

There were 83 contributions like these which I think were related with cleaning up the Utopian feed.

This prompted me to add the filter:

Remove all contributions whose review times exceeded 6 days or 144 hours

Here were the stats side-by-side for comparison:

WITHOUT filter	WITH Filter

It didn't make much difference. Here were the categories that had these cleanup activities which affected the review times.

The average review time across all categories is still 32 hours or 1.3 days.

4. Conclusion

Based on this analysis, we saw that there were more than 24000 contributions reviewed by 121 moderators with an average of 32 hours review time from January 19 to March 12, 2018.

These contributions had a 60% acceptance rate (and consequently a 40% rejection rate) with the translation category's higher at 77%.

We also looked at the rejection rate of the top 3 moderators with each moderator rejecting more than 50% of the contributions they reviewed with the rejected ones mostly coming from the ideas and graphics categories. This indicated poor quality submissions in these categories.

Afterwards, we also looked into the detail of the moderation review time and saw that the quickest review time was 26 seconds and the longest reaching 545 hours (23 days). We learned that the latter with review time exceeding 144 hours (6 days) were related with Utopian feed cleanup by supervisors.

Tools

Power BI for charts
Utopian.io local DB (MongoDB) for the data
Studio 3T MongoDB for exporting data from local DB

db.posts.find({created: { $gte: '2018-01-19' }}, { 'json_metadata': 1, created: 1, author: 1 } )
db.posts.aggregate([
    { $match: { created: { $gte: '2018-01-19' } } },
    {"$group": { _id: "$json_metadata.type", 
                count:{$sum: 1} } },
    { $sort : { 'count' : -1}}
])

^{I am part of a Blockchain Business Intelligence community. We all post under the tag #BlockchainBI. If you have an analysis you would like carried out on Steemit or Blockchain data, please do contact me or any of the #BlockchainBI team and we will do our best to help you...}

^{You can find #blockchainbi on discord https://discordapp.com/invite/JN7Yv7j}

Posted on Utopian.io - Rewarding Open Source Contributors