The rating system

[LaLia]

Can we talk about the rating system for a moment?

It just struck me again that having "only" 3 points is kind of silly. I don't really want to give anyone a zero—or simply not rate the post at all. So, everyone deserves a 1 if I’ve read it, even if I don't think it's very good. A score of 3, on the other hand, is for good stories, which means everything else inevitably has to get 2 points.

I would prefer a scale of up to 4 or even 5 points.

1 point = read it, thanks for posting, plenty of room for improvement (school grade 4–5)

2 points = quite good, but still has some flaws (school grade 3)

3 points = very good, engaging/arousing, but with minor drawbacks (school grade 2)

4 points = perfect story

[AdmiralPiet]

If it is not to your liking at all why not just not rate it?
Giving someone a +1 when it is defined as "not good" is functionally the same as a 0

1-2-3-4 has the same range as 0-1-2-3

I understand where you are coming from. At times I wished for a finer graduation.
But changing it, would also throw all scores up to now out of alignment.

[Shocker]

There are days when I wish for a bit more of differentiation , but that’s usually contest entries, when all the authors have given their best effort anyhow. At the same time I’m very liberal with my assigning full score, as this is an encouragement for authors to write more. Bizarrely this leads to my usage of the lower rankings much more often to well established authors here. You folks simply have set the bar for what an excellent story out of your hands looks like much higher than the typical first timer.

I can see @AdmiralPiet ’s point as well, it would skew existing stories. But quite honestly there is already quite a bit of skewed ranking from the starting phase of the forum. There are some excellent stories hidden in the depths of our archive that have received barely any rating, simply due to the amount of members who have seen them. Occasionally one of those stories is getting unearthed, and suddenly receives a good amount of high ratings.

[Lucius]

The single greatest problem of the RA rating system is that it's woefully underused, simple (silly?) as it is.

If it is not to your liking at all why not just not rate it?
Giving someone a +1 when it is defined as "not good" is functionally the same as a 0

1-2-3-4 has the same range as 0-1-2-3

Absolutely. And nobody (and no story!) is perfect, even among those who like it hot...

[Interception]

I think Lia's argument is correct. Giving 3 points really prevents good differentiation.

The statement that 0-1-2-3 should be used in the same way is, to be honest, completely wrong. The 0 is just as bad as not being read. Then 0 would also have to be selected and displayed as a 0 rating. Lia described why at least the 1 deserved everything.

[AdmiralPiet]

I think Lia's argument is correct. Giving 3 points really prevents good differentiation.

The statement that 0-1-2-3 should be used in the same way is, to be honest, completely wrong. The 0 is just as bad as not being read. Then 0 would also have to be selected and displayed as a 0 rating. Lia described why at least the 1 deserved everything.

You are right, I could still give 0 in a 4 point system, so we indeed increase to a range of 5.
But will that really chnage that much?

You also have to consider the fact we don't have an objective or unified way of rating.
A zero can mean: Not read at all.
Or: Read it, but hated it and didn't want to award any points
Or: Read it, liked it, forgot to rate
Others might go with: I will award 1 point if I read it to the end, others will still give zero because they think it isn't very good.
Then again people like Shocker will judge authors differently, being more lenient with newbies.

Adding 1 point to the range will not change that much.
To really make a difference we would have to increase further and give some guidelines to make it more consistent.
Like Newgrounds perhaps: there you can award 0 to 5 Stars in increments of 0,5. So 11 grades. And "unrated" is a different state, it is not at 0 automatically, you have to actively award that.
Beside the stars there is a picture of the page mascott, with changing expressions.
Glowing red and looking angry for 0 stars, starting to smile at 3,5, and beaming with joy at 5

[Claire]

I think this topic would actually be better placed in questions and feedback. This is not writing related.


But to address the actual topic. Adjusting the rating system right in the way it was suggested here is a very bad idea.

First, there is the obvious reason that @AdmiralPiet mentioned: If you suddenly allow for 4 and 5 point ratings, you devalue all of the existing ratings. The only way to address that would be to scale up the current ratings by multiplying all existing ratings by a scaling factor that reflects the new maximum rating. That scaling factor would be 4/3 for a maximum of 4 points and 5/3 for a maximum 5 points.

Here is the problem though. You can comfortable scale up the aggregate rating of a story. Let's say a story has a rating of 30, then 30*5/3 becomes 50. But what do you do with the individual ratings that make up the aggregate rating? An individual rating of 3 would become 5, so that works. But what about 1-point ratings? 1*5/3 is 1.67. Do you round that up to 2? Then three 1-point ratings become a total of 6 whereas a single 3-point rating is only 5.

So the problems you generate with this, you can only avoid if you increase the maximum rating to multiples of 3, i.e. 6, 9, 12, etc. Then you get nice scaling factors of 2, 3, 4 etc. that you can apply to aggregate and individual ratings alike without triggering any inconsistencies.

You would also have to scale all the reputation power values and the threshold values for when a story becomes popular/favorite and when a user becomes a Pillar with the scaling factor you choose.

So that is the obvious problem you get. For practical purposes, the only realistic suggestion is a scaling factor of 2, making 6 the highest possible rating instead of 3. Even that I would caution against because the idea behind the suggestion fundamentally misunderstands how the rating system here works.

The forum very deliberately does not use a 5-star or 10(0) point rating system like Literotica or a site like Metacritic does. The aggregate rating of a story is not the average of the individual ratings but the sum total. That means that our system here is similar to likes on a YouTube video and less like a 5-star raing system on Literotica. The system does not say:

1 point: bad
2 points: ok
3 points: good

like the ratings on Literotica do. The ratings here are likes, or kudos if you prefer the AO3 terminology. And we added a minor twist that you as a user can give a story more than one like, up to three likes in total. That is how you should view this system.

What is the reason for that? Think about the implications that a rating system based on average-rating has versus a like-counter which is what we use.

How do you define popular stories now? Literotica does that by saying that a story is considered Hot if it reaches an average rating of 4.5 AND has at least 10 ratings. This has several consequences:

First: a story can lose its Hot-status again if enough bad ratings come in. That would be a huge issue here because a moderator has to move a story manually to the popular/favorite board. So what do you do when a story is no longer popular after getting a few bad ratings? Move it back and forth over an over again if it keeps changing status? What happens to the whole "This story achieved a rating milestone"-practice of shouting out authors and their stories when they make it to popular/favorite status? You can't really do that if they might lose that status again once a few additional ratings come in.

Second: Literotica has essentially only one good rating. Even a 4 out of 5 stars rating, which sounds good on paper, effectively just draws your story below the 4.5 average rating threshold you need to reach for a story to be considered Hot. The moment you base your rating system on averages, you also effectively introduce negative ratings. If you highlight very high average ratings, basically every rating below that threshold becomes negative.

Third: A rating system based on averages doesn't work in a community space like ours. On a site like Metacritic, the creators of the media being rated don't interact with each other and don't rate each others' art. We do. Here, we have authors rating the stories of other authors who they then interact with and expect to be rated by themselves. Think through what that implies.

Take @sinfulwords story for example. At the moment, her story is sitting on the edge of becoming a popular story with a rating of 28. In the current system, nobody can take that away from her. In a rating system based on averages, she could be sitting just at the threshold of getting a popular story only for me to swoop in and hit her with a negative rating that ruins that. The immediate effect of that is a very frustrating experience.

But the fallout of that is even bigger. Now she gets informed that @Claire came in last second and ruined her precious average rating. Out of resentment, she might now downvote my stories. So to deal with that, you would now have to remove notifications for ratings and make them anonymous. But now you get rid of that experience where as an author you get a notification and learn "Oh hey, this person liked my story. Cool!" And sure, you probably would have prefered to get a 3-point rating instead of a 1-point rating, but that one point is still good, it still makes the like-counter on your story go up, it still gets you closer to popular/favorite status.

So what I'm saying this: The suggestion we're discussing here is reflective of what you would like to do in a system based on average ratings and not in a system where the rating is a like-counter. An average-rating based system is the equivalent of pouring fire on a community space like this. It only works in a space where you have consumers rating the things they consume and you have little community interaction. In a system where creators rate each others' creations, an average-rating based system is nothing but a machine to create resentment between users that will ultimately undermine the sense of community that's essential for a place like this.

Which brings us to the final point. Why not increase the number of maximum likes from 3 to 6 without switching to an average-rating based system?

First of all, if this was a system based on a one-like system like YouTube or AO3 or other social media plattforms, would you even consider the idea to give a user up to 5 likes per story? Probably not. The idea of a 5-point system is inspired by average-rating based systems. But let's look at what that would change.

The higher you increase the number of likes a user can give one story, the more do you increase the importance of the individual rating. Take for example one of our community contests that is decided by rating and not by a poll. In the current system, a single user's rating can have at most the same effect as the ratings of three other users. If 4 people gave a story three 1-point and one 2-point rating the story would have a total rating of 5. Even if you give your maximum number of likes to another story, you can't outvote four other people with your one vote. If the number of kudos you can give is increased to 6, you might be able to do that. In general, the higher you put that ceiling, the more do you emphasize the importance of how much people like a story versus how many people like a story.

The rating ceiling of 3 is trying to strike a balance between how much people like something and how many liked it. In a simple one-like system like on YouTube you'd only count how many people liked something. In an average-rating based system you only count how much people liked something on average. In our three-like system, you try to balance the two extremes. The higher you raise the ceiling on the number of possible likes per story per person, the more do you emphasize the how-much factor over the how-many factor.


I would not change the established rating system for the simple reason that changing the established system will cause more problems than any benefit it might bring. Think of the ratings here as a number of likes a story received where a user can give up o three likes. don't interpret the individual ratings as scale like a traditional 5-star rating because the forum doesn't treat it that way. The suggestion that was made here comes from confusing our system with an average-rating based system that, if introduced, would ruin this community by breeding resentment and revenge ratings. And make use of the possibility to not rating something at all if you truly don't like it. Whenever you give something 1 point, you do express that you like it. You do help that story get closer to becoming a popular story. No normal user on the forum can distinguish whether the 3 points on a story came from 3 users giving 1 point or 1 user giving 3 points. That is not a bug, that is a feature. So be aware that even your lowest possible rating still expresses "i liked this" and treat it accordingly.

The purpose of the rating system has never been to assess the quality of a story. At best its a popularity measure. But even that is secondary. The purpose of the rating system is to facilitate interaction, to give authors a bit more engagement, to create milestones to thrive for, to make sure that popular stories are not forgotten by giving them a place where they remore visible than on page 9 of the public stories board. The rating system's purpose is not to say "My story got a rating of 38, and this other one only a rating of 32, so my story is better." We are not Metacritic, meaning we are not trying to provide a service where consumers rate things to inform consumers about which story is the best/most popular.

As @Lucius correctly said: the rating system's problem is not a lack of differentiation, but underutilization.

[Vela Nanashi]

I can say that on KB and RU the forums we sort of came from, there was a lot of hurt feelings and toxicity from having negative ratings, as I became admin it was one of the first things I turned off. It does not remove the ability to be enemies, but it does remove the forum systems from that. I also as part of upgrading the rsting extension we use here ripped out all the code related to negative ratings, and I personally like that we now have a linear counter up to 3 likes per person and story thing going on. I do not feel like it needs more range, if you dislike story do 0, if you don't and got some enjoyment on some level give it a 1, then if it was better than that give it more also don't forget that comments are in some ways much better than ratings, but both have value for how the forum is designed to work.

Now I think that if we were to despite all that increase the range, it could be handled by allowing upgrading and not changing how many points a person can use, but I would not want to do that change until it is easy for users to see everything they have rated and make it easy to go to said thing and rate it more, that is on the vague todo list I hope to get to at some point in the future but evem when I do I still think 3 is a very nice number

[AdmiralPiet]

@Claire
Damn, you are right. I never thought about it being aggregate and not average.

Most aggregate ratings have one or two points.
One: Kudos on AO3, Likes on Twitter, Tumblr, Instagram and Facebook
Two: Like/Dislike on Youtube, VoteUp/VoteDown on HentaiFoundry

Those with more points do indeed go for an average, like Newgrounds or Literotica.

With that in mind the whole thing shifts even more towards: "Keep it as it is"
As far as I see:
For the forum to funktion as originally intended we want to stay with the aggregate-system over an average
We don't want to have negative ratings

If we change:
Old ratings will be devalued
There is no consistent rating anyway. Even with guidelines.
Effect is dubious
The hassle of implementing it and correct old ratings

In addition: If we see this as a "Like"-System you really can like something twice. Thats a thing people on other platforms sometimes wish they could have.

[sinfulwords]

Personally I like the rating system as is however I agree with what @Lucius said: I think it’s under used. But so is every other like-button across the internet Again, personally, I never saw the rating as grades, I inherently saw them like @Claire described: as likes.

Claire went on to raise several good points on why changing the existing system isn’t the best nor easiest idea and I just so happen to agree with her