Three Boxes

[dpacaba]

Hello:

I am just wondering about the boxes.

In order to get a "Proven" status, your average message rating should be 5 stars; and in order to get "Trusted" status, your average message rating should be 3 or 4 stars.

How do we get "Active/Enthusiast" and "Regular/Veteran" status?
How many post should we contribute in order to be an "Active/Enthusiast"?
How many times should we log-in in order to be a "Regular/Veteran"?

Thanks in advance

[Molly K]

Debonair -

I'm going to reuse some of the post that I posted to John Rich in the Welcome to the NI Discussion Forums thread...

One of the features we really liked about this new tool was the flexibility that we have in the ranking system. We really like that users can earn recognition faster than was possible before and that there are multiple steps that can be earned along the way.

We chose three metrics that we felt would help members earn the recognition they deserve for their participation in the community. We set three levels for each of the metrics and have room to add higher levels when they become necessary. If you do the math, that makes 27 possible user ranks -- three metrics and three levels per metric. This is a significant change from the 2 ranks we used to have!

The first metric is ratings. The three levels are unnamed, Trusted, and Proven.

The second metric is posts. The three levels are unnamed, Active and Enthusiastic.

The third metric is logins. The three levels are Member, Regular, and Veteran.

As far as disclosing the actual thresholds for each of the metrics, one theory is to keep them a surprise to prevent users from generating false activity simply to change their rank (or someone else's rank using the ratings metric). This also gives us the opportunity to change the thresholds as the community matures, etc. The other theory is that if we don't disclose the thresholds, community members will dig around until they find them and potentially generate false activity to confirm their suspicions.

So, let's take a vote on whether you think it is best for the community thresholds to be a lovely surprise when you achieve them or whether you would like to know where the bar is set so that you can try to achieve it. The poll will be available from the Feedback on NI Discussion forums Board Page.

Thanks!

[altenbach]

---

@Molly K wrote:
As far as disclosing the actual thresholds for each of the metrics, one theory is to keep them a surprise to prevent users from generating false activity simply to change their rank (or someone else's rank using the ratings metric). This also gives us the opportunity to change the thresholds as the community matures, etc.

---

Molly,
It is of course quite a concern to prevent ...
(1) serial posting just to increase the post number.
(2) Voting clans, e.g. user A and B each artificially vote up each others posts.

I am sure some statistician can come up with elaborate formulas to prevent this. Some simple suggestions.

To prevent (1), the rating should be normalized to the total number of posts. Two users with 50 ratings of 5.0, one having 50 posts and one having 500 posts should get a different score. It means that the second users has an artificially high amount of relatively low quality posts, possibly caused by a desire to increase the post count, while the first user consistently produces high quality and informative posts. The first user should probably get a better rating.

After posting, there should be a 30 minute window where the post can still be edited. This prevents second posts due to e.g. forgotten attachments or simple oversights.

Some other ideas:
- Add weighting to each vote. Five stars from a top contributor should count more than five stars from a newbie poster that blindly votes up every answer given.
- Limit the number of votes that can be cast per day and per member.
- Hide the rating for each post (make it only visible to the poster itself), and condense the overall ratings within a thread into an thread vote. This way, the most interesting threads still stick out.


On a side note, there is still considerable confusion on the rating system. Mathematically speaking, we don't know if the number of stars should be considered an U8 or an I8 number.
Are all rating positive (U8) or should 1 and 2 stars be considered negative with 3 stars neutral?

I still believe, that even a single star should be considered positive. 🙂

[Ben]

Please do not limit how many stars I can give out. On busy days, the stars may be my only contribution to the Exchange.

Molly, did the rating count get reset when the exchange was converted? I thought I had given out thousands of stars by now but my profile shows less than 200!

Yes some poking around will give you a good clue as to were the thresholds are. In an environment populated by engineers and scientist, it is only a matter of time before determines the inverse function.

I am guessing that ther quite a big step between Active and Enthusiast.

It seems like the 500 and 1000 response thresholds should get some recognition.

>500 "Very-Active"
>1000 "Extremely-Active"

Dennis has already exceeded 3700 and is closing in on 4000 soon!

so...
>3000 "Super-Enthusiast"
>4000 "Grand-Enthusiast"

Here is an example of a good thread.

http://forums.ni.com/ni/board/message?board.id=170&message.id=98484

I agree that 1 star is good.

Re: Veteran
How come there are "relative rookies" that have only bee around for a year or so showing up as "Veterans" while the "Old-Timers" (i.e. the people who have caried this exchange for years) are only recently acheiving "Regular" status (please insert your favorite smiley face emoticon here)?

Re:Voting blocks
I beleive I have seen these come and go over the years. In the long run I do not think they make much of a difference.

Re:Giving out stars
I propose that the ratings be modified such that the assignment of star values is multiplied by a scaling factor. These scaling factors could be as follows.

For each of the three boxes a user is assigned a numeric value ranging from 1-3. The users rating is then averaged and used to multiply the star rating.
eg #1
Newbie
un-named 1
un-named 1
member 1
total 3

avg 1

assigns star rating of 5

multplying, the post get 5 stars.

eg #2
Old timer
Proven 3
Enthusiastic 3
Regular 2

avg 2.7

assigns rating of 5 stars
multipying the post gets 13 stars!

Note:
The original poswter of the Q should get an automatic X2 multipler. They are the best people to judge if their Q has been answered.

So...
Please let me "rate away". I think it;
helps me find good questions
flags others to interesting Q's
Flags good A's
Encourages contributors both old and new
Lets me contribute without getting myself into a rats nest of follow-up Q's.

Please share your thoughts,

Having fun,

Ben

PS
To those Old timers that have been here for a while, let me stop and thank you all for your contributions here on the Exchange. This is as close to "sitting at the feet of Plato" as I will ever come. Without your work and contributions this exchange would be far less than it is.

THANK YOU ALL!

[altenbach]

---

@Ben wrote:
I agree that 1 star is good.

---

Opinions on this clearly differ, how else would you explain the star for This thread? It clearly look like an intended "punitive" rating to me.

Similar to your suggestion, we could remove all doubt by defining the rating as total number of stars (A post with 3x"5 stars", 2x"4 stars" and 1x "1 star" would show up as "24 stars" (if I did the math right...).

So, how should we deal with posts that deserve a negative rating one way or another ?

- Spam -> Click options..report abuse to moderator.
- offtopic post, thread crapping -> same.
- Incomprehensible post -> Just wait, somebody might actually understand it and reply.
- Completely incorrect answer -> Reply in thread and give better answer! 🙂

The menu item "report abuse to moderator" should be a bit toned down. Abuse is too strong a word. It should be simply a "Bring to the attention of a moderator" or similar. I have not tried that option, but hopefully, there will be a text box to voice the concern about the post in more details. This option should also be used by the author to request an edit of a critical unintentional mistake (=definitely nothing to o with "abuse").

[Dennis_Knutson]

You are right - it is inconsistent. I always compared the forum rating to the ratings you can give on the instrument driver or development library pages. There, you can give a 1 to 5 rating and and they are clearly labeled poor to excellent. Adding this type of header information would makes things a bit clearer.

[Dennis_Knutson]

I just wanted to add something that I just realized. If original posters are given large number of stars for their question, it changes their status to trusted. I've seen some posts with pretty basic questions but a five star rating. It would seem that only answers with a high rating should be calculated to give someone that status.

[Ben]

I do not think the ratings of Q's should be applied to the first bar.

I do think that we should be consistent in being in consistant!

After thinking about a bit, I do not think there is anyway to switch from the old method to the new.

All of the old ratings would just have to be thrown away. It would be like monitoring a voltage and logging it for years. The old meter goes bad, then you replace it with another new fresh out of cal. Turn it on, and it starts giving you values that differ from what you saw on the old. Because the old one is broke, you can not compare it's values with the new so it would be impossible to say if the vlaue being measured has changed between when using the old meter and the new or if the old meter was way out of cal and the measure values is still the same as it was before.

Because we have no way to "calibrate" the old ratings, there is no way to merge them with a new system.

You end up having to throw all of the old one away. I do not like the idea of all of my old stars being thrown away.

So...
I suggest we stay with the inconsistancy and let users decorate Q's and A's anyway they want.

Ben

[Molly K]

altenbach -

You have some very good ideas about the ratings and rankings. Two of them seem to be most feasible and could be included in this community in the future: the ability to weight ratings from advanced users higher than ratings from newbies, and the ability to edit posts for a specific time period after they are posted.

As promised, I did go and look at other active forum communities -- technical and not -- to get an idea of what the standard was for the ratings scale. There were several communities that had very complex ratings structures that do not match the architecture here, so I have excluded their schemas. Here are some of the definitions of the scales I saw out there:

5 - Exactly the right information
3-4 Lots of helpful information
1-2 Points in the right direction, but still need more information

N/A: The answer was simply a point of clarification to my original question
1-3: The answer didn't really help answer my question, but thanks for your assistance!
4- 7: The answer helped with a portion of my question, but I still need some additional help!
8-10: The answer has solved my problem completely! Now I'm a happy camper!

So, it does seem that the lower ratings should be seen as encouragement for users rather than as negative feedback. There are still users who will take offense to or be discouraged by poor ratings they receive. For this reason, constructive feedback should probably be offered in a reply to the thread rather than an unexplained low rating.

Thanks for all of the suggestions!

[Molly K]

Ben -

I don't think it is actually possible to limit the number of ratings users can pass out right now, so I think you are safe for the time being.

The ratings you have received were migrated, but the ratings you have given were not migrated which explains why your count was reset.

You are right that the step between Active and Enthusiast is a big one. Forum theory suggests that steps should be exponential to give the community room for growth and to truly differentiate the users. We haven't quite implemented an exponential step, but we wanted to have the most elite users stand out amongst the rest. Adding levels in the middle is an option, but we will give the community some time to get used to the upgrade from two levels to 27 levels before we start adding more! 🙂

The Regular and Veteran levels are achieved by the number of times you have logged in to the new community. The data for everyone was reset to zero at launch (Nov 19th). As the "old-timers" continue to carry the community along, they will reach the veteran status shortly. If there are some new users who have been visiting the community a lot to check in, they are going to reach that veteran status as well. The trick is that it will be hard for the new guys to catch up to that Active or Enthusiastic level! 🙂

I'm glad you are having fun, Ben!

I second Ben's thank you to all the "old-timers" out there and extend a big thanks to him as well!