396 posts
Mini-Profile Background: {"image":"","color":"1d81bf"}
Mini-Profile Name Color: 1c2833
Mini-Profile Text Color: aed6f1
|
Post by Cheeseman on Oct 27, 2015 2:55:45 GMT -5
We've had topics about needing more level judges, what the mods and admins should be doing in the level suggestions, what level judges should and shouldn't be doing etc. but now it seems clear that something needs to be done about how the system works. Disagreements among level judges and level judges and general people who judge levels anyway (like me) are getting more and more common. Me and Jumpyhead have had a little dispute about it, ending in us both agreeing that something needs to be done with the system, and the example here is another clear case of somebody said something, another person said another thing and a mod ended up moving it somewhere else. It is clear that the system at the moment is people judge however they want to and the mods move to wherever they feel like moving it to. Doesn't work very well. One thing is certain. Everybody has their own opinion and no matter what that opinion is, as long as it is reasoned, should be fully respected. If someone explains as to why they dislike a certain level when everyone else likes it, this opinion should still be respected as it is reasoned. Opinions like "u is a fagit ur lvl sux dik lelz" should be ignored and are completely needless. So no matter how much the system changes, all rational opinions should be considered. If you don't like the negative yet reasoned opinions, tough. What I suggest is a more concrete system. Right now, every judge, every person and every mod has their own way of evaluating a level. Some give more importance to gameplay (me), others give more to design and decoration, and others just throw a number out there. It's one thing having an opinion as to how good the gameplay is and how good the design and decoration is, but everybody having their own opinion on how to allocate the points isn't going to allow for a solid reviewing/judging system. We need the judges and the mods to come to an agreement on how this should be done. I have set up a spreadsheet that I use for myself, and if anybody wants to see it just let me know. That's how I do things now (although I have only done two reviews with this new system). If you can come up with one solid system for all to follow, that's great. If you come up with another idea that could work just as effectively or even more effectively, that's really great. But remember, everybody's reasoned opinion is as valid and as worthy as your own reasoned opinion, so ALL opinions must always be taken into account. /rant EDIT - Tagging level judges and mods that seem involved more than others in this system: megaman9 Cinereal RyanAB @rikka @stardust1971 TygrysekAnd most importantly: R4Z0R (who was online 14 hours ago, so he can't be having too many issues accessing the forums anymore) |
|
|
|
Post by LuMaIchArgI on Oct 27, 2015 5:47:27 GMT -5
3 much ranting in the forums these days ;3;
|
|
|
|
Post by Mechanguis on Oct 27, 2015 6:52:33 GMT -5
I think that at least 3 people should be judging a level and then the average score of those 3 judges should be the final score! But then again it's difficult to find 3 judges that are active,so the levels in the Suggestions board will be stuck forever... R4Z0R should give judges like Jumpyhead or Cinereal the rights to move threads and recruit some more that are very active on this forum! So what I'm trying to say is that not only 1 person should review a level and it gets moved instantly... we should wait for more people to judge it! Cinereal isn't very active as a level judge though  |
|
WillFlame
Owner
Direful Reflection
An idea has taken root.
5,190 posts
Discord: WillFlame#5739
Favorite Level: Colourful Overnight
Mini-Profile Background: {"image":"https://i.imgur.com/4y98NDS.png","color":""}
Mini-Profile Name Color: 00a3ff
Mini-Profile Text Color: ffffff
|
Post by WillFlame on Oct 27, 2015 7:06:38 GMT -5
I want to see the spreadsheet.  As a side note, we also need to decide how many points deserves which board, because there has been a lot of level displacement, as mentioned in this thread by Lumpy. |
|
|
|
Post by megaman9 on Oct 27, 2015 13:14:37 GMT -5
I want to see the spreadsheet. As a side note, we also need to decide how many points deserves which board, because there has been a lot of level displacement, as mentioned in this thread by Lumpy. 0-2 Needs Improvement 3-5 Decent 6-8 Good 9-10 Feature Worthy IT'S NOT THAT HARD PEOPLE ;A; |
|
396 posts
Mini-Profile Background: {"image":"","color":"1d81bf"}
Mini-Profile Name Color: 1c2833
Mini-Profile Text Color: aed6f1
|
Post by Cheeseman on Oct 27, 2015 14:46:28 GMT -5
3 much ranting in the forums these days ;3; It's what I do best I want to see the spreadsheet. Will send you a PM with a link to the spreadsheet ASAP As a side note, we also need to decide how many points deserves which board, because there has been a lot of level displacement, as mentioned in this thread by Lumpy. 0-2 Needs Improvement 3-5 Decent 6-8 Good 9-10 Feature Worthy IT'S NOT THAT HARD PEOPLE ;A;This is also a problem. My system is wildly different to this. I find "Needs Improvement" to be anything below 5, because it is more bad than good. "Decent" is 5-7. "Good" is 7-8.5. "Feature Worthy" is 8.5-10. Completely different. This is probably another thing that needs to be agreed on. Everybody's values have completely different meanings and in order for things to be more objective some sort of agreement has to be made. We can't just go "this system, ignore the rest". I do make things simpler though as I always list next to my score where I think the level belongs, so maybe that is another solution. While everybody uses their own system, they need to make it clear as to where they believe it belongs. Damn, the harder I think about this, the more complicated the situation and even the solution seems to become. |
|
|
|
Post by Jumpyhead on Oct 27, 2015 17:11:49 GMT -5
We've had topics about needing more level judges, what the mods and admins should be doing in the level suggestions, what level judges should and shouldn't be doing etc. but now it seems clear that something needs to be done about how the system works. Disagreements among level judges and level judges and general people who judge levels anyway (like me) are getting more and more common. Me and Jumpyhead have had a little dispute about it, ending in us both agreeing that something needs to be done with the system, and the example here is another clear case of somebody said something, another person said another thing and a mod ended up moving it somewhere else. It is clear that the system at the moment is people judge however they want to and the mods move to wherever they feel like moving it to. Doesn't work very well. One thing is certain. Everybody has their own opinion and no matter what that opinion is, as long as it is reasoned, should be fully respected. If someone explains as to why they dislike a certain level when everyone else likes it, this opinion should still be respected as it is reasoned. Opinions like "u is a fagit ur lvl sux dik lelz" should be ignored and are completely needless. So no matter how much the system changes, all rational opinions should be considered. If you don't like the negative yet reasoned opinions, tough. What I suggest is a more concrete system. Right now, every judge, every person and every mod has their own way of evaluating a level. Some give more importance to gameplay (me), others give more to design and decoration, and others just throw a number out there. It's one thing having an opinion as to how good the gameplay is and how good the design and decoration is, but everybody having their own opinion on how to allocate the points isn't going to allow for a solid reviewing/judging system. We need the judges and the mods to come to an agreement on how this should be done. I have set up a spreadsheet that I use for myself, and if anybody wants to see it just let me know. That's how I do things now (although I have only done two reviews with this new system). If you can come up with one solid system for all to follow, that's great. If you come up with another idea that could work just as effectively or even more effectively, that's really great. But remember, everybody's reasoned opinion is as valid and as worthy as your own reasoned opinion, so ALL opinions must always be taken into account. /rant EDIT - Tagging level judges and mods that seem involved more than others in this system: megaman9 Cinereal RyanAB @rikka @stardust1971 TygrysekAnd most importantly: R4Z0R (who was online 14 hours ago, so he can't be having too many issues accessing the forums anymore) ^ Agreed We NEED a proper rubric that everyone will follow. Also, megaman9 I've seen you give levels an 8 and then move them to feature worthy, but now you say 8 is good? I'm most annoyed that without even offering their opinions, some mods just move the thread where they want without even READING what people wrote. So annoying. Cheeseman I've seen this happen to some of the levels you review, you rate it like decent, and it gets moved to good? Like what?! Razor certainly has been online and reading these issues, is he just being lazy? |
|
972 posts
Clans: AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
Hardest Demon: ffffff
Mini-Profile Background: ffffff
Mini-Profile Name Color: {"image":"http://3danimation.e-spaces.com/backgrounds/hd_background_american_flag.jpg","color":""}
|
Post by DyucFkemAdd on Oct 27, 2015 20:10:44 GMT -5
Level Judges never should of existed in the first place, it's useless and it takes up time
I don't want my level rated by some hippie
|
|
WillFlame
Owner
Direful Reflection
An idea has taken root.
5,190 posts
Discord: WillFlame#5739
Favorite Level: Colourful Overnight
Mini-Profile Background: {"image":"https://i.imgur.com/4y98NDS.png","color":""}
Mini-Profile Name Color: 00a3ff
Mini-Profile Text Color: ffffff
|
Post by WillFlame on Oct 27, 2015 20:21:27 GMT -5
I use the ratings sort of like grades, but with some little changes: 0-5.9 = Needs Improvement 6-7.5 = Decent (varies) 7-8.9 = Good (varies) 9-10 = Feature-worthy I rate gameplay, decoration and technical (colours, music sync) out of 10 each, then take the mean of the three numbers as the final score. Just offering an idea, but does that sound fair? I'm not sure what R4ZOR is doing, but he doesn't seem as active as he used to be... :/ EDIT: Level Judges never should of existed in the first place, it's useless and it takes up time I don't want my level rated by some hippie It's not useless; people want feedback. But yes, it does take up quite a bit of time with the amount of levels coming in. You don't have to have your level rated, but the point of this is to get some feedback from someone who has actually seen your level firsthand. Also, that grammar mistake in "should of" is bothering me so hard. It's "should have", people. |
|
|
|
Post by megaman9 on Oct 27, 2015 20:43:55 GMT -5
Also, megaman9 I've seen you give levels an 8 and then move them to feature worthy, but now you say 8 is good? When have I done that? 8 has always been good for me. |
|
|
|
Post by Jumpyhead on Oct 27, 2015 22:34:30 GMT -5
I use the ratings sort of like grades, but with some little changes: 0-5.9 = Needs Improvement 6-7.5 = Decent (varies) 7-8.9 = Good (varies) 9-10 = Feature-worthy I rate gameplay, decoration and technical (colours, music sync) out of 10 each, then take the mean of the three numbers as the final score. Just offering an idea, but does that sound fair? I'm not sure what R4ZOR is doing, but he doesn't seem as active as he used to be... :/ EDIT: Level Judges never should of existed in the first place, it's useless and it takes up time I don't want my level rated by some hippie It's not useless; people want feedback. But yes, it does take up quite a bit of time with the amount of levels coming in. You don't have to have your level rated, but the point of this is to get some feedback from someone who has actually seen your level firsthand. Also, that grammar mistake in "should of" is bothering me so hard. It's "should have", people. Lol willflame. You told kappa to fix his grammar, but you wrote "Bothering me so hard" XD |
|
|
|
Post by 6359_ on Oct 28, 2015 3:40:14 GMT -5
What we need is: 1. Restrucuturing the rating system so that it's centralized and everyone follows the same template. 2. Letting 3 reviewers rate the level and then move it; based on average score. 3. An actual "ingame mod" camping this section; either a new level mod or someone new. 1. As for the rating system:I had a thread somewhat like this entire section once. The rating system was based upon ((Gameplay 0-10)+(Decorations 0-5)+(timing 0-5)/2)+(eventual bonus points)Where gameplay measures how fun it is to play, Decorations measures how nice it looks, Timing measures how timed the level is to the music, and Bonus points are if the level impress me in a way. The bonus points can also be negative; if the level does something that makes me cringe etc. (for example spanish dialogue in bossfights) If we put in an example level in here: Generic sunsetlevelGameplay: 2 Decor: 5 Timing: 1 One bonus point for some pretty nice decorations. Then we get ((2+5+1)/2)+1=5.5Total score: 5.5/10 (the highest level I've rated with this system got 11.5/10  ) 2. Three judges per level.This is to stop eventual bias. All scores are added together and divided by 3 to get an average. This score decides the level's placement. This is how I view the scores:0-4 Needs improvement 4.5-6 Decent 6.5-8 Good 8.5-10 Feature worthy (should be renamed to very good; as good and even decent levels often is feature worthy. 3. An ingame mod here to actually speed up the processEither someone here should apply for moderator status (how do you even do that) or we should get a moderator over here to check through the lists almost daily. As of now, they've sorta stacked up in a big mess; so even if RubRub checked it right now; he'd miss a lot. |
|
|
|
Post by Mechanguis on Oct 28, 2015 4:57:23 GMT -5
What we need is: 1. Restrucuturing the rating system so that it's centralized and everyone follows the same template. 2. Letting 3 reviewers rate the level and then move it; based on average score. 3. An actual "ingame mod" camping this section; either a new level mod or someone new. 1. As for the rating system:I had a thread somewhat like this entire section once. The rating system was based upon ((Gameplay 0-10)+(Decorations 0-5)+(timing 0-5)/2)+(eventual bonus points)Where gameplay measures how fun it is to play, Decorations measures how nice it looks, Timing measures how timed the level is to the music, and Bonus points are if the level impress me in a way. The bonus points can also be negative; if the level does something that makes me cringe etc. (for example spanish dialogue in bossfights) If we put in an example level in here: Generic sunsetlevelGameplay: 2 Decor: 5 Timing: 1 One bonus point for some pretty nice decorations. Then we get ((2+5+1)/2)+1=5.5Total score: 5.5/10 (the highest level I've rated with this system got 11.5/10 ) 2. Three judges per level.This is to stop eventual bias. All scores are added together and divided by 3 to get an average. This score decides the level's placement. This is how I view the scores:0-4 Needs improvement 4.5-6 Decent 6.5-8 Good 8.5-10 Feature worthy (should be renamed to very good; as good and even decent levels often is feature worthy. 3. An ingame mod here to actually speed up the processEither someone here should apply for moderator status (how do you even do that) or we should get a moderator over here to check through the lists almost daily. As of now, they've sorta stacked up in a big mess; so even if RubRub checked it right now; he'd miss a lot. I'm pretty sure this was already mentioned, but the 3 judges idea won't work because too few people review levels. It would take ages to move a level. |
|
|
|
Post by 6359_ on Oct 28, 2015 6:58:01 GMT -5
What we need is: 1. Restrucuturing the rating system so that it's centralized and everyone follows the same template. 2. Letting 3 reviewers rate the level and then move it; based on average score. 3. An actual "ingame mod" camping this section; either a new level mod or someone new. 1. As for the rating system:I had a thread somewhat like this entire section once. The rating system was based upon ((Gameplay 0-10)+(Decorations 0-5)+(timing 0-5)/2)+(eventual bonus points)Where gameplay measures how fun it is to play, Decorations measures how nice it looks, Timing measures how timed the level is to the music, and Bonus points are if the level impress me in a way. The bonus points can also be negative; if the level does something that makes me cringe etc. (for example spanish dialogue in bossfights) If we put in an example level in here: Generic sunsetlevelGameplay: 2 Decor: 5 Timing: 1 One bonus point for some pretty nice decorations. Then we get ((2+5+1)/2)+1=5.5Total score: 5.5/10 (the highest level I've rated with this system got 11.5/10 ) 2. Three judges per level.This is to stop eventual bias. All scores are added together and divided by 3 to get an average. This score decides the level's placement. This is how I view the scores:0-4 Needs improvement 4.5-6 Decent 6.5-8 Good 8.5-10 Feature worthy (should be renamed to very good; as good and even decent levels often is feature worthy. 3. An ingame mod here to actually speed up the processEither someone here should apply for moderator status (how do you even do that) or we should get a moderator over here to check through the lists almost daily. As of now, they've sorta stacked up in a big mess; so even if RubRub checked it right now; he'd miss a lot. I'm pretty sure this was already mentioned, but the 3 judges idea won't work because too few people review levels. It would take ages to move a level. Add more level judges then! |
|
396 posts
Mini-Profile Background: {"image":"","color":"1d81bf"}
Mini-Profile Name Color: 1c2833
Mini-Profile Text Color: aed6f1
|
Post by Cheeseman on Oct 28, 2015 10:07:53 GMT -5
The likelihood of more level judges being added becomes slimmer everyday thanks to R4ZOR's seemingly ignorant attitude to what is going on (I am only saying this because he does come on the forum every now and again... and says nothing), and the chances before already seemed quite slim.
I think at this rate we'll just have to have "unofficial" level judges.
|
|
)