Hello everyone!
Today I logged in to see that my quality score fell from 8.1 to 6.6 (the lowest I ever encountered)
Now I came to understand that, as of today, all goChecks we went through will be taken into account, not just the last 10, I just checked and it represents 4 years of work and 70 GoChecks for me. I know that the consitency is taken into account, but bear with me: I went through my archives and compiled every single score earned these last 4 years.
My average is 8.74. That's two points higher than my current score. However, as gengo explains it, the consistency is also taken into account. Fair enough:
47 of the total scores are situated between 9 and 10 (that's 67% of them), 26 of them being straight up 10s.
On 70 GoCheck scores, 59 are rated above 7, that's 84.2%.
Now, I do realise that there is room for improvement, I certainly do.
My question is: if all of that sends me down at 6.6/10, how can I hope to pass above 8 again? I mean in a foreseable future? My digging for these scores showed me that, other than several exceptions, I was evaluated roughly about once a month. Even if I pulled 10s upon 10s from now on (which I fully intend to do), I feel like it would be years before getting a decent score at this rate.
Even if I look at my past 10 scores, 2 of them are below 9, one of them being a 2,94 which, for the record, I disagree with, I could fill up a whole post about that but that's not the point here. The other being a 7,43.
I genuinely feel trapped here.
Am I the only one in this kind of situation I wonder?
41件のコメント
Unfortunately, no.
When I started working for Gengo, it was after I lost my dear mother on June 2, 2014, after a long and debilitating illness: she had a stroke. It was a terrible time both for her and me, especially as her condition progressively got worse. Those who have gone through it will know what it means, those who haven't will, sooner or later, and I hope it is much, much later.
I don't think one can ever get over it—one just learns to live with it.
So, my first job happened absolutely unexpectedly. At the time, I was not ready to do a translation, and just looked at the available jobs. Imagine my surprise when I came back to the computer after having a meal and saw that I had started translating a job. Frankly, I hadn't pushed the button. How it happened I have no idea. I panicked: the time left was less than an hour! It was a disaster, and the score was accordingly very, very low.
I couldn't work well at that time: my heart beat like crazy, my hands trembled, and it was terrible.
But a couple of years passed, and I suddenly felt better translating. I think my scores represent two different persons and, in a sense, they do.
The big question is: how can I live down my early unfortunate experiences with the new scoring system?
Here's how to understand the new scores:
So with 6.6, "you're doing well". I think all of us got a much lower score with this consistency calculation. I had 10 in periods previously, and close to that most of the time, but I've had some bad scores too. However, it doesn't seem like you need a score above 8 to keep your pro qualification.
I guess it's a bit more reassuring when seen with this perspective, thank you for that.
Still frustrating, but a bit reassuring.
I think a "visual feedback" a tad less anxiety inducing than your score appearing in bright red might have been a nice touch then, however.
P.S: By the way, while digging on my previous scores, I stumbled upon a gocheck that was utterly dismal but with only "minor errors" outlined by the reviewer. The thing I noticed though, is that when hovering on said mistakes, the "correction" or "suggestion" from the reviewer was sometimes displayed 2 or 3 times for each mistake. Is it possible that the scoring algorithm thus counted 2 or 3 distincts corrections for each mistake and thus made me end up with a score reflecting the double or triple amount of mistakes I actually was flagged with? Thank you if anyone as any input on this.
Hello,
Yes that has happened to me a number of times. The errors are counted twice or three times if they are all highlighted and by requesting a new review on the last one, I had that mistake corrected.
Previously, I hadn't bothered but things are getting a bit tight, so now I'll be extra careful with mistakes made by the reviewer themselves!
My score also went down a point from 9.9 to 8.8.
Alex
Wow, thanks, that also explains some stuff...
Said review is 1 month old, you think I can still ask for a "recount"?
The same thing happened to me, and it was even more dramatic in my case: I went from 9.5 to 6.7 just like that.
I understand the importance of consistency but this, sincerely, is brutal. We have all had bad days and bad reviews and we've all learnt from the experience. I think this change should have come with a warning before being implemented, giving us translators time to adjust and prepare, and perhaps calculating the consistency score from now and forwards, not for the *entire* work history.
The problem is the qualifications are taken into account forever. That makes that people who have been translating for years in Gengo now see their average score decreased because their scores were much lower in 2014 or2015. This is unfair, because it is obvious that they have now better skills and only consistency in their last months should be taken into account. Is not that your case, Alicia L. alonso?
Yes, that would be my case. That and the fact that I have had some high scores and then a few low ones like I assume is the case with some. We all make slips and mistakes and learn from it.
I guess the best way to take this now is to try and be as consistent as possible moving forward.
As I wrote in a parallel discussion on the forum:
Taking into account all our scores (especially those when we were taking our first steps in translation for some of us) is certainly a big disadvantage for those who have actually made a lot of progress in their skills.
Suppose Person A is somebody having worked for Gengo for 3 years or so and who has seriously refined their skills to deliver consistently top quality translations today but who was struggling in 2015 or 2016.
Then consider Person B: Person B is a recently qualified translator on Gengo and she/he delivers the same quality translations as Person A.
Yet, in the new system, Person A's Gengo score is below Person B's.
So what exactly does the new scoring system reflect?
To all,
You may would like to read the following topic named "Consistency Score".
https://support.gengo.com/hc/ja/community/posts/360017912794--Consistency-Score-
This topic is fairly long. So I would lke to summarize it for you.
-Accoridng to Gengo's member, this "consistency score" is not a new thing, but addition. They showed this to translators in order to make it easier for translators to where we stand in regard to consistency of our quality. (However, I doubt whether this actually apply to us in current situation. Change in scoring system cannot be taken as "addition" in my point of view.)
-At least, several translators are either received of warning, or revoked of their qualification, due to consistency score, in these several months. Gengo has been using consistency score for evaluating translators for quite a while.
As I wrote, that topic is fairly long, and some of information contained in the topic might be unnecessary, but hopefully it can be some help to you.
I wonder why the comment I made yesterday at 18:19 is still pending.
@Alicia L. Alonso
My experience seems to echo yours. My 9.2 average plummetted to a 6.6 when I checked my score this morning. I hadn't visited Gengo for about a week up until that point, so it really came out of left field...
I was so shaken, I ended up averaging my past 35 scores (everything for 2018), and whaddya know--a 9.2/10. Three of those 34 were marked below a seven: one at 5.2, another at 2.9, and another at 6.7.
I've made my feelings on the matter known to Support and they have echoed every other Support member who has chimed in here and in other threads on the topic. I understand the need for a consistency score in theory, but I feel the implementation has been handled with a distinct lack of finesse. Anecdotally, if I am being brought in for a year-end review and I have NAILED my projects out of the park with glowing praise from my clients, three less-than-remarkable results should not drag my overall score down to a 6/10 from what should be a 9/10. A reasonable supervisor will say, "Yeah, you've had a few bad days, but 9 times out of 10, you're doing great." That 6 on my dashboard? Not helping me feel valued, Gengo.
I've been with Gengo since 2015, and I'll be the first to admit that I wasn't bringing my A game back then, but anyone with access to my scores can see that I knock it out of the park on a regular, and most importantly consistent, basis. With over 800,000 characters under my belt and way more happy clients than Gengo to worry about, I'd like to think I'm at a point in my career where I know what I'm doing.
Why punish us for mistakes we didn't even know we were making? Once again, I see the value of a consistency score and support its implementation, provided that it is done correctly. Personally, I would suggest that the new scores only account for reviews made after the date the news post was made. That would at least level the playing field and give people an idea of the standards expected. As it stands, I see absolutely no way to bring my score above a 6 based on almost 4 years of backdated reviews, meaning a new recruit with fewer reviews will assuredly be placed in a higher bracket by default, where I'm left walking the tight-rope being weighted down by obsolete results.
I spent a lot of my afternoon in tears because, for a dark moment, Gengo had me convinced I was less than what I thought I was worth when I know damn well that I'm a fine translator.
I can see a lot of people walking if Gengo doesn't get their act together over this, myself included. For now, though, I'll just look at the "Wordsmith" and "Top Scorer" headings on my profile and the Gengo T-shirt hanging up in my closet and just shake my head.
I agree with most everything said here. I also think that, while support's comments that translators will have to readjust to scores' new meanings (7 in many ways being the new 8) are reasonable, there is something to be said about the new score calculations not being very intuitive. For example, my score just got brought down from a 9.3 to a 7.6. Okay, I've had some low scores before, I get how that happened. But then I get a job reviewed just today, with a score of 8.45. Now my overall score is 7.5. I can only imagine that anyone new coming to Gengo who sees their score drop after getting a better review than their current score would be very put off, and might hesitate to continue doing work here. Also, the example calculations on the support article are confusing, since even in the example of the translator who IS supposed to be consistent, their total score is lower than any one of their individual scores. I think scaling the scores to be more understandable, since I'm sure the actual calculations are a bit of a trade secret, would go a long way to keeping the relationship between support and translators positive.
I've sent this same question to the support team, Antoine (still waiting for an answer).
"How could we, translators that have been with Gengo for a long time, 3+ years, achieve a 8+ (my QS decreased to 7.3) score? In a matter of fact, what 10 jobs reviewed with 10s would impact in my QS (notice that I should have 100+ jobs reviewed, I don't really now)?
For me, this new system seems highly disadvantageous to older translators (that have waaaay more jobs reviewed). For us, I don't really see how we can achieve a 8+ score. From what I see, after reading about the new system, that it is impossible. There's simply no way. I could get those 10 10s and get like a 0.1~0.2 upgrade in my quality score, while a, I don't know, 2.4 rating, would get it way down (since lower scores apparently are given way more importance).
I've read many posts on the subject in this and another forum.
Everyone seems to find the new scoring system unfair, especially to older translators. All replies by Gengo's staff are only efforts to explain the "unexplainable" and they don't convince anyone. There are many threads opened, but see if you can find Nelson's posts who makes a very detailed and accurate analysis of the situation.
I personally find the new scoring system totally unfair and frustrating. The figures below indicate why I feel betrayed to see my score drop from 10 to 9.4 after more than 6 years as a Gengo translator:
Number of GoChecks 119
Number of jobs translated 26,035
Number of words translated 557,618 (plus another 50-60 thousand in special projects)
Customers served 380
I understand that 9.4 is still a high consistency score but it's based on a very small number of low ratings dated months or years ago. As others have said, even one low rating will never allow me to regain a score of 10.
Well done Gengo, someone(s) in your company will be very satisfied for adopting his/her/their idea of changing the scoring system.
You will certainly find that not even one translator is satisfied.
I don't expect to get a reasonable reply, just the usual canned responses.
Thanks for listening (?)
Hi fellow translators,
This post is so interesting and I think we are making a great effort to let Gengo know what we think about their sooooo weird and unfair new quality system. However, there is a VERY interesting post started by an user called Antoine which deals with it and also the MACHINE TRANSLATION NEW SYSTEM, giving a very interesting standpoint and showing why this MT system is unfair too. Could you please check it out and leave a comment?
The post is A bit of feedback on translation scores and MT
I am in the same boat as many of you. My score dropped from 8.8 to 6.5. In the 33 reviews I've accumulated in my time with Gengo, I've had three scores that were below 6.5. Strange.
My two bad scores were translations with only 21 units. It seems to me the new system doesn't really factor in the amount of units a job has. So making one mistake in every 10th short translation seems to be worse than one mistake in one long translation. That just doesn't seem to make sense to me as human error happens at random and doesn't care if there is only one sentence in a translation or many. We all make mistakes at some point.
So to me, the new system would be highly discouraging to accept low volume jobs since the risk of having my score put at risk is simply too high. Unfortunately, most jobs at non-pro level are very short jobs.
This consistency score thing definitely has to be rethought. Is it now better to be a consistently poor translator than to be a good translator and have occasionally made a mistake on a short translation that resulted in a couple of unrepresentatively low scores? That seems to be what Gengo is saying with this new system. It's as though we're being discouraged from improving. If we get 7.2 in one review are we supposed to try to make 2.8 points worth of errors in every translation after that in order to be consistent? Because if we get a 10.0 next time it seems as though they'll be docking us points for the 2.8 difference rather than rewarding us for doing better.
Yes, I feel extremely discouraged and demotivated. Japanese-speaking and Chinese-speaking translators are quite upset too. I just wanted to leave a note here so that I get a notice each time someone posts on this thread.
I too have been with Gengo since over 6 years, and my score dropped from 9.4 to 8.3 after the implementation of this very questionable new scoring system. But what is really strange here: I had just one review (a 10/10) after that implementation, and my score dropped further to 8.2. Now please, what is the logic here? I can certainly see the need for "consistency", but this new system seems very arbitrary and frustrating, for all of the reasons mentioned on this thread.
@Yasemin Erdem
@Lara
This is what Lara wrote in the discussion thread "Consistency Score"
"The previous version of the scorecard was based on a weighted average of the last 10 GoCheck scores. This gives a general idea of overall quality but fails to take into account fluctuations or inconsistencies. The new version adds a buffer around this average of 10 GoChecks based on the standard deviation of past Gocheck scores. This new score (average of the 10 last GoChecks minus standard deviation) represents the lower threshold of quality that a translator delivers consistently. Your new score represents that most of the jobs delivered will be above this score."
With this formula, an exceptionally good score will increase your standard deviation and therefore decrease your consistency score !
Wonderful, isn't it? I guess Gengo needs to go back to the drawing board on this one.
Alex
Thanks Alex, I agree. Although I had quite a few jobs with a 10-rating (and not just recently, but consistently over the past years), so I'm not even sure about what already seems like a warped reasoning here. What is the logic? Shall we try to include minor errors, so that the standard deviation is lower and thus the score remains the same? In other words, aim for 9 rather than 10, so that you keep your score? That is just ridiculous.
I think that no matter what, a HIGHER SCORE should never reduce your existing score. So yes, back to the drawing board.
"With this formula, an exceptionally good score will increase your standard deviation and therefore decrease your consistency score !"
This is not entirely true.
Because the score takes into account the average of the most recent 10 GoChecks, that "exceptionally good score" will likely raise your average, from which the standard deviation is subsequently substracted. How much your standard deviation will increase, or not, will depend on more factors than simply "getting an exceptionally high score", including previous fluctuations and how "exceptionally high" this score is and in relation to what. Any fluctuations between 7 and 10 shouldn't result in a low consistency score, as the formula will start with a pretty solid average score to begin with.
Yes I agree with you Lara,
But whatever the usual scenario, in the case of Yasmine and unless she is mistaken or missed another review, that is exactly what happened... and you can really understand that she is frustrated to see her score go down after a perfect 10.
@AlexF - I've taken a look at Yasemin's history and I have now a better picture of her specific situation. (Comment edited so as to not disclose any translator specific information - Lara)
@Yasemin - Please read below to understand your new score:
The translator scorecard has previously displayed that a 7 was a good score. In reality, this was the bare minimum expectation for all translators and an overall score of below 7 could cause qualifications to be revoked. With an average calculation, a score of 7 means that half of the scores could be below 7 and a translator would still retain their qualifications. What is considered a good score is redefined with the new calculation, with a 7 being a great score: it means that most delivered jobs will be consistently above this score.
Because the score consists of two parts, the average of your most recent 10 GoChecks and a fluctuation calculation, the timing of when you receive a low score will impact your overall score differently. For a translator who has fluctuated in the past but has improved, their low scores will only slightly affect their fluctuation calculation, whereas their recent jobs will equal a strong average. For a translator who is recently fluctuating, their score will be doubly affected, both in the fluctuation component and in the recent average. The score still leans towards emphasizing recent performance.
Hi Lara,
In general, I would prefer you write to me directly (PM) about this, as I am really curious what you are referring to. And yes, it still is frustrating to receive a 10 (which I received very often, and not just in the recent past) and go from 8.3 to 8.2.
@Yasemin - Of course, let me drop you an email :)
Thanks, I have written back. There is an obvious mistake here (involving a different language pair I used to work in but have stopped a long while ago), please check my email. Thanks.
In my opinion, the consistency score is absolutely discouraging people to take on short jobs and collections. Even the best translator will make a mistake at some point every 100 or so sentences. Sometimes it is even due to an incorrect source text etc.
If every sentence is split up as a single job in a collection and that one sentence gets picked for review, he/she will get one very very low score. In the past, these low scores were weighted with the unit amount of the job and not that big of a deal - I am not saying that I didn't care about getting bad reviews or that I was careless but it didn't have such a massive impact on the score and it went away with a couple of good new reviews. Overall, a bad score was all the more motivation to keep on working with Gengo to replace that bad score with ten new great ones.
Now, one such bad score has a massive impact on the consistency score. And it will stay there forever. How should that be motivating to continue working more and harder?
I am confident to say that I can score very consistently over 8/10 for all translations with several sentences. But I cannot guarantee that there will be not one single mistake in any of the translations that I submit. Nobody can. And if that mistake happens in a short translation, Gengo assumes that I am extremely inconsistent and will never forgive or forget. That just doesn't make any sense.
You at Gengo seem to be a bit distracted with the new system as well. Within a couple of days in a row, I got three e-mails. The first one stated that due to my excellent score, I now get access to more jobs due to the new distribution system. A day or so later, I get an e-mail stating that due to my low scores, my qualifications are at risk. That was confusing to me and the time I learnt about the new glorious system. The next days, I was basically just waiting to have my qualifcations revoked thanks to my new 6.5 score. But then I got another email, inviting me to take the pro test in my language pair because I continuously deliver such high quality in my work. So what is it now?