6

Several hours ago, I received e-mail from Gengo's team, which states that from now on, translators will be evaluated with not only scorecard, but also, with "consistency score".

They explain that this "consistency score" is set up for evaluating consistency of translation quality. For instance, if your last 3 GoCheck scores are "7.0", "7.0", "7.0", your average score is "7.0". Meanwhile, if your last 3 GoCheck scores are "10.0", "10.0", "1.0", your average score is sitll "7.0", but your "consistency score" will be signicantly lower than the previous case (at least, I understand Gengo's explanation as so).

Well, the concept is understandable. Any customer should be able to expect consistent quality of translation, of course.

But seriously, why does Gengo team make such important change without making any prior notice? They say that this "consistency score" will be used for evaluation of translators.
I have searched across twitter, blog post, support page etc., but no information whatsoever was found about this "consistency score".

Besides, it makes situation very hard for translators to keep his/her status as translator.

Along with "consistency score", new "performance dashboard" was revealed. Surprisingly, scores of (probably) every translators of Gengo can be checked from it from now on. I have checked scores of several translators.

To my surprise, many translators with above 8.0 (or sometime 9.0) on his/her scorecard, receive below 7.0 for "consistency score". Such persons received some low score (below 6.0) few times, which makes his/her consistency score significantly lower than his/her "scorecard".

As far as I heard, if your "consistency score" becomes below 7.0, your status as Gengo translator is at risk. In short, even when your scorecard shows above 8.0 or 9.0, when you make even one bad mistake in any of your job, you are in trouble from now on.

Anyway, I wish to ask Gengo's team about specific explanation about this "consistency score". How it is calculated, how it will exactly affect my translator status, etc.
I myself am currently contacting with Gengo's support team about it, but thought that it would be good idea to share this information with you all.

70 comments

  • 7
    Avatar
    Nelson Bras

    Dear Lara,

     

    That's certainly a nice effort to get closer to a better solution. Thank you.

     

    Nevertheless, the main problem is yet to be solved.

     

    Being a 45-year old seasoned translator/proof-reader/reviewer/editor for some of the major translation companies worldwide, with more than 1 million translated words and 665 happy customers served just in Gengo (top scorer, pro translator, Gengo wordsmith, millionaire's badge and numerous "preferred translator" awards), having an extensive portfolio (including multinational companies, governments and international organisations) and some of the best quality ratings in the industry, and providing for a family of three through my translation work, exclusively, there must be a good reason why, for the first time since 2014, I keep declining more and more tasks here….

    As a matter of fact, there is!

    It's not because I don't know how to translate these jobs, but rather because I fear to be trapped in the unlucky situation where my target language allows for multiple choices with the same meaning, and my skills don't include guessing the senior translator's "mood" regarding my choice of words!

    Being a proof-reader/reviewer myself, the time has taught me that I will always be learning till the end of my days - Modesty is a great thing, since it allows you to improve with every experience, even by learning from the very individual you are tasked to review!  - No one knows everything about every subject. And this is why, before rating something as a mistake, I always make sure to be properly backed up by the opinion of one or several experts. My reviews are never based on "because I say so and I am on a higher level", but rather substantiated through links to expert sites, real grammar books and practical examples. Most of the times, I know the reasoning by heart… but I always try to pass that knowledge to the translator, by providing him/her the opportunity to check the official resources and improve his/her fluency. There are certainly cases where I don't know/use a certain term… however, this is not reason enough to immediately consider it a mistake - instead, I always try to do some research on the term - even though this term is not used by my circle of friends or in my region, even though I don't remember seeing it in any of my usual reading material… perhaps this is just a term which is commonly used elsewhere, a specific industry jargon term, a more literary term… So, I research!

    As far as I can see, none of the above applies to Gengo's senior translators. Their opinion, right or wrong, is law. No need to explain, no need to teach, no need to justify, no matter if they can't find any official rule to support their choice (when this is the case, they just make new rules as they go and invent a whole new grammar, according to their personal preferences and to what they think is the "prettiest" word!) The translator has no chance to win this game, because the game is rigged from the beginning!

    Remember this case referred to in my previous posts? - " The senior translator was convinced that every grammarian in the world is wrong and they should learn with Gengo's senior translators (e.g. https://ciberduvidas.iscte-iul.pt/consultorio/perguntas/ligacao-a-vs-ligacao-com/34828 - by the way, this was never corrected);"

     

    I have just received another review showing that this is not a "one-time situation".

    Now, let me start by saying that this is a 3584-units job (standard!).

    Three mistakes have been marked by the ST:

    1 - There was an "s" missing at the end of a preposition+article contraction ("à" should be "às", in order to agree with the plural word right next to it). Even using a second spell checker (MS WORD), this little guy managed to escape my attention! Well… fair enough. This is definitely a mistake. What I don't understand is the decision from the ST to select/affect two words just to rate a one-letter typo, thus aggravating the weight of a micro-mistake… mmm… is there any limit for the number of words they are allowed to select?

    Now comes the most interesting part.

    Perhaps the text was too long, and the reviewer got bored of not finding any real errors, because this is what happened next…

    2 - "The duration of your stay also plays a role." - translated as "A duração da sua estadia também desempenha um papel importante".

    According to the ST, by using "desempenha um papel" (literally, "plays a role"), I am committing a serious offense against grammar rules, since "the duration" is not a person, ergo, it cannot play any role! Just like that, as in "the Sun revolves around the Earth, because this is what I see", with no further substantiation added.

    Upon requesting a new revision and explaining the meaning/importance of using figures of speech/rhetorical devices, such as the prosopopoeia and the hypallage, in both English and Portuguese literature (as one can easily determine by reading Eça de Queiroz, Fernando Pessoa, Saramago, etc.), I was finally successful in preserving the good name of several generations of Portuguese literary masters! The "mistake" has been deleted by the ST.

    My question is: if a Senior Translator blatantly fails to recognize this kind of material, is he/she fluent/knowledgeable enough to perform the task? Am I supposed, as a translator, to invest my time teaching basic grammar to an ST just to preserve my scoring each time I deliver a translation? Am I supposed to always be afraid to accept a task just because I don't know if the ST is going to apply official or personal criteria?

    OK, STs are also human beings, they do have "bad days", and they are not all alike. Nevertheless, what really bugs me is the frequency of the recidivism and the lack of modesty preventing some of them to do proper research before chopping someone's head off! If I do that research, why don't they?

    On a side note: I remember a time when the STs and I always had good "win-win" conversations when an error was found. Back then, the STs used to identify themselves, we all learned from each other and the use of "hello"/"goodbye"/"thank you", etc. was always present. I really miss that kind of human touch. Is Gengo hiring almighty unreachable gods nowadays? Are they too perfect to say "Hi" to simple human beings? Or do they fear to be immediately linked to their bad decisions?

    3 - Speaking of "fluency", the third "mistake" was all about "lack of fluency" (which I was accused of by the same ST who never heard of figures of speech in his/her life!)

    Let me remind you that we are still talking about the same "standard" job as before, not a "pro" job (and even if it was…) and tell you that this "mistake" was not cleared after the second review.

    Since we only have one chance to get a re-review and the "comments" field is often not enough to let me copy/paste the "homework" the ST should have done before using his/her "sword", this is now irrevocable, thus affecting me for the rest of my life at this translation platform.

    Here's the "mistake":

    "Many people also speak excellent English and Spanish" - translated as - "Muitas pessoas também falam inglês e espanhol de forma excelente".

    According to the ST, "de forma excelente" is not fluent enough, and I should have used the word "fluentemente" (fluently) instead.

    Well… since the ST did not bother to support his/her opinion on anything else but his/her personal belief, the burden of proof rests, once again, on the translator.

    There are too many ways to prove the ST's opinion not only is absolutely wrong but also denotes severe lack of fluency. Yes, I mean it, because fluency includes mastering a language in a way that allows you to use different constructions to express the same meaning. If one only knows a couple of sentences/words to express a certain meaning in a language as rich as Portuguese… it can only mean that his/her vocabulary is too limited to be considered an "expert".

    Regarding this particular "mistake":

    1. The original wording is "Many people also speak excellent English and Spanish". If the author intended his/her words to be translated as "fluentemente", I am pretty sure that he/she would have said "Many people also speak English and Spanish fluently". Who am I to say the author is wrong? Especially when there is no grammar rule preventing him/her to use the construction above in English?
    2. Likewise, there is no rule preventing me to use that same expression in Portuguese… so, why not respect the author's choice and be a translator instead of an overzealous editor?
    3. If there was still any doubt, "de forma excelente" can be found in thousands of official Portuguese documents - https://context.reverso.net/translation/portuguese-english/de+forma+excelente /// https://www.linguee.pt/portugues-ingles/search?source=auto&query=de+forma+excelente /// https://tvi24.iol.pt/videos/merkel/portugal-cumpre-programa-de-forma-excelente/53f517153004540d1c50d86a /// https://tvi24.iol.pt/videos/merkel/portugal-cumpre-programa-de-forma-excelente/53f517153004540d1c50d86a /// https://tvi24.iol.pt/sociedade/incendio/dispositivo-de-combate-a-fogos-esta-a-responder-de-forma-excelente, etc.
    4. Since the ST states that "de forma excelente" cannot be used with the verb "falar" (perhaps this is the only verb "allergic" to excellency???), he/she should have shown at least some evidence to support his/her belief!
    5. Even if we consider the need to "polish" the text due to the fact of being a "pro" job… WAIT A MINUTE, this is a STANDARD job, remember??? Is there any difference between "pro" and "standard" jobs besides the fact that a "standard" job pays you 0.03 and a "pro" job pays you 0.08??? So… less pay, same level of demand/risk???

     

    Now… add this to the fact that a bad rating will haunt you forever under the new system… and you have the perfect recipe for disaster, as well as the answer to why so many great experienced translators are declining more and more jobs nowadays.

     

    Are you looking for a cheap solution to improve quality? Why don't you start by establishing clear rules for STs regarding the situations above? Rule number 1: Every decision must be duly substantiated. If no rule can be found to substantiate the decision, the ST should refrain from pointing errors based on personal convictions; etc.

    Remember: in contrast to what happens when a translator makes a mistake, a bad decision from a single ST may easily turn into an outbreak of bad translations by the thousands!

    The way I see it, it makes no sense to keep spending more and more time/money pruning branches when the real problem lies in the root of the tree.

    Please accept my apologies for such a long text and thank you for your cooperation.

     

    UPDATE: I have just received another review - again, 3  unreal mistakes (seems like this is the ST's magic number)… and here I go again… losing precious time, asking for a re-review and teaching grammar to someone who is paid to be an "expert". Mmmm… Don't get me wrong, but the corrections are so unreal that it almost feels like someone is trying to trim down the competition! I really hope this is not the case.

     

    Best regards,

     

    Nelson Brás

    Edited by Nelson Bras
  • 8
    Avatar
    Nelson Bras

    Dear Lara, fellow translators and Gengo team.

     

    Thank you all for these joint efforts to find the best solution.

    While I am waiting for my last post to be approved,  please allow me to cover a different subject: math.

     

    Gengo's math is also very difficult to understand. For the sake of transparency, would you be so kind as to answer the following questions?

     

    1 - How is it possible that 2 minor mistakes covering 4 words (word structure - low / syntactic error - low) in a 3584-words job can produce the same effect (3%) in your score as if 108 words had been affected by a severe mistake?

     

    2 - Is the scoring affected by the number of words selected by the reviewers? If this is the case, why do they select additional words, punctuation and even unrelated tags?

     

    3 - Is there any difference - weight, type, criteria, etc. - between errors that can be found in "pro" and "standard" jobs?

     

    4 - Do the ST's have an option to include a comment/suggestion without pointing an error? If so, why do they always point an error when they make an improvement suggestion for the future?

     

    5 - Does a re-review count as an additional review, instead of substituting the "wrong" review, in terms of standard deviation? Because this is what it seems to be happening.

     

    6 - Do you really think it is fair to apply the same standard deviation and "worst rating deletion" formula to someone who has already translated 1 million words (thus having much more opportunities to commit mistakes or to be subject to unfair reviews, etc.) and a recent translator who only has 500-2000 words in his/her portfolio to prove his/her value? I am pretty sure that this is unfair for both! Those who think this change is enough will certainly change their minds after translating 1 million words! The only way to create a levelled playing field would be to limit the standard deviation to a pre-established period - 30, 60, 90 days…

     

    7 - Since the result of a re-review is now a ghost that will haunt you for the rest of your days, don't you think it would be fair to allow the right to appeal to higher instances in case of an absolute well-founded disagreement?

     

    As I mentioned before, the way I see it, it makes no sense to keep spending more and more time/money pruning branches when the real problem lies in the root of the tree.

     

    Note:

    My score is 9.1 nowadays. Should I feel happy and keep quiet? No way! - I can see clearly where this path is going in the near future for every translator, and even for Gengo as a whole. I love this platform. It is still one of the best. That's why I keep working here despite the low rates. Please, don't let it go down as a result of a couple of bad but crucial decisions!

    Thank you for considering my thoughts.

     

    Best regards,

     

    Nelson Brás

    Edited by Nelson Bras
  • 2
    Avatar
    Xavier

    Sorry to hear about your frustration Nelson!

    I haven't had many issues with the LS corrections of my language pair, but there is sometimes room for a little subjectivity that should not be counted as an error. From what I understand, there are now 3 levels of "error", so hopefully the lowest one doesn't affect the score much.

     

    Regarding Lara's latest statement saying that Gengo "remove the lowest GoCheck score in [our] history when calculating the overall translator score. This change will ensure that a single low score does not outweigh an otherwise consistently good performance, while also giving newer translators a fair chance to prove their skills when they are just starting out with Gengo." I am mildly satisfied to be honest. It is nice to see that our complaints were heard and that some change was made, but it is far from what we were asking. Most translators who participated said that they wanted the consistency to take into account only recent jobs (a few months, up to 6 months maybe?) or to take into account only the scores obtained since the consistency score was put into place. Removing only one low score seems to me like throwing us a bone, it is nice but it is not enough.

    Like Nelson said it again, for those of us with hundred thousands of words translated and years working for Gengo, there is a chance that we got more than one low score. Compared to new translators, that system puts us at a disadvantage.

  • 1
    Avatar
    Gyuri

    I am not a  big mathematician. As an economist studied some mathematical statistics. My gut feeling is, that the approach itself: deducting the standard deviation (i.e. the average absolute deviation, that is calculated from the average of the given figures and the differences of the figures from the average) from the average is a constructed figure that is not showing anything. Not to mention the problem of calculating from ten figures for those, having been working on many jobs - as it was mentioned by others. All the noises around it sound as if it would be the case.
    I had an average before the calculation 9.4-9.7. And now it is 7.7. My ego is OK, as long as I am note prevented to work just because of a questionable calculation.

    Regards

  • 0
    Avatar
    christophv.ogel

    I just got the mail that my english-german translation license has been revoked. I'm not not happy at all, so please excuse me if in the following lines I have to be rather direct.

    After several months of working with gengo, first my score dropped significantly, from 8,4 just a week ago to 5,3 yesterday, and now I see that my l license is revoked. I checked the feedback on those review-messages I received and I've got some commentaries about those, since I can't, or don't how to answer them directly.

    -I remember there being exactly one mistake, marked as grave in a text concerning slot machines, where I used "Münzautomat" instead of "Glückspielautomat". Initially I had used the word suggested by the person who reviewed it, but I decided to translate it the way I did to differentiate between physical slot-machine, where you actually have to put coins in and virtual ones.

    -There are several mention of double space, which I find rather odd, since I run all my translations through the word grammar and spell checker, who usually tells me, if there are double spaces, double commas, etc.

    Back to the point, after several months of good cooperation with gengo, now I feel treated somewhat overly harshly, also since those reviews are for jobs that had already been accepted. Had there been any comment or request to re-check my work, while they were still marked as "Pending", then I would have gladly done so, but getting kicked down the ladder for jobs already finished...

    First, I took the lower scores as an sign that might have been slacking of a bit, so maybe I needed a wake-up kick in the butt. I was ready to take jobs more seriously, something that right now is not possible, since the license is revoked and the "Test" page says that aren't accepting new submissions for my lenguage pair.

  • 0
    Avatar
    Cheryl

    For the consistency score, how many 10s in a row does it take to start nudging the overall rating up again?

  • 0
    Avatar
    AlexF

    It's pretty easy to find out using a spreadsheet and a lot depends on the amount of reviews you already have.

    Consistency score = weighted average of last 10 scores - standard deviation(of all scores since starting to work for Gengo except the worst)

    I did a quick calculation for my scores and after gaining that perfect 10 average which is the best way to increase the score rapidly with ten 10s, I would need an additional 24 perfect 10s to get my score up 0.1 point by modifying the standard deviation.

    Depressing isn't it?

    At the current frequency of reviews I am getting, that means one year to get an average of 10 and an additional 2 years to nudge my consistency score up an extra 0.1 points...

    More bad news? It only gets harder ! It gets more and more difficult to budge that standard deviation with a higher number of reviews.

    My advice: ignore your consistency score. 

    Edited by AlexF
  • 0
    Avatar
    Cheryl

    I did the spreadsheet, and I can't see where the consistency score is. Over 93 reviews, my average, unweighted score is 9.54, weighted scored on my profile is 8.5, I had one low rating this summer and I swear I don't remember doing that translation- is there a way they can tell who did a translation in a collection? (like if I receive a partially done correction, can they review one of the jobs I didn't do?) If so, is there a time limit to try and find out? (it was on June 5, I was overseas and missed the email).

  • 0
    Avatar
    Gyuri

    As you can see my comment above, I completely disagree with the way of calculation. I have just complained again through the support. This was my mail:

    "Aug 29, 16:00 JST

    Hello, I keep trying to refrain myself to comment the evaluation system, but now I can't.
    I did not comment, when my evaluation dropped from 8.0 to 7.9 after a 9.4/10 evaluation to 7.9. This is because of the standard deviation stuff. OK.
    Now I got a 10/10 evaluation and the average has dropped from 7.9 to 7.8. So the higher the result, the lower the average. I do have concerns about that.
    Kind regards, György Beck "
    The reply up till now was (maybe not a direct reply, but an automatic note) a link to this discussion. 
    My mathematical guess is, that in the case of an average of 8, a value of 10/10 (So there a good evaluation!) increases the standard deviation, hence the average gets lower. Congrats for the think tank!

  • 1
    Avatar
    Lara Fernandez

    @Cheryl - Partial submission of collections was decommissioned over a year ago. Which means that if you worked on a collection where some jobs had been performed and then declined by a previous translator, they hadn't been submitted to the customer at that stage (and the first translator had not received payment whatsoever for them), and it was still your responsibility to correct them if they weren't up to par, as you were being paid for the full collection. 

    Edited by Lara Fernandez
Please sign in to leave a comment.