6

Several hours ago, I received e-mail from Gengo's team, which states that from now on, translators will be evaluated with not only scorecard, but also, with "consistency score".

They explain that this "consistency score" is set up for evaluating consistency of translation quality. For instance, if your last 3 GoCheck scores are "7.0", "7.0", "7.0", your average score is "7.0". Meanwhile, if your last 3 GoCheck scores are "10.0", "10.0", "1.0", your average score is sitll "7.0", but your "consistency score" will be signicantly lower than the previous case (at least, I understand Gengo's explanation as so).

Well, the concept is understandable. Any customer should be able to expect consistent quality of translation, of course.

But seriously, why does Gengo team make such important change without making any prior notice? They say that this "consistency score" will be used for evaluation of translators.
I have searched across twitter, blog post, support page etc., but no information whatsoever was found about this "consistency score".

Besides, it makes situation very hard for translators to keep his/her status as translator.

Along with "consistency score", new "performance dashboard" was revealed. Surprisingly, scores of (probably) every translators of Gengo can be checked from it from now on. I have checked scores of several translators.

To my surprise, many translators with above 8.0 (or sometime 9.0) on his/her scorecard, receive below 7.0 for "consistency score". Such persons received some low score (below 6.0) few times, which makes his/her consistency score significantly lower than his/her "scorecard".

As far as I heard, if your "consistency score" becomes below 7.0, your status as Gengo translator is at risk. In short, even when your scorecard shows above 8.0 or 9.0, when you make even one bad mistake in any of your job, you are in trouble from now on.

Anyway, I wish to ask Gengo's team about specific explanation about this "consistency score". How it is calculated, how it will exactly affect my translator status, etc.
I myself am currently contacting with Gengo's support team about it, but thought that it would be good idea to share this information with you all.

70 comments

  • 22
    Avatar
    Nelson Bras

    Dear Gengo,

     

    I usually don't have time to participate in the forums, but this is far too serious to be ignored.

     

    Let's start by saying that my score went from 9.8 (last week) to 8.8 (this week), after 975,000 words translated for Gengo and 90 GoChecks. Nevertheless, this is not bad, according to Gengo, right? I have to disagree!

    1 - The score is the first thing a client looks at when he checks our profiles. My 8.8 score will be immediately compared to the score of someone who has just started and displays a perfect 10, just because he/she didn't have the chance to undergo 90 GoChecks, as I did. On the other hand, this is not the kind of score I would proudly share on any media, because it does not reflect my real, present performance… it means nothing!

    2 - I am selling my current quality, which derives, among other things, from the know-how obtained translating almost a million words here. What the client and Gengo are buying is my quality of today, not my performance from 4 years ago. The fact that a certain translator can have the same "consistency" effect on his/her score if he/she was bad at the beginning and now is good, as another translator who was good at the beginning and now is bad… doesn't make any sense!

    3 - Since I am a very active translator and Gengo is not my only place of work, my time is very limited, so I have ignored most of the reviewing issues… assuming that, after 3 reviews, the problem would be solved (in terms of scorecard). However, not only this will no longer be true in the future (you should prepare yourselves to receive a lot more feedback on the reviews), but also, I am going to be "punished" according to the new "law" for "crimes" for which the consequences were very different in the past, according to the "old law". Should I have known that you would be introducing such a severe rule with retroactive effects in the future… of course, I would have acted differently, notably, in terms of "appealing to the Supreme Court"!

    Why am I using legal terms? Because, if this rule was something "official", it would be illegal in almost every jurisdiction worldwide. There is a good reason why the "Principle of non-retroactivity of the criminal law" exists: http://users.cecs.anu.edu.au/~James.Popple/publications/articles/retroactive/2.shtml ///  https://en.wikipedia.org/wiki/Ex_post_facto_law

    The "consistency rule" should enter into force at the date of its publication, with no retroactive effects.

    The "consistency rule" should only cover a certain reasonable number of the latest GoChecks, in order to ensure equal treatment for every translator (newcomer or otherwise) and to promote an incentive for them to get better and see their efforts reflected in the following period.

    The consistency score should never be made public - the clients will always tend to interpret it as if they were looking at the usual scores they can see everywhere in the market. This is bad for both Gengo and the translator.

    I rest my case.

     

    Best regards,

    Nelson Brás

    Edited by Nelson Bras
  • 19
    Avatar
    Nelson Bras

    By the way, the whole "consistency" system is based on a GoCheck procedure that has never been perfect and, I dare to say, still isn't (you should GO and Check how many changes have been made to the senior translator's platform since the early days), along with the less than perfect intervention (they are only human, after all) from senior translators. However, the retroactivity considered under the "consistency" rule assumes that everything is perfect from the beginning.
    After 975,000 words translated for Gengo, of course, I have already faced a lot of the following situations:

    a) The senior translator was absolutely right;

    b) The senior translator was convinced that every grammarian in the world is wrong and should learn with Gengo's senior translators (e.g. https://ciberduvidas.iscte-iul.pt/consultorio/perguntas/ligacao-a-vs-ligacao-com/34828 - by the way, this was never corrected);

    c) The senior translator considered a minor mistake as a major mistake;

    d) The senior translator marked the same error twice;

    e) Only one segment with a misplaced tag was reviewed, out of a job where the total word count was several hundreds of words - note: the mistake would be as visible to me as it was to the senior translator if we were using the same platform!

    f) The senior translator considered style differences (subjective) as a mistake;

    g) The senior translator considered that a certain word (synonym) was more frequently used than the one I chose… in a language (Portuguese) so rich in synonyms that we pride ourselves (it is even an established good writing rule!) to avoid using the same word twice in a sentence! A mistake? Really?

    h) A senior translator considers something wrong, while another senior translator considers it right in a different job;

    i) A senior translator considers something as being an error, just because he/she lacks the knowledge in a certain field, which makes him/her trust in his/her common sense + Wikipedia, even in the presence of a scientific substantiation that he/she does not understand;

    j) A certain word can be found in Microsoft's glossary… and the next day the same word is not there anymore, making it impossible to prove why that specific term was chosen;

    k) An error is detected seconds after the job was submitted - as the jobs cannot be edited anymore, the mistake will be there, no matter what. I would suggest the introduction of a time limit (say… 10 minutes?) to make any corrections before delivering the job to the client (most of the jobs are finished before time, anyway).

    l) After requesting a correction to Gengo's support, along with the correction came an error made by the person who changed the text… my mistake… I should have checked the whole translation before telling the nice lady from Gengo's support that "everything is OK now. Thanks";

    m) A translation was submitted before time, by mistake;

    n) After facing certain types of issues (power failures, hurricanes, technical problems, either from my side or Gengo's side, etc.), with not much time left to deliver (say… seconds) I only had 2 options: submit the translation without proofreading… or lose several hours of dedication and the corresponding payment, by declining the collection and offering the work already done to a different translator, who would not have to make any special effort to earn 400/500 USD in 10 minutes.

    o) Etc.

    These are all issues that should be addressed by Gengo before adopting such a drastically retroactive measure (consistency score).
    Note: I understand that senior translators are also human beings. They make mistakes and have bad days, like everybody else. So, I don't blame them. What I mean is that these issues must be taken into consideration, to avoid building a new system on the sand, instead of solid rock. Gengo, however, seems not to share this perception, since the new system allocates a stronger weight to every decision taken so far, as well as future decisions, by the senior translators... as if they were rock-solid and error-free.

    Best regards,

    Nelson Brás

  • 17
    Avatar
    Nelson Bras

    Dear Lara and fellow translators,

     

    Thank you for reading my post and taking it into consideration.

    As a matter of fact, I have just found a different post on the same topic elsewhere (https://support.gengo.com/hc/en-us/community/posts/360020796593-A-bit-of-feedback-on-translation-scores-and-MT?page=1#community_comment_360002687294), where the translator says:

    • I wholeheartedly subscribe to the fact that a translator with a string of 7s is more reliable and sounder from a business standpoint than one with several 10s and a few 3s thrown in the mix, as exemplified by your "how it works" scoring page. But to hammer it on: you then have to make really sure that the 3s really are worth 3s. And by that I'm not necessarily saying "it should be 10 or3", but rather: "3 obviously implies mistakes, but do those mistakes really account for such a plummet in ratings?". If not, you run the risk of having otherwise pretty decent translators working with a sword of Damocles above their heads every time they submit a translation, this is both unfair and detrimental to the business and the overall quality of the work. It's a lose/lose/lose situation for Gengo, the customers, and the translators. I get that Gengo is a business, I get that running it implies handling both carrot and stick with your contractors, but there's simply too much stick here, or rather, to be more precise: handling of said stick is carried on too haphazardly.

     

    Let's focus on the "but do those mistakes really account for such a plummet in ratings?"

    Here's a real-life example on how unfair the "lifetime consistency" system can be:

     

    Check score: 2.34/10.00

     

    Source: Open your {1}store shipping settings.{/1}

    Translation: Abra a sua {1}store shipping settings.{/1}

     

    Senior translator comment: "store shipping settings" should have been translated.

     

    Damn! (Excuse my French). The senior translator is absolutely right! Anyone could see that something was missing, right?

    How is it possible that a seasoned translator like me could do something so blatantly stupid? (I am a Virgo, maybe this is why I am really severe when it comes to self-criticism and ratings below 10/10)

    The thing is… the senior translator and I… we were not playing on a levelled field. This is how the same segment is shown at the translator's platform:

     

    Source: Open your <a href="{STORE_URL}/wp-admin/admin.php?page=wc-settings&tab=shipping" target="_blank">store shipping settings.</a>

     

    Translation: Abra a sua <a href="{STORE_URL}/wp-admin/admin.php?page=wc-settings&tab=shipping" target="_blank">store shipping settings.</a>

     

    If I was using the same platform as the senior translator, I am pretty sure that the mistake would be absolutely clear to me as well…

    As a side note: considering the fact that Gengo is not a technical specialized platform, that most translators lack the knowledge to foresee the final result in cases like the one above, and that a lot of us wake up in the middle of the night, or put other tasks on hold, to help Gengo deliver a short translation and gain another happy customer for the future, I think it would be useful for everybody if the platform was adapted to simplify our work, by showing a clean text, exactly like the one we can see when looking at the GoCheck platform. The implementation of such a system is not that hard. If this happened to me, a seasoned translator who has already localized thousands of apps for every major player in the industry… think how easy it is for a less experienced translator to fall into the same trap.

    Nevertheless, I totally agree that I should be penalized for such a huge mistake. 2.34/10.00 is pretty fair if we consider it objectively. This is what I don't think is fair:

    In relative terms, considering the fact that the large majority of my GoChecks produced a 10/10 and that, within a 4-year period, I have only 2 cases where I received 4/5 feedbacks from the clients instead of 5/5 (for no special reason)… is the situation above so important that I should bear its weight on my lifetime consistency score "till death do us part"? Does it really say anything about the overall quality of my work?

     

    Now, let's focus on the "you run the risk of having otherwise pretty decent translators working with a sword of Damocles above their heads every time they submit a translation."

     

    Well… I couldn't agree more.

    The objectivity of the GoCheck system is so restrictive nowadays that accepting certain risks for certain amounts is just… silly.

    Back in the day, being a seasoned translator, I was that kind of guy who never chose or declined a job… "let them come" was my motto. I was really fast due to my experience with large texts (not careless fast!) and my average scores were never far from 10/10. It was easy for me to put the Minamata Convention on hold for a couple of seconds to translate a simple 0.4 USD job and go back to what I was doing before.

    Nowadays, I always have to think twice before accepting a 200 USD job, just because a single mistake or a different opinion from the senior translator can have a gigantic impact on my ratings.

    Why am I not afraid to work for major companies, governments and a lot of different organizations… but always feel the presence of the "Damocles sword" when I look at Gengo's dashboard?

    1 - The number of words + type of error system is too restrictive. The GoCheck objectivity does not take into consideration that there are several degrees of low/medium/critical severity mistakes, from almost irrelevant to "me Tarzan, you Jane" speech; My suggestion: more severity levels;

    2 - A single huge mistake can affect your score for life (as explained above). My suggestion: abolish the "lifetime consistency" thing as a whole or, at least, consider a specific time frame and implement the word count variable as explained in point 5.

    3 - Things that should be considered "improvement suggestions" and, as such, not being marked as errors, are now reason enough for you to lose your qualifications. (What happened to the "Unlike many other translation agencies, Gengo does not promise to deliver a perfectly polished translation (see our Quality Policy)" and so on???) My suggestion: establish a clear distinction between improvement suggestions and severe mistakes; By the way, as a professional proof-reader myself, I never flag an error without adding a proper reference, in order to help the translator understand the mistake, improve his/her work for the future and be certain that the flag is not there just on a "because I think so" basis.

    4 - Unlike other platforms, we have no chance to change anything after submitting a job by mistake. My suggestion: establish a 10 min tolerance threshold;

    5 - The weight of the word count only refers to that "damn, bad luck" segment (again, excuse my French), not taking into consideration if you have already translated 5,000 words since your last GoCheck or just 50. My suggestion: I really think that the weight of the error should be measured relatively to the number of words translated so far since the last GoCheck, in order to best reflect your quality and consistency, as if you were working on a single larger document. Here's a useful" consistency variable!

    6 - I still see (through the editing jobs) a lot of situations where the translator has no clue of what he/she is doing. Yes, I am talking about really, really, really bad translations. They are still there… so… the GoCheck system is not producing the most important expected results. My suggestion: the only way to detect and solve this kind of situation would be… by assessing the translator's consistency! However, I am not talking about his/her "lifetime consistency". What I am talking about is having a look at different segments within the same collection, in order to establish whether the error derives from a one-time distraction/bad luck moment… or if this is exactly the kind of guy who should be kicked out immediately for faking the admission test! And here's another useful consistency variable.

    7 - Etc. (please, check my second comment above).

     

    Maybe this is why I can see two 6-7 USD editing jobs hanging on my dashboard for… I think… 3 days!

    It doesn't make any sense to take the risks, as mentioned above, rewrite the whole thing (508 units) for 7 USD (6,17 EUR!!!) and allow the original translator to get away with a huge collection of mistakes + a nice looting.

     

    Thank you for your patience.

     

    Best regards,

     

    Nelson Brás

     

     

     

    Edited by Nelson Bras
  • 12
    Avatar
    sesztak.zsolt

    Freelance translator speaking here with 11 years of experience.

    This scoring system is a major mistake and Gengo already pulled out, at least partially, and soon, silently, cowardly, they will pull back completely. But we know how corporate PR works: they will never acknowledge that they made a huge mistake, but there will be the "we are listening to feedback and we are working on the solution" bullshit. The truth is that this new system overpenalizes translators for a single bad score. Add to the mix that their proofreaders are also translators and many of them look at other translators as competitors and give unjust low scores in the hope they will have more work in the future by eliminating competitors. I have seen this malicious and unprofessional behaviour many times during my whole career and on several occasions I even started legal actions against such "proofreaders".

    But if Gengo treats translators as a commodity and does not care about destroying one, believing that they are infinitely replacable, that's OK. There are many clients for me, Gengo is not the only one. If this company decides to shoot himself in the foot and trashes the translators who made them to grow, that will be their problem, not mine.

    BTW all the major brands (eBay, Amazon, Tripadvisor, etc.) use simple average for evaluation/feedback/review score. No need to reinvent the wheel. End of story.

    This comment will be soon removed by Gengo staff, so if you are reading this, consider yourself lucky.

    Edited by sesztak.zsolt
  • 11
    Avatar
    Nelson Bras

    Dear all,

     

    Let me start by thanking you all for your inputs, your presence and your interest. It seems like it is paying off and our voice is being heard.

    Lara… what a difficult job you have! We do understand your position and thank you for your efforts.

    Let's get to the point:

    Changing the consistency system in order to reflect only a more recent time frame is not a solution per se.

    By doing that alone, the benefit would be targeted only to translators who had lower scores at the beginning (not my case. I am actually being benefited by the current system, comparatively to other translators, because my scores at the beginning rarely went below 10). Not only that, but the benefit for the translators in question wouldn't last long, since the remaining factors are still there, lurking in the shadow, waiting for a distraction to eat you alive. I don't think it's a good idea to try to stop a huge bleeding with a small Minion's bandage bought at the supermarket.

    So… if my score is still 8.8 (great score, according to Gengo), what am I advocating here? A way to have my score pushed down??? Of course not! I am just looking at the whole picture (as I hope you all are), considering everything beyond my personal immediate interests, thinking about the translators who have been negatively impacted by the recent changes and imagining the possibility to be in the same position one day… because the system, as is, tends to grow more and more unfair by the day.

    By using a 1-10 scale, and by applying it to very small jobs, the weight of a single mistake (fairly or unfairly pointed out…) is too high! Just consider my example above. That, per se, is enough for any good translator to have a huge impact on his/her score, overlooking the fact that he/she, for instance, made a mistake of 6 words after perfectly translating 10000! When you look at a 2.34/10 score, you only get the immediate impression that the translator is one of those who fail to correctly translate 77% of the source text… period. You have no idea if you are talking about someone who has already translated almost a million words in Gengo alone… and you can't see that this individual was having 1 bad day out of 1440, trying to do his best at 4 AM, with a nasty conjunctivitis, just to be there for the clients when they need him (that's one way of gaining and keeping the preferred translator status), and earn enough to pay his rent… cent by cent, by accepting small jobs.

    Statistics are interesting but can also be very unrealistic if used alone. As every economist knows, if I have a rice bowl and you have none… both of us have an average of half a rice bowl… but you won't be able to eat yours!!!

    You just have to do the math to realize how this can produce a huge impact on every translator's work - e.g. in my case, the extrapolation would be: 978,000 words / 90 GoChecks - the system is assuming that 10,866 words have been reviewed per GoCheck (the reality is faaaaaar), so when I receive that one and only 2.36/10, the system treats it as if it was referring to 10,866 words, instead of half a dozen!

    On the other hand, you have this Joe who, most certainly, faked his admission test (e.g. by asking a friend to do it for him), who fails to correctly translate 90% of a large collection (not statistically, but in real terms) and who can easily deceive the system for some time when a single well-translated segment of 10 words that has been reviewed is extrapolated as if it was referring to 10,000 words! Add that to the fact that this large collection will be edited by a decent translator… and no one will ever notice!

    I believe a mix between my suggestions and Alex F's suggestions (great statistical work, by the way!) since they basically cover every input from everybody else (thank you all), would be the solution to our problem, not only for now but for years to come!

    If you approach the problem from the "consistency" point of view alone… it will never work. You need to reinforce both columns (consistency + GoCheck)… otherwise, the building is doomed to collapse!

    A lot more could be said… however… it's 6 AM, I haven't slept yet and I still need to finish a 23,000 words job… because this is what I do for a living. It's time for me to let the management and stats issues for those who earn the big bucks to think about it.  

    Best regards,

    Nelson Brás

  • 10
    Avatar
    Xavier

    So now I am a Top Scorer, Pro Translator, and a Gengo Wordsmith with 86 GoChecks and I can only access short low-paid jobs because of some bad scores I got 2 years ago... Way to go Gengo!

    Since there is no real reward apart from a T-shirt for long-term translators, and the clients I used to work for (Oodji, BuzzFeed) are now gone, I even considered starting from scratch and creating a new profile, but my language pair is already full of translators.

    And because I can only access short low paid jobs, I don't really do many, which means that it will take months before I get enough reviews to climb back up to a decent score. I raised that point before, when I got asked to evaluate some changes Gengo wanted to make on the dashboard (including allowing only translators with a score of 8 or above to access bigger jobs): once your score falls below that, since you get fewer jobs, it is harder to get reviewed and you might get stuck indefinitely.

     

    Like Nelson said, some of us used to accept pretty much any job before, and just do our best. Since the only bad thing that could happen was getting one low score, as long as you worked well the rest of the time, it was OK. Ultimately it is not good for Gengo, I agree, and now even one low score can make you lose your qualification, so I don't do it anymore. I actually decline a lot of jobs nowadays and only do the ones I am sure I can do right. But I am still getting punished for my past attitude and inexperience. Gengo changed the rules, but the retroactive weight of the low scores from before is unfair!

    Edited by Xavier
  • 10
    Avatar
    Nelson Bras

     

     

    Dear Lara,

    "Maybe this is why I can see two 6-7 USD editing jobs hanging on my dashboard for… I think… 3 days!

    It doesn't make any sense to take the risks, as mentioned above, rewrite the whole thing (508 units) for 7 USD (6,17 EUR!!!) and allow the original translator to get away with a huge collection of mistakes + a nice looting."

    Update: 5 days now! No smart translator will take these.

    Forget about lifetime consistency. If consistency is what you want, have a look at these tasks and try to assess the consistency of the far too many mistakes contained in a single job! Not just one segment, but the whole thing. If you only pick one segment, maybe that guy is lucky enough to be assessed based on a couple of segments with no issues... and receives a perfect 10/10!!!

    This is what Gengo should be worried about! This is what really hurts Gengo's image. This is what good translators here are competing against. This is the real issue!

    I believe that, by implementing my suggestions, the problem would be solved, once and for all.

    Meanwhile... several much better translators are being deprived of the opportunity to do a much better work... just because the system has turned into a casino game.

    Best regards,

    NB

  • 10
    Avatar
    JY-LEE

    Hi everyone, I really don't like standing out, but I also really wanted to say something about the recent changes in Gengo's evaluation system. So please allow me to talk about it.

    As a student of economics, I'd like to point out that the rewards and penalties resulting from an incentive mechanism will make individuals act in a certain way. This certain way may not be the one which was intended in the first place, especially when the incentive mechanism is not designed properly.

    - With the recent changes in the evaluation system, Shorter jobs impose a heavier risk to translators than before. Since the full text is very short in these jobs, one minor error can take up a large portion of the whole job and can do a critical damage to the review score.
    This is more threatening to the experienced translators since they are now more vulnerable to lower review scores than newer translators. Eventually, experienced translators will avoid these shorter jobs (I have seen shorter jobs being declined many times after the introduction of this new evaluation method in my language pair. I also started to avoid accepting shorter jobs recently; $0.2 is not worth the risk) and relatively inexperienced translators will accept these jobs.

    The problem is that these shorter jobs can act as a threshold to some customers. These customers may order shorter jobs first to figure out what Gengo's capable of. They won't come back with another job if they think the translation quality of those shorter jobs is not satisfying. To sum up, experienced translators will now avoid taking shorter jobs, and if the quality of these jobs drops, it will make the customers unhappy and they may not use Gengo next time. This can cause serious damage to this translation platform in the long run.

    - Also, The new evaluation system can be punitive and harsh to translators. This can be justified only when the review system is fair and professional. But I'm not sure that this system is working well. The reasons are as follows:

    1. Conflict of interest - According to this post, LSs are allowed to work on the same jobs as regular translators. So, LSs are our competitor and reviewer at the same time. I have no doubt that the LSs are fair and conscientious people, but the system should not allow this awkward situation. Any LS can abuse his/her power to rule out competitors, and there is no way to prevent it. Since the evaluation scores have become more important than before, this matter should be taken into account seriously. I suggest that LS should be only dedicated to reviewing, not competing for the same jobs with regular translators.

    2. A matter of professionalism - I'm not questioning LSs' professionalism in the field of language and translation here; I do not have doubt about the reviewers' expertise and professionalism and I truly respect the LSs. This is more about their role as interpersonal service providers. LSs are expected to be fair and consistent, and they should communicate to the translators with a cooperative attitude, not in a high-handed manner, since their job is to assess the translators' performance precisely for Gengo, not to punish or discourage the translators.

    Before the new evaluation system was introduced, I have requested some re-reviews, since some parts of my translation were marked as errors for reasons that I could not agree. The requests were made in a courteous, non-aggressive way (I thanked in advance a lot, and politely asked the LS to teach me and provide some guidance if I was really wrong) and I presented sufficient explanations about my translation.

    But the score of the re-review was about 4 points lower than the first review, and it came with somewhat upset comments by the LS (I think the re-review was done by the same LS who did the first review, since there is no reason for the LS to be upset if it was done by another LS). The re-reviewer did accept some of my feedback, but he/she marked a lot of other parts of my translations as "wrong terms"(Some of them were not marked as errors in the initial review, and I think it is a matter of writing style which does not affect the accuracy of the translation). I know that my translation isn't perfect and I do make errors sometimes. That was the reason I asked for re-review, to know what I have done wrong and grow as a translator, in order to do better next time. But I guess the reviewer misunderstood my request for re-review as a challenge to his/her authority. So I plucked up my courage and contacted Gengo support team regarding that issue.

    But the response from the support was quite disappointing. I'll just copy and paste some part of it below;

    "The best thing is to just accept the initial review by the reviewer and keep the feedback in mind for future translations. Requesting a re-review because you disagree with the score doesn't mean you will get a higher score the second time. "

    "You should only use the re-review form if there is clearly a mistake in the language specialist's feedback. If it is a matter of style or preference, you may be better off just accepting the score and improving on your future translations. "

    "... it is best to accept the score given by the LS and to always keep the customer in mind."

    I have requested re-review about 3 times in different jobs but never received a score higher than the ones from the first review, because the LSs marked new errors that were not present in the first review. To me, some were acceptable but some were not. At some point, I gave up and stopped requesting re-reviews because I realized it is a waste of time and will not help me in any way.

    But under the new evaluation system, many translators will request re-review since the review scores have become more important. To handle these kinds of issues smoothly, the LSs should be more aware of their role. I think this cannot be done on a personal level. Gengo should provide guidelines to their LSs regarding consistency, fairness, how to communicate properly with translators, etc, since the purpose of this reviewing system is to encourage the translators to work hard, not to demand absolute obedience to their reviewers.

    - Gengo is a decent translation platform, and I want to keep working with Gengo. But I'm a little worried since I don't think recent changes will bring out their intended outcome - The prosperity of this translation platform. I hope Gengo is aware of the fact that the fair performance evaluation system is very important, and reward/penalty according to this fair evaluation is critical to the motivation of us translators.

     

    Whew, It is always exhausting to write in a language other than my native language. I hope I have expressed myself clearly.

    Best regards,

    JY

    Edited by JY-LEE
  • 8
    Avatar
    masanpra

    Sometimes Language Specialists commit big mistakes. How does gengo control the quality of the internal reviews? How many "major correction mistakes" are they allowed to make before being warned by the Quality Control Team? If a Language Specialist were consistently corrected by a second Language Specialist, would gengo take action? Should translators report major review mistakes to the Quality Control Team besides asking for a re-review?

  • 8
    Avatar
    Xavier

     

    Thank you Lara for your answer! I would like to add one more feedback/opinion based on what you said and on the way Gengo works overall:

    "This is why we would like to really emphasize that not only has your score changed, but our understanding of what good scores are has also changed. [...] Even though our customers value consistently high quality, the previous overall translator scores focused only on whether quality was good on average. The new score builds on top of that to add quality consistency into the calculation."

    It is true that everything we translate will be used by someone who is paying for a service, and that we should always aim to deliver the best translation possible. It sounds normal then for Gengo to emphasize the need for better quality and the strictness of the reviews.

    But in my humble opinion, the way Gengo works overall has a flaw: those "reviews" are done after the job/collection has been sent to the client. The GoCheck system is a punitive way of checking how translators perform, and as many people have already stated, the LS are also human beings. No matter how professional and experienced they are, they can make mistakes too. And though recent changes sound like Gengo is trying to get rid of under-performing translators, I am not sure that the general level of quality will increase, since its low rates attract mostly inexperienced translators.

    Gengo was my first job as a translator (needless to say, I wasn't always deliver top quality work at the beginning), but I have since worked for different translation companies, and that has made me realize some of the things Gengo is doing really wrong. And the GoCheck for me is maybe the biggest one. Instead of that, I think Gengo should hire a lot more LS or editors, and proofread (I don't want to take about reviews anymore) as many jobs as possible before sending them back to the clients. That way, the client would get a flawless translation, Gengo's reputation would improve, and the translators could get a meaningful feedback from the editors that will help them work better in the future (instead of focusing on the grade like a student and ignoring all the feedback). I have gotten much better by working for a company that uses this system, because I received constant positive feedback that helped me grow. Here, every time I receive an email saying that my job has been reviewed, all I feel is stress and fear. That is not a good way to work ... and since this new system has been put in place, it's even worse!

    I know you are going to tell me that Gengo is "cheap" for clients, it can't afford to pay so many editors, and that clients already have to option to get the job reviewed before receiving it (by another translator, not necessarily qualified to edit...) but I truly believe it is the only way Gengo can become a serious translation company and not an OK translation platform.

     

  • 8
    Avatar
    masanpra

    If gengo applied the consistency score to the reviewers, how many of them would keep their qualifications?

  • 8
    Avatar
    Nelson Bras

    Dear Lara, fellow translators and Gengo team.

     

    Thank you all for these joint efforts to find the best solution.

    While I am waiting for my last post to be approved,  please allow me to cover a different subject: math.

     

    Gengo's math is also very difficult to understand. For the sake of transparency, would you be so kind as to answer the following questions?

     

    1 - How is it possible that 2 minor mistakes covering 4 words (word structure - low / syntactic error - low) in a 3584-words job can produce the same effect (3%) in your score as if 108 words had been affected by a severe mistake?

     

    2 - Is the scoring affected by the number of words selected by the reviewers? If this is the case, why do they select additional words, punctuation and even unrelated tags?

     

    3 - Is there any difference - weight, type, criteria, etc. - between errors that can be found in "pro" and "standard" jobs?

     

    4 - Do the ST's have an option to include a comment/suggestion without pointing an error? If so, why do they always point an error when they make an improvement suggestion for the future?

     

    5 - Does a re-review count as an additional review, instead of substituting the "wrong" review, in terms of standard deviation? Because this is what it seems to be happening.

     

    6 - Do you really think it is fair to apply the same standard deviation and "worst rating deletion" formula to someone who has already translated 1 million words (thus having much more opportunities to commit mistakes or to be subject to unfair reviews, etc.) and a recent translator who only has 500-2000 words in his/her portfolio to prove his/her value? I am pretty sure that this is unfair for both! Those who think this change is enough will certainly change their minds after translating 1 million words! The only way to create a levelled playing field would be to limit the standard deviation to a pre-established period - 30, 60, 90 days…

     

    7 - Since the result of a re-review is now a ghost that will haunt you for the rest of your days, don't you think it would be fair to allow the right to appeal to higher instances in case of an absolute well-founded disagreement?

     

    As I mentioned before, the way I see it, it makes no sense to keep spending more and more time/money pruning branches when the real problem lies in the root of the tree.

     

    Note:

    My score is 9.1 nowadays. Should I feel happy and keep quiet? No way! - I can see clearly where this path is going in the near future for every translator, and even for Gengo as a whole. I love this platform. It is still one of the best. That's why I keep working here despite the low rates. Please, don't let it go down as a result of a couple of bad but crucial decisions!

    Thank you for considering my thoughts.

     

    Best regards,

     

    Nelson Brás

    Edited by Nelson Bras
  • 7
    Avatar
    Sara

    Nelson said: "Since I am a very active translator and Gengo is not my only place of work, my time is very limited, so I have ignored most of the reviewing issues… assuming that, after 3 reviews, the problem would be solved (in terms of scorecard). However, not only this will no longer be true in the future (you should prepare yourselves to receive a lot more feedback on the reviews), but also, I am going to be "punished" according to the new "law" for "crimes" for which the consequences were very different in the past, according to the "old law". Should I have known that you would be introducing such a severe rule with retroactive effects in the future… of course, I would have acted differently, notably, in terms of "appealing to the Supreme Court"!"

    I definitely agree with that. There were definitely reviews I didn't agree with, and while I appealed some, I was busy and my average was usually high enough that I figured my average would sort itself out after a couple more checks. If I had known consistency checking was going to be a factor, then I would have appealed every 'questionable' review.

    I appreciate that the job of the Gengo reviewers is not easy, as I have done reviews for 2 other companies. One thing I have always tried to do when faced with stylistic differences as a reviewer is to indicate in my feedback what I thought would be a better sounding option, but not mark it as an error if it were truly a matter of stylistic preference. Looking back, I feel that I was unfairly penalised at times as a Gengo translator for making a different stylistic choice than the reviewer, and now I am sorely regretting failing to appeal those reviews. I also had a least one low score due to a technical error. Once again, I just let the situation go because I thought my average would quickly improve itself. I regret that as well. I am really not a fan of the new system.

    Edited by Sara
  • 7
    Avatar
    Sara

    Lara, thank you for your kind and detailed response. I do, though, have a question about GoChecks. I have recently requested a re-review of 3 recent translations in the DE>EN(UK) pair. My score improved upon re-review in two of the theee (haven't heard back yet about the third). In each case, the same reviewer was re-reviewing his or her own work. Is that typical? I thought another reviewer performed re-reviews instead of the original reviewer. Has there also been an overhaul of the reviewing teams behind the scenes as well? Does my pair only have one reviewer now?

    To the original reviewer's credit, he or she did remove some marked errors and my overall score has increased. But that makes me wonder how much better my score might be if I had requested a re-review of every questionable review since I started translating in 2016. As I said before, I do understand that the job of a reviewer is not easy. But as the errors they mark can lead to us having a poor consistency score and losing our qualifications, it is certainly important that translations only be marked for true errors and not just stylistic preferences. Is there a way that reviewers can give us feedback when they feel their suggestion is a better option, rather than marking our stylistic choice as an error?

  • 7
    Avatar
    Nelson Bras

    Dear Lara,

     

    That's certainly a nice effort to get closer to a better solution. Thank you.

     

    Nevertheless, the main problem is yet to be solved.

     

    Being a 45-year old seasoned translator/proof-reader/reviewer/editor for some of the major translation companies worldwide, with more than 1 million translated words and 665 happy customers served just in Gengo (top scorer, pro translator, Gengo wordsmith, millionaire's badge and numerous "preferred translator" awards), having an extensive portfolio (including multinational companies, governments and international organisations) and some of the best quality ratings in the industry, and providing for a family of three through my translation work, exclusively, there must be a good reason why, for the first time since 2014, I keep declining more and more tasks here….

    As a matter of fact, there is!

    It's not because I don't know how to translate these jobs, but rather because I fear to be trapped in the unlucky situation where my target language allows for multiple choices with the same meaning, and my skills don't include guessing the senior translator's "mood" regarding my choice of words!

    Being a proof-reader/reviewer myself, the time has taught me that I will always be learning till the end of my days - Modesty is a great thing, since it allows you to improve with every experience, even by learning from the very individual you are tasked to review!  - No one knows everything about every subject. And this is why, before rating something as a mistake, I always make sure to be properly backed up by the opinion of one or several experts. My reviews are never based on "because I say so and I am on a higher level", but rather substantiated through links to expert sites, real grammar books and practical examples. Most of the times, I know the reasoning by heart… but I always try to pass that knowledge to the translator, by providing him/her the opportunity to check the official resources and improve his/her fluency. There are certainly cases where I don't know/use a certain term… however, this is not reason enough to immediately consider it a mistake - instead, I always try to do some research on the term - even though this term is not used by my circle of friends or in my region, even though I don't remember seeing it in any of my usual reading material… perhaps this is just a term which is commonly used elsewhere, a specific industry jargon term, a more literary term… So, I research!

    As far as I can see, none of the above applies to Gengo's senior translators. Their opinion, right or wrong, is law. No need to explain, no need to teach, no need to justify, no matter if they can't find any official rule to support their choice (when this is the case, they just make new rules as they go and invent a whole new grammar, according to their personal preferences and to what they think is the "prettiest" word!) The translator has no chance to win this game, because the game is rigged from the beginning!

    Remember this case referred to in my previous posts? - " The senior translator was convinced that every grammarian in the world is wrong and they should learn with Gengo's senior translators (e.g. https://ciberduvidas.iscte-iul.pt/consultorio/perguntas/ligacao-a-vs-ligacao-com/34828 - by the way, this was never corrected);"

     

    I have just received another review showing that this is not a "one-time situation".

    Now, let me start by saying that this is a 3584-units job (standard!).

    Three mistakes have been marked by the ST:

    1 - There was an "s" missing at the end of a preposition+article contraction ("à" should be "às", in order to agree with the plural word right next to it). Even using a second spell checker (MS WORD), this little guy managed to escape my attention! Well… fair enough. This is definitely a mistake. What I don't understand is the decision from the ST to select/affect two words just to rate a one-letter typo, thus aggravating the weight of a micro-mistake… mmm… is there any limit for the number of words they are allowed to select?

    Now comes the most interesting part.

    Perhaps the text was too long, and the reviewer got bored of not finding any real errors, because this is what happened next…

    2 - "The duration of your stay also plays a role." - translated as "A duração da sua estadia também desempenha um papel importante".

    According to the ST, by using "desempenha um papel" (literally, "plays a role"), I am committing a serious offense against grammar rules, since "the duration" is not a person, ergo, it cannot play any role! Just like that, as in "the Sun revolves around the Earth, because this is what I see", with no further substantiation added.

    Upon requesting a new revision and explaining the meaning/importance of using figures of speech/rhetorical devices, such as the prosopopoeia and the hypallage, in both English and Portuguese literature (as one can easily determine by reading Eça de Queiroz, Fernando Pessoa, Saramago, etc.), I was finally successful in preserving the good name of several generations of Portuguese literary masters! The "mistake" has been deleted by the ST.

    My question is: if a Senior Translator blatantly fails to recognize this kind of material, is he/she fluent/knowledgeable enough to perform the task? Am I supposed, as a translator, to invest my time teaching basic grammar to an ST just to preserve my scoring each time I deliver a translation? Am I supposed to always be afraid to accept a task just because I don't know if the ST is going to apply official or personal criteria?

    OK, STs are also human beings, they do have "bad days", and they are not all alike. Nevertheless, what really bugs me is the frequency of the recidivism and the lack of modesty preventing some of them to do proper research before chopping someone's head off! If I do that research, why don't they?

    On a side note: I remember a time when the STs and I always had good "win-win" conversations when an error was found. Back then, the STs used to identify themselves, we all learned from each other and the use of "hello"/"goodbye"/"thank you", etc. was always present. I really miss that kind of human touch. Is Gengo hiring almighty unreachable gods nowadays? Are they too perfect to say "Hi" to simple human beings? Or do they fear to be immediately linked to their bad decisions?

    3 - Speaking of "fluency", the third "mistake" was all about "lack of fluency" (which I was accused of by the same ST who never heard of figures of speech in his/her life!)

    Let me remind you that we are still talking about the same "standard" job as before, not a "pro" job (and even if it was…) and tell you that this "mistake" was not cleared after the second review.

    Since we only have one chance to get a re-review and the "comments" field is often not enough to let me copy/paste the "homework" the ST should have done before using his/her "sword", this is now irrevocable, thus affecting me for the rest of my life at this translation platform.

    Here's the "mistake":

    "Many people also speak excellent English and Spanish" - translated as - "Muitas pessoas também falam inglês e espanhol de forma excelente".

    According to the ST, "de forma excelente" is not fluent enough, and I should have used the word "fluentemente" (fluently) instead.

    Well… since the ST did not bother to support his/her opinion on anything else but his/her personal belief, the burden of proof rests, once again, on the translator.

    There are too many ways to prove the ST's opinion not only is absolutely wrong but also denotes severe lack of fluency. Yes, I mean it, because fluency includes mastering a language in a way that allows you to use different constructions to express the same meaning. If one only knows a couple of sentences/words to express a certain meaning in a language as rich as Portuguese… it can only mean that his/her vocabulary is too limited to be considered an "expert".

    Regarding this particular "mistake":

    1. The original wording is "Many people also speak excellent English and Spanish". If the author intended his/her words to be translated as "fluentemente", I am pretty sure that he/she would have said "Many people also speak English and Spanish fluently". Who am I to say the author is wrong? Especially when there is no grammar rule preventing him/her to use the construction above in English?
    2. Likewise, there is no rule preventing me to use that same expression in Portuguese… so, why not respect the author's choice and be a translator instead of an overzealous editor?
    3. If there was still any doubt, "de forma excelente" can be found in thousands of official Portuguese documents - https://context.reverso.net/translation/portuguese-english/de+forma+excelente /// https://www.linguee.pt/portugues-ingles/search?source=auto&query=de+forma+excelente /// https://tvi24.iol.pt/videos/merkel/portugal-cumpre-programa-de-forma-excelente/53f517153004540d1c50d86a /// https://tvi24.iol.pt/videos/merkel/portugal-cumpre-programa-de-forma-excelente/53f517153004540d1c50d86a /// https://tvi24.iol.pt/sociedade/incendio/dispositivo-de-combate-a-fogos-esta-a-responder-de-forma-excelente, etc.
    4. Since the ST states that "de forma excelente" cannot be used with the verb "falar" (perhaps this is the only verb "allergic" to excellency???), he/she should have shown at least some evidence to support his/her belief!
    5. Even if we consider the need to "polish" the text due to the fact of being a "pro" job… WAIT A MINUTE, this is a STANDARD job, remember??? Is there any difference between "pro" and "standard" jobs besides the fact that a "standard" job pays you 0.03 and a "pro" job pays you 0.08??? So… less pay, same level of demand/risk???

     

    Now… add this to the fact that a bad rating will haunt you forever under the new system… and you have the perfect recipe for disaster, as well as the answer to why so many great experienced translators are declining more and more jobs nowadays.

     

    Are you looking for a cheap solution to improve quality? Why don't you start by establishing clear rules for STs regarding the situations above? Rule number 1: Every decision must be duly substantiated. If no rule can be found to substantiate the decision, the ST should refrain from pointing errors based on personal convictions; etc.

    Remember: in contrast to what happens when a translator makes a mistake, a bad decision from a single ST may easily turn into an outbreak of bad translations by the thousands!

    The way I see it, it makes no sense to keep spending more and more time/money pruning branches when the real problem lies in the root of the tree.

    Please accept my apologies for such a long text and thank you for your cooperation.

     

    UPDATE: I have just received another review - again, 3  unreal mistakes (seems like this is the ST's magic number)… and here I go again… losing precious time, asking for a re-review and teaching grammar to someone who is paid to be an "expert". Mmmm… Don't get me wrong, but the corrections are so unreal that it almost feels like someone is trying to trim down the competition! I really hope this is not the case.

     

    Best regards,

     

    Nelson Brás

    Edited by Nelson Bras
  • 6
    Avatar
    AlexF

    Dear @Lara,

    Dear fellow translators,

    Dare I suggest that Gengo look at another measure of consistency than the standard deviation parameter?

    Standard deviation works well on a "normal" distribution of scores, a Gaussian curve where the average value is more or less at the center of a symetric bell-shaped curve, with more or less as many values above as below (median close to average)

    In the case of our scores at Gengo, this bell shaped curve is pushed against an upper limit of 10 (because we all try to reach it consistently of course :-)) and the occasional "accident" has a much bigger impact on standard deviation than an exceptionally good score because there is not much room on the right of the medium compared to the left...meaning that compensating for an accident is particularly difficult....(distances to the mean are squared before being added.. but maybe that's going into to much detail.)

    In the current case, with very asymmetric distribution patterns, it appears much more relevant to look at decile or quartile values.

    For example, the value for which 90% of the scores are above (1st decile).

    (note: Gengo's current formula, if it were a perfect normal distribution, is the value for which 84.1% of values would be above, so 90% is even more stringent...)

    In the case of a bad day (or of a contested Gocheck review) that would lower that 1st decile value, the translator would need 9 higher scores to get it out of the picture.

    It also means that if you have 50 or 100 Gocheck scores under your belt, the 5 or 10 lowest of those will be ignored and the 1st decile value will be your next worst score.

    (However, I also vote for a time limit on the scores that are taken into account, considering how my skills have improved over the last three years..)

     

    As I obviously don't have access to real-life translator datasets of Gengo scores, I would like to ask my fellow translators to try it out on their scores and give me their feedback. 

    (BTW, it would be nice to have access to all our scores in our Gengo account, instead of having to extract them all out from emails dating back a few years...)

     

    Just a suggestion but I would love to have any feedback on this, good or bad!

    (If you don't want to share your results, just vote my post up or down based on what it means for your consistency score. Does it work better for you? Does it have more meaning? Would it be encouraging? or the opposite?)

    Hope this helps

    Alex

     

     

  • 6
    Avatar
    gunnarbu

    I have a suggestion for improvement of the system which would probably increase the score for many translators, while still maintaining the intention of measuring consistency as one of the important quality parameters. 

    As in many other connections, like e.g. in scoring of performance in sports, one could remove the single lowest score (Or the lowest score for each whole year, or similar models …..)  from the track record of each translator, to smoothen the curve. In this way, a top scoring translator with a long time consistent track record of only top scores, but only one single very low score from a bad day, will not have this hanging with him or her forever, which is what happens now. 

    I am in a similar situation myself, and I feel that it is very demotivating to know that this one single very low score will haunt me forever, making it very hard to improve my score significantly.

     

    Just a suggestion.

     

  • 5
    Avatar
    Antje

    The translator support information page about job reviews (https://support.gengo.com/hc/en-us/articles/231441287) still states:

     

    "Please keep in mind that we will not remove your qualifications if you only have 1 bad score.

    Qualifications are removed if you consistently fail to meet Gengo's quality standards."

     

    Yeah, right...

     

     

     

    Edited by Antje
  • 5
    Avatar
    Xavier

    Thank you for raising those points Nelson!

    I thought the consistency score was only based on the last 10 jobs, so I didn't think much of it. But after seeing my score drop from 9 to 6.7 suddenly, and reading your comment, I realized that that score is based on the "entire checked job history".

    As you said it, that seems rather unfair for old translators! We improve with experience and practice, the scores we got a year ago or two years ago shouldn't be taken into account anymore! And if we had known about this before, maybe many of us would have stayed away from jobs that were difficult, in a poor source language, with tight deadlines, etc. We are now penalized for something that, before, could only at worst give us a low score that would disappear after 10 more reviews... (and I am with you on the LS comment, they do a great job, but mistakes can happen. It is already hard enough to plea your case and get a re-review, the scores have too much power now).

    In my opinion, Gengo should reset all the translator scores before implementing this new harsh system :(

     

     

  • 5
    Avatar
    tashioh

    Hi Lara and fellow translators,

    Lara, thank you for your clarification, "stylistic choices that are otherwise correct should not be marked as errors"! However, my stylistic choices were counted as minor errors numerous times as well and I discovered recently that I was not alone in Japanese speaking community. It was one the main topics even before the implementation of this new consistency scoring system. Filling out a re-review request takes a chunk of my time otherwise could've been spent translating so I didn't bother doing it each time I found an unfair assessment, and now I regret it too. I wonder how many more I'll end up filling out now that we have a new punishment system. : (

     

  • 4
    Avatar
    Jon

    Hi Everyone!


    I’ve already pretty much said my peace to Gengo Support regarding this, but I figured I’d post it in the forum too, in case there were others in a similar spot, or maybe just to vent a bit more, likely both. I don’t want to call out Gengo for anything here or try to pass off the blame, all I hope to do is illustrate that there are a vast number of translators on here who can all be affected by this is varying ways.


    I promise I will try to keep it as light-hearted as possible!

    I believe I may be a victim of this “Consistency Score”.

    Tonight I received a message informing me that my qualifications are in jeopardy with Gengo due to new quality standards in my language pair (Jp-En).

    I was a bit shocked. My score’s at 8.2, even though that is the lowest it has ever been during my time here, I guess there is no point in being embarrassed about sharing it since scores are now visible to everyone anyways.

    I am not proud of my 8.2. I actively avoid going to the screen that shows it, because it shames me. I kept a 10 for the longest time when I started out, and I find anything else unsettling.


    I vividly remember the first poor review I got. It was in January, four or five months after I’d started working with Gengo, on a one or two sentence fragment of a larger collection. It was 2.5. I was not pleased. My second came sometime in the summer, June or July I think. A customer had sent back a job they did not like, a first for me. I got a 1.3. The job had been a bizarre mixture of English, Japanese, and emojis, with little definable sentence structure. I shouldn’t have taken it, but at the time I thought it looked fun. I considered disputing the review due to the nature of the source text, but I figure if the customer didn’t like it, it is a failure on my part, regardless of if it gets reviewed or not.


    Now, I’m semi-active on the site, I’ve got other commitments, but I’d say I average about $200 a week on here. So, two unacceptable reviews should be easily overcome, especially nearly 11 months after the first one.


    My issue is, between my time zone and other commitments, it’s difficult to grab some of the smaller jobs, so I’ve always tended to aim for the bigger collections I can do over a few days when I find the time.


    Unfortunately, these jobs are almost always sent as files, and not on the workbench. So they do not get reviewed. This makes any single poor review difficult to recover from. Of course, these large jobs can be sent back for review if the customer is not satisfied (emoji job!), so you can’t skimp on quality. However, jobs accepted by the customer, jobs with good customer feedback, and “Preferred Translator” designations etc. are in no way considered when scoring you.

    So you can go down (hard), but not climb back up. It is very difficult to accept your work being called inconsistent due to two reviews when over that same time period you have completed several thousands of dollars of work for Gengo.


    At this point, I’m busier with life than I was when I joined Gengo last year, so I haven’t been able to take quite as many jobs as I once did. But I do have a few “Preferred Translator” customers who provide rather consistent work, and who are willing to work around my schedule. I like working with these customers.


    So my dilemma is, now that I know my future employment here hangs by a thread, and that one poor review will do me in, what am I to do?


    At this point probably 90% of my Gengo income comes from regular customers.


    Now, do I risk it? I hate the 8.2., but regardless of the amount of confidence I have in my work, I see very little reason for me to go out of my way to take some $6 job off the site, and run the risk, however low, of losing my qualifications and regular customers. The risk/reward just isn’t worth it.


    Now the obvious rebuttal is that, if my work was consistent, I’d have no problem doing a few jobs to improve my score.


    I’ll counter this by having them look at my list of completed jobs. With the steady stream of hearts representing satisfied, repeat customers. I’d say that’s rather consistent.

    Ok, I doubt anyone actually read this far, but I’m done.

    It's 4:30 AM and I’ve been doing Gengo stuff for the past 15 hours. Of course, only have of that was working, and the other half was complaining about being underappreciated to, myself.


    Lara, I know you’ll have to read this horribly long, rambling, post. I'm sorry! Thanks for the tote bag! Got it last week!

  • 4
    Avatar
    Xavier

     

    Hi 亮!

    I am personally in favor of the new system using only the scores since the date of the implementation (about 1-2 weeks ago, was it?) to calculate consistency, because like we discussed it in great  lengths, it is not fair to see scores we got when the rule was different interfere with the current score that gives us or not access to more jobs. Like I said before, there are many jobs that I wouldn't have taken if I had known that I wasn't only risking one low score, but a long-term effect on my potential incomes as well (just before Christmas by the way, thank you Gengo!)

    And in the future, for that consistency to use scores dating only a few months top, not all our history.

    What do you think guys?

     

  • 4
    Avatar
    Tony

    My mind is boggling at this drastic change, really, it smacks of senior management blanket decision making, at someone in the upper echelons who's had enough of just enough. I hope I'm wrong, but if not, so be it.

    The reason I am finding this change so unfathomable is, under the new changes, I am wondering whether there is a single Gengo translator with a 10 (or even 9+) score. Is a 10 score even realistically possible now??

    If, as a customer, I find out that there's this crowdsource traslation firm with a pool of slightly above average quality translators (based on their 6-ish 7-ish 8-ish average scores across all translators). It's called Gengo. I'll look for another, thanks but no thanks.

    Notice how your public profile is still showing your older, higher score. No doubt that'll be changing eventually, I think I read from someone, sometime back in the past.

    I'm all for high competency and quality at all times, something I continuously strive to aim for (I can't aim for a 10 any more), but this is quite a heavy burden for me to shoulder (I have to aim for just surviving), others too it seems. What sort of company am I working for? The type of company that's ok with sevens and eights?

    Peace.

  • 3
    Avatar
    Antje

    I have just had one of my language pair qualifications revoked because of just 1 (one!) bad review that I got 6 months ago.

    I don't know what exactly went wrong back then, but as I was just starting out working with Gengo (after registering years ago and then not being allowed do the tests) and the score was a ridiculously low 1.1 (according to the email listing "recent" reviews that I got a few days ago) and I certainly never handed in work that was that bad, so I can only assume that this job had a technical error of some sort - maybe a file didn't save properly or I messed up the html code or somthing like that.

    Other than that one outlier, I had never gotten a review below 7.0 and in fact only one other of my reviews in that language pair was ever below 8.0 (with more than half of them between 9.0 and 10.0), which is why my scorecard quality score was still at 8.1 despite the one very low outlier still being counted towards the average when the qualification got revoked.

    The feedback I got from customers also almost always was 5/5, excepting one mediocre 3/5 score from a completely unreasonable customer who didn't provide a well-written or at least grammatically correct orginal text and then didn't even react to comments or ask for revisions or tell me what I did wrong in the feedback. (I think they just got annoyed that I, trying to be helpful, politely pointed out to them that their original English text had too many mistakes and weird sentence structure to be readily understood by their prospective English-language website users and that they should better get a proper translation from their own language into English before comissioning translations from English into other languages.)

    Because I got a vaguely worded warning about my impending revocation a few weeks ago per email, I did try to improve my scorecard score (not realizing that the problem was just the 1 outlier and not the average score), but then I barely ever got reviewed on the about half dozen jobs I managed to do in this language pair (there wasn't much offered) over the time since I got the warning email. For one week right after the warning email, it was as if all the reviewers had gone on vacation simultaneously, as I didn't get reviewed in the reverse, much more active language pair either. This lack of job opportunities and reviews, and bad internet connectivity issues keeping me from working at Gengo over 3 months in the summer, is why I have only 3 recent reviews in the language pair in question - though those were 9.2, 10.0 and 8.3, so I did my best to prove that I can do better, as the warning email urged me to do.

    How am I supposed to improve my score if you won't review most of the jobs, even for a relative beginner? Also, as the OP explains, it wouldn't have mattered, as the consistency score is apparently not really improvable, no matter how many good scores you have.

    And as my revocation shows, this assurance in Lara's comment above:

    "For someone with a stable record who never falls below 7.0, a lower score would be an anomaly. While it would affect their consistency score, it wouldn't hurt it as much as in the case of someone whose work fluctuates between lower and higher scores every time they receive a GoCheck, therefore lacking consistency."

    is simply not true.

    They WILL kick you out if you make only one serious mistake, ever!

    (Or if you ever have just one extremely unreasonably customer as the translator above me describes. So, the message from Gengo is: Don't try to do the frequently apprearing jobs from Asian customers who submit those barely comprehensible English texts to be translated into other Western languages and who can't or won't communicate in English to clear up misunderstandings. Don't try to be helpful and do your best to turn those texts into something understandable by making guesses and judgment calls. Just ignore those jobs. Otherwise you will be fired sooner or later.)

     

    And of course I can't requalify because Gengo has blocked the tests in the language pair they've kicked me out of.

     

     

    P.S. Do I understand the previous commenter right? Do we automatically get a worse review score if the customer requests a revision of the job? Because I've been telling my customers that I'll be happy to discuss changes and to alter my translation accordingly if they send it back for revision, especially if the customer did not manage to answer clarification questions in time before I had to hand the translation in, as is usually the case for jobs done in the workbench or even the shorter Word / Excel file translations. I do this as an extra (unpaid) service to help the customers get the best possible results from Gengo, so they will use the service again. But if Gengo punishes this use of the revision system, then I will stop trying to be helpful in the future.

  • 3
    Avatar

    To All

    This is the OP.

    First of all, I feel very sorry that jwcpetty and Antje have received warning, or even worse, been revoked of your translator's qualification.

    Regarding this topic, I found some interesting material today.

    Have you read the topic named "New standards for quality of jobs" in this Gengo's community forum?

    It was set up almost month ago by the translator named Xavier. According to his comment, he received an e-mail from Gengo, which content was, as he stated:

    "[Gengo was] saying that they were "doubling up [our] quality efforts" in my language pair (EN>FR), and I assume other pairs as well. Doubling up later translates as "we are now doubling up on our efforts to take action when translators do not meet this requirement" (aka getting scores above 7). Until that point I didn't have much of an issue with this statement. But then it says "any translator who receives two scores below 7 will have their qualifications removed", and I am starting to panic a little."

    It was, from my point of view, interesting because what Gengo was explaining in the email sent to Xavier was very much like what they explain about "consistency score". In that forum, he was also showing an answer he received from "community manager":

    "There is no specific timeline for the measures you mention, meaning that a translator who receives two scores under 7 has high probabilities of being demoted on the spot. However, in cases like those you mention (when those scores happen months apart), this is unlikely to apply. The keyword here, and what we are really looking for is "consistency". So we are looking closely at score fluctuations. Think for example of someone who receives a 10 today, a 5 tomorrow, a 7 the next day, and a 3 next... obviously this person's quality is pretty inconsistent and varies from GoCheck to GoCheck - these are the kind of people who are most likely to get demoted after two scores below 7. On the other hand, if you consistently score above 7, but you receive two lower scores, months apart, when we look at the fluctuation we would see that those are isolated cases and your score is actually pretty consistent."

    It is getting more and more like "consistency score". Only difference is that there was no mentioning about the term "consistency score" in his comment. I wonder when they come up with the idea...

    There is also one more thing I wish to comment about this topic: I have not received any such e-mail received by Xavier and other translators. Maybe these e-mails were only sent to EN>FR translator, but who knows.


    Anyway, what mentioned in Xavier's comment was that two lower score will make you revoked of your qualification.

    On the other hand, Antje was saying to have received only one lower score and revoked of qualification.

    My conclusion: Gengo is currently developing new evaluation system and things are changing quite rapidly, perhaps even now. That is the reason why they cannot show their new evaluation method on their support page.


    In my previous comments, what I have been asking was this: please give us more information about this new evaluation method.

    It seems like I need to do this once again. If no new information can be given, at least, I wish to know more inforamtion about their schedule.

    (Ex. When can we find more information in our support page?)

    The problem is, I have not received an answer to my previous request from Gengo support (regarding this topic) for quite a long time by now. Maybe that they have been discussing about what to do with the matter, but again, who knows?

    Anyway, hopefully this new information can be any help to you all!

  • 3
    Avatar
    AlexF

    Hello again,

    I would also like to add that all our statistics should be shown on our Dashboard.

    Had you left our rolling average untouched, and simply added a new indicator called consistency (explaining how it was calculated/used, and replacing the totally useless "hours translating" indicator for example), I doubt you would have created such an upheaval.

     

    A positive side-effect of all this upheaval is a pretty vibrant forum! I sincerely  hope we can keep it up on some other community-oriented topics!

     

     

     

  • 3
    Avatar
    Nelson Bras

    A senior translator considers that the word 'x' is more often used than the word 'y', and this is enough to be recorded as a mistake with a huge impact on your overall score...

    A senior translator considers a minor mistake as a major mistake just because the option is there...

    And another one bites the dust!

    According to Gengo, the reviewer has no access to the final score upon finishing his/her work... Well... maybe they should, in order to have a clear view on the result of their actions and... perhaps... go back and try to be less severe in certain instances.

    Just a thought.

    Have a nice week everyone!

     

  • 3
    Avatar
    Xavier

    That is good news! Let's hope that the changes will be real and not just for the show...

    Since the consistency score has been implemented and I can't access higher-paying jobs, I refuse to work for Gengo and earn my money on other sites. And considering that I have recently received 4 or 5 emails about collections/jobs that need to be done urgently and offering an extra incentive (as opposed to one a month before for S [customer name has been removed - Lara], which nobody likes), it looks like I am not the only one :)

    Not to mention the TM jobs which we are discussing on another thread as well, that pay even less than the already low rates Gengo offers, make us waste more time than anything, and offer a significant risk of getting a low score since we have to work both as a translator and an editor (which with the current system is a too serious to take any chances)...

    I don't know if Gengo earned more money in 2018, but as for translators satisfaction, I personally went from an 8-8.5 to a 3!

  • 2
    Avatar

    To all,

    This is the OP. In case if anyone is reading this topic right now, I wish to tell you this: Gengo has changed its scoring&qualification process so drastically.

    https://support.gengo.com/hc/en-us/articles/231441287-How-Gengo-measures-quality

    As all of you might have seen the above-mentioned URL already, fron now on, the "consistency score" has become the standard way for reviewing Gengo's translator. My score has become significantly lower than before (but somehow manage to maintain my qualification). But in the very least, now that we all know how we are judged.

    Since the announcement of "consistency score", I have been confused by Gengo's attitiude. It was unkonwn to me whether they were using "average score" or "consistency score" for judgement, because they did not put any explanation about consistency score on their website, and instead told us about "consistency score" only with their e-mail. I have been requesting them to make their explanation on their website and...now that they do, and thier explanation is so detailed one.

    As a translator, I believe I should be satisifed with this. Personally, I still feel that their judgemetn criteria is too severe, and I am not sure how many translator can actually survivie as the Gengo's translator after this change in judgement criteria. But that is not my business.

    Well, good luck folks. Hopefully you can make it.

     

  • 2
    Avatar
    Sara

    Lara and fellow translators: I misunderstood - it turns out, another reviewer was re-reviewing my work after all! But I still wish I had requested more re-reviews all along since I started at Gengo due to the current scoring system. But, oh well! I've sent Lara an email and I am glad that we all can have a dialogue here in the forums about the scoring situation.

Please sign in to leave a comment.