"Consistency Score"

70件のコメント

• 1
  
  swing 2018年10月18日 03:48

  Thanks for sharing. I can't agree more with you.
  Also, I don't get the idea why Gengo revealed the "performance dashboard" where all translators' score history can be checked. Everyone's score detail should only be checked by himself/herself. Why Gengo is now publishing this information?

• 1
  
  亮 2018年10月19日 03:54

  This is the topic owner.

  To swing:
  I have just found out that Gengo team changed the "performance dashboard", and it became impossible to review "history of GoCheck" of each translator. "Scorecard" and "consistency score" of each translator can still be reviewed, but in order to review them, one has to know exact ID number of that translator.
  Well, now that privacy is secured. Hopefully you feel safer. But why they did not inform us about such change anywhere?

  Anyway, let's go into the main topic.
  I have received answer from Gengo's support team.

  They said that this "consistency score" is "not exactly new rule" because " we were already expecting from all translators to meet our quality requirements (a minimum score of 7.00) on every job. We have just made more visible for you what we are looking at."
  
  I also asked that this "consistency score" can become significantly bad with just one mistake. Their answer was like this:
  "This actually posed a problem in the past, where a translator could slack off and deliver poor translations during a certain period of time, and then have it fixed "easily" by getting 10 consecutive good reviews. This consistency score shows the overall performance more accurately, considering the entire checked job history, so now all translators know they need to stay sharp, always. "

  As I mentioned, I can understand what they wish to do. But in practice, I believe that this "consistency score" poses a problem to us all.

  That is because, as far as I have researched in "performance dashboard", it seems practically impossible to get "consistency score" back to above 7.0, after you had some bad score (below 6.0 or 5.0?) even once.
  While I was reviewing "performance dashboard" before the above-mentioned change was occurred, I could check history of GoChekc of each translator.
  I found one translator who gained above 9.0 for last 10 consecutive GoChecks, but below 7.0 for "consistency score".
  I found one translator who received 10 GoChecks in the past, with 9 of them are marked as "10", but one of them was somewhere around 3.5. As a result, his/her consisntency score was below 7.0.

  Yes, support team said that is the purpose of "consistency score", in order to make us (translator) "stay sharp, always". But that does not have to mean that we should be marked as "inappropriate translator" forever, after making one big mistake.
  I understand Gengo's policy as "if your scorecard/consistency score is below 7.0, you will be inspected by Gengo's translator team, and unless you have any special reason, you are likely to be revoked of your translator qualification".
  If someone gain below 7.0 for consistency score, and if it is really hard (or practically impossible) to get that score back to above 7.0, will that person have to fear of losing his/her qualification, as long as he/she works with Gengo?

  That is my concern. I asked this question to Gengo's support team, but have not received any reply yet.
  Hopefully I can receive an answer soon, and find out that I did not have to worry about these after all.

• 0
  
  swing 2018年10月19日 04:18

  You might always have to worry about it. That is the reality of this world....unfortunately..

• 0
  
  Lara Fernandez 2018年10月19日 19:51

  Hi all!

  Thanks for this discussion.

  I'm jumping in just to clarify a couple of things :)

  You should have received a follow-up email today informing you that we've temporarily disabled the performance dashboard while we implement some changes and improvements according to the feedback we have received over the last couple of days. We will inform you once the dashboard is available again. For the time being, I believe the email you received should be displaying your data (both average score in your scorecard, and the consistency score mentioned above.)

  One issue I would like to address is the OP's mention of Gengo introducing the consistency score as a change without prior notice. We'd like you to understand that, at the moment, this is not a change, per se, but an addition. We're still looking at your average score as displayed on your translator dashboard to measure your quality. However, consistency is something that we take seriously, highly value, and have been looking at for a while now. The fact that we expect translators to meet our quality requirements in every job is not new, and the consistency score makes it easier for you to see where you stand in this regard. The consistency score has been introduced based on the need to ensure that our translators are providing good quality in a consistent and reliable manner. In other words, this score shows how reliable the quality displayed on your scorecard is. Please also note that, while the average quality score currently shown on your scorecard is calculated based on your last 10 jobs only, the consistency score looks at a broader history of your jobs.

  It's also important to note that the Performance Dashboard does not replace your translator dashboard. We specifically sent this special dashboard to a number of selected translators in language pairs that have recently undergone large scale quality audits as a way for them to continue to track their performance after said audit. 

  Hope this helps!
  Lara

  Lara Fernandezにより編集されました 2018年10月19日 19:54
• 0
  
  亮 2018年10月20日 03:22

  To Lala,

  Thank you for your kind and detailed comment about this topic.

  Howevever, please kindly let me know of one thing.
  You kindly told us that: "We're still looking at your average score as displayed on your translator dashboard to measure your quality."

  Does it mean that "consistency score" will not be used, at least not directly, to revoke our qualification as Gengo's translator?
  As I wrote in my previous comment, what I worry most is that "If someone gain below 7.0 for consistency score, and if it is really hard (or practically impossible) to get that score back to above 7.0, will that person have to fear of losing his/her qualification, as long as he/she works with Gengo?"

  Hopefully I can hear from you soon.
  Best regards

• 0
  
  Lara Fernandez 2018年10月20日 14:58

  Hi 亮, 

  Does it mean that "consistency score" will not be used, at least not directly, to revoke our qualification as Gengo's translator?

  The answer to this question is that at the moment we're looking at both your quality score and your consistency score.

  I believe that Antonio, from Support, already explained this to you very clearly (please allow me to quote):

  "We were already expecting from all translators to meet our quality requirements (a minimum score of 7.00) on every job. We have just made more visible for you what we are looking at."

  So indeed, if a person's consistency score is below 7.0, that's an indicator that the person is not meeting the minimum score on every job, and therefore they're not actually meeting our quality requirements. I am unsure as to how much of a translator's GoCheck history the Performance Dashboard was showing, but I suspect that it was showing only the most recent checks for each translator (I might be wrong) and not all the checks that are taken into account to calculate the consistency score. Someone who rarely falls below 7.0 in their GoCheck score would have a much higher consistency score than someone who often fluctuates up and down with every review.

  I'm quoting below one of your comments in your posts above as it's inaccurate as well:

  I found one translator who gained above 9.0 for last 10 consecutive GoChecks, but below 7.0 for "consistency score".
  I found one translator who received 10 GoChecks in the past, with 9 of them are marked as "10", but one of them was somewhere around 3.5. As a result, his/her consisntency score was below 7.0.

  This comment above is incorrect and inaccurate because we look at a broader history of a translator's work with us to calculate the Consistency Score, and not only at their last 10 jobs (as is the case with the Quality Score) as you did with these examples.

  Long story short: if you aim to always consistently score 7.0 and above in your GoChecks, you don't have anything to worry about. For someone with a stable record who never falls below 7.0, a lower score would be an anomaly. While it would affect their consistency score, it wouldn't hurt it as much as in the case of someone whose work fluctuates between lower and higher scores every time they receive a GoCheck, therefore lacking consistency.
  
  Hope this makes sense to you!
  Lara

• 1
  
  亮 2018年10月20日 17:17

  To Lara,

  Thank you for your comment. However, you misunderstand me. What you wrote in your previous commet was wrong.
  I clearly understand that consistency score is NOT calculated only with last 10 GoChecks.

  What I wish to show you with those examples was that "even someone gain good Quality Scores in last 10 GoChecks, his/her consistency score may not be so good, because his/her bad score in the distant past is still damaging his/her consistency score".

  In fact, while I was researching the Performance Dashboard, I checked scores of 60 translators, and among them, 15 translators receives good Quality Score (above 7.) and bad consistency score (below 7.0).
  All these people gain good Quality Scores recently, but some bad score in their past damage his/her consistency score quite badly.

  What I read from your previous comment is that "if you gain good Quality Score consistently, you can gain good consistency score".
  However, what I wish to know is "what can we do, if we ever fail to do so". Really, what can we do?

  At least, please kindly answer to my following question. This is perhaps the thing I wish to know most right now.

  1. Is there anyway to get my consistency score back to above 7.0, after it was fallen below 7.0? With samples I found in the Performance Dashboard, and with explanation given from you and Antonio, it seems practically impoosible to do so. But is it?

  Best regards,

• 0
  
  Lara Fernandez 2018年10月20日 17:31

  Hi 亮,

  I'm not misunderstanding you. What I'm trying to tell you is that you're looking at a limited amount of data, and therefore your conclusions regarding other translator's consistency scores are highly inaccurate.

  You say "What I wish to show you with those examples was that "even someone gain good Quality Scores in last 10 GoChecks, his/her consistency score may not be so good, because his/her bad score in the distant past is still damaging his/her consistency score." but the fact is that you have no idea how many actual bad scores they have been getting in that distant (or not so distant) past. If their quality is consistent, one single bad score "in the distant past" would not easily bring it below average. What you are not seeing here is that these translators may have had several quality fluctuations in the past that result in their current consistency score. I would strongly advise you against focusing on other translators' scores in this particular case, as you don't know their past scores or their work history with Gengo, and therefore you can't reach an accurate conclusion in that regard. It would be more beneficial to you to focus on your own work history and first-hand experience.

  1. Is there anyway to get my consistency score back to above 7.0, after it was fallen below 7.0? With samples I found in the Performance Dashboard, and with explanation given from you and Antonio, it seems practically impoosible to do so. But is it?

  Once again, the "samples" you found are inaccurate, simply because you don't have enough data regarding other translators' history to get an overview of what kind of fluctuations led them to their current consistency score. In the same way, getting your consistency score back to above 7.0 if it was to fall below it would depend on the gravity of your score fluctuations -- how low your GoCheck scores fall, and also how often they fluctuate, so I can't make a blanket statement about this. The best way to maintain a good translator status with us is to always strive to provide the best quality possible -- checking the Style Guide in your language pair regularly if necessary, using our Resources page, and taking feedback from our LS as an opportunity to learn and improve. 

  Thanks!
  Lara

• 1
  
  亮 2018年10月21日 01:05

  To Lara,

  Thank you for your comment, and I feel very sorry that my comments have upset you.
  That is certainly not my intention and, I have to apologize to you for that.

  What you said was true. I was talking with "limited amount of data", "limited amount of fact" and "limited amount of information".
  That is all because I simply wish to know more about this new "consistency score", since I was only given with "limited amount of information".
  Unlike previous GoCheck system, even right now, I cannot find any information about this "consistency score" in your support page, twitter and anywhere.
  My current conclusion might be, as you said, "inaccurate" but I wish to be accurate by obtaining more information which may be given from you and other support team.

  However, I have to say that I used wrong way of proceeding this discussion. Deducing from "limited amount of data" is certainly not very good idea while making it. I shold have remembered that.

  Once again, thank you for all these comment and help you gave to me. To be honest, I wish to know and ask more about this consistency score. But I believe, for now, I should wait.
  And again, I feel very sorry that I have upset you.

  Best regards,

• 0
  
  Lara Fernandez 2018年10月21日 08:31

  Hi 亮,

  No worries at all! I’m not upset — I was just explaining to you how the Consistency Score works and why the conclusions you shared on the forums were not accurate. After all, you need to remember that this is a public forum for all Gengo translators, and information like the one you shared may be taken the wrong way by other people, especially those who have not received the link to the Performance Dashboard (because as I explained, this has been shared with only a select number of translators for the time being.)

  Have a lovely rest of the weekend :)
  Lara

• 4
  
  Jon 2018年10月23日 08:56

  Hi Everyone!

  
  I’ve already pretty much said my peace to Gengo Support regarding this, but I figured I’d post it in the forum too, in case there were others in a similar spot, or maybe just to vent a bit more, likely both. I don’t want to call out Gengo for anything here or try to pass off the blame, all I hope to do is illustrate that there are a vast number of translators on here who can all be affected by this is varying ways.

  
  I promise I will try to keep it as light-hearted as possible!

  I believe I may be a victim of this “Consistency Score”.

  Tonight I received a message informing me that my qualifications are in jeopardy with Gengo due to new quality standards in my language pair (Jp-En).

  I was a bit shocked. My score’s at 8.2, even though that is the lowest it has ever been during my time here, I guess there is no point in being embarrassed about sharing it since scores are now visible to everyone anyways.

  I am not proud of my 8.2. I actively avoid going to the screen that shows it, because it shames me. I kept a 10 for the longest time when I started out, and I find anything else unsettling.

  
  I vividly remember the first poor review I got. It was in January, four or five months after I’d started working with Gengo, on a one or two sentence fragment of a larger collection. It was 2.5. I was not pleased. My second came sometime in the summer, June or July I think. A customer had sent back a job they did not like, a first for me. I got a 1.3. The job had been a bizarre mixture of English, Japanese, and emojis, with little definable sentence structure. I shouldn’t have taken it, but at the time I thought it looked fun. I considered disputing the review due to the nature of the source text, but I figure if the customer didn’t like it, it is a failure on my part, regardless of if it gets reviewed or not.

  
  Now, I’m semi-active on the site, I’ve got other commitments, but I’d say I average about $200 a week on here. So, two unacceptable reviews should be easily overcome, especially nearly 11 months after the first one.

  
  My issue is, between my time zone and other commitments, it’s difficult to grab some of the smaller jobs, so I’ve always tended to aim for the bigger collections I can do over a few days when I find the time.

  
  Unfortunately, these jobs are almost always sent as files, and not on the workbench. So they do not get reviewed. This makes any single poor review difficult to recover from. Of course, these large jobs can be sent back for review if the customer is not satisfied (emoji job!), so you can’t skimp on quality. However, jobs accepted by the customer, jobs with good customer feedback, and “Preferred Translator” designations etc. are in no way considered when scoring you.

  So you can go down (hard), but not climb back up. It is very difficult to accept your work being called inconsistent due to two reviews when over that same time period you have completed several thousands of dollars of work for Gengo.

  
  At this point, I’m busier with life than I was when I joined Gengo last year, so I haven’t been able to take quite as many jobs as I once did. But I do have a few “Preferred Translator” customers who provide rather consistent work, and who are willing to work around my schedule. I like working with these customers.

  
  So my dilemma is, now that I know my future employment here hangs by a thread, and that one poor review will do me in, what am I to do?

  
  At this point probably 90% of my Gengo income comes from regular customers.

  
  Now, do I risk it? I hate the 8.2., but regardless of the amount of confidence I have in my work, I see very little reason for me to go out of my way to take some $6 job off the site, and run the risk, however low, of losing my qualifications and regular customers. The risk/reward just isn’t worth it.

  
  Now the obvious rebuttal is that, if my work was consistent, I’d have no problem doing a few jobs to improve my score.

  
  I’ll counter this by having them look at my list of completed jobs. With the steady stream of hearts representing satisfied, repeat customers. I’d say that’s rather consistent.

  Ok, I doubt anyone actually read this far, but I’m done.

  It's 4:30 AM and I’ve been doing Gengo stuff for the past 15 hours. Of course, only have of that was working, and the other half was complaining about being underappreciated to, myself.

  
  Lara, I know you’ll have to read this horribly long, rambling, post. I'm sorry! Thanks for the tote bag! Got it last week!

• 3
  
  Antje 2018年10月23日 10:58

  I have just had one of my language pair qualifications revoked because of just 1 (one!) bad review that I got 6 months ago.

  I don't know what exactly went wrong back then, but as I was just starting out working with Gengo (after registering years ago and then not being allowed do the tests) and the score was a ridiculously low 1.1 (according to the email listing "recent" reviews that I got a few days ago) and I certainly never handed in work that was that bad, so I can only assume that this job had a technical error of some sort - maybe a file didn't save properly or I messed up the html code or somthing like that.

  Other than that one outlier, I had never gotten a review below 7.0 and in fact only one other of my reviews in that language pair was ever below 8.0 (with more than half of them between 9.0 and 10.0), which is why my scorecard quality score was still at 8.1 despite the one very low outlier still being counted towards the average when the qualification got revoked.

  The feedback I got from customers also almost always was 5/5, excepting one mediocre 3/5 score from a completely unreasonable customer who didn't provide a well-written or at least grammatically correct orginal text and then didn't even react to comments or ask for revisions or tell me what I did wrong in the feedback. (I think they just got annoyed that I, trying to be helpful, politely pointed out to them that their original English text had too many mistakes and weird sentence structure to be readily understood by their prospective English-language website users and that they should better get a proper translation from their own language into English before comissioning translations from English into other languages.)

  Because I got a vaguely worded warning about my impending revocation a few weeks ago per email, I did try to improve my scorecard score (not realizing that the problem was just the 1 outlier and not the average score), but then I barely ever got reviewed on the about half dozen jobs I managed to do in this language pair (there wasn't much offered) over the time since I got the warning email. For one week right after the warning email, it was as if all the reviewers had gone on vacation simultaneously, as I didn't get reviewed in the reverse, much more active language pair either. This lack of job opportunities and reviews, and bad internet connectivity issues keeping me from working at Gengo over 3 months in the summer, is why I have only 3 recent reviews in the language pair in question - though those were 9.2, 10.0 and 8.3, so I did my best to prove that I can do better, as the warning email urged me to do.

  How am I supposed to improve my score if you won't review most of the jobs, even for a relative beginner? Also, as the OP explains, it wouldn't have mattered, as the consistency score is apparently not really improvable, no matter how many good scores you have.

  And as my revocation shows, this assurance in Lara's comment above:

  "For someone with a stable record who never falls below 7.0, a lower score would be an anomaly. While it would affect their consistency score, it wouldn't hurt it as much as in the case of someone whose work fluctuates between lower and higher scores every time they receive a GoCheck, therefore lacking consistency."

  is simply not true.

  They WILL kick you out if you make only one serious mistake, ever!

  (Or if you ever have just one extremely unreasonably customer as the translator above me describes. So, the message from Gengo is: Don't try to do the frequently apprearing jobs from Asian customers who submit those barely comprehensible English texts to be translated into other Western languages and who can't or won't communicate in English to clear up misunderstandings. Don't try to be helpful and do your best to turn those texts into something understandable by making guesses and judgment calls. Just ignore those jobs. Otherwise you will be fired sooner or later.)

   

  And of course I can't requalify because Gengo has blocked the tests in the language pair they've kicked me out of.

   

   

  P.S. Do I understand the previous commenter right? Do we automatically get a worse review score if the customer requests a revision of the job? Because I've been telling my customers that I'll be happy to discuss changes and to alter my translation accordingly if they send it back for revision, especially if the customer did not manage to answer clarification questions in time before I had to hand the translation in, as is usually the case for jobs done in the workbench or even the shorter Word / Excel file translations. I do this as an extra (unpaid) service to help the customers get the best possible results from Gengo, so they will use the service again. But if Gengo punishes this use of the revision system, then I will stop trying to be helpful in the future.

• 5
  
  Antje 2018年10月23日 11:15

  The translator support information page about job reviews (https://support.gengo.com/hc/en-us/articles/231441287) still states:

   

  "Please keep in mind that we will not remove your qualifications if you only have 1 bad score.

  Qualifications are removed if you consistently fail to meet Gengo's quality standards."

   

  Yeah, right...

   

   

   

  Antjeにより編集されました 2018年10月23日 11:19
• 0
  
  Jon 2018年10月23日 11:59

  Hi Antje,

  If you're referring to my comment regarding the revision system. No, not at all.

  The job I had sent back was sent directly to Gengo. The customer, unfortunately, made no attempt to contact me. I had no idea anything was wrong until the score drop.

  Using the revision system shouldn't affect your score in any way. I've also always made a point of telling customers I'm happy to change things or look over things again. I have had some of them send jobs back to me, and I've addressed the issues and sent them back. Gengo won't punish you for this.

  It's actually one of the best things you can do. I wouldn't have the repeat customers I'm afraid of losing if it wasn't for use of the revision system. 

  It's perhaps the best way to actually forge a relationship with a customer. 

  Sorry if my post was confusing. It was quite late. Time for me to go drink a pint of espresso. 

   

   

• 3
  
  亮 2018年10月23日 14:10

  To All

  This is the OP.

  First of all, I feel very sorry that jwcpetty and Antje have received warning, or even worse, been revoked of your translator's qualification.

  Regarding this topic, I found some interesting material today.

  Have you read the topic named "New standards for quality of jobs" in this Gengo's community forum?

  It was set up almost month ago by the translator named Xavier. According to his comment, he received an e-mail from Gengo, which content was, as he stated:

  "[Gengo was] saying that they were "doubling up [our] quality efforts" in my language pair (EN>FR), and I assume other pairs as well. Doubling up later translates as "we are now doubling up on our efforts to take action when translators do not meet this requirement" (aka getting scores above 7). Until that point I didn't have much of an issue with this statement. But then it says "any translator who receives two scores below 7 will have their qualifications removed", and I am starting to panic a little."

  It was, from my point of view, interesting because what Gengo was explaining in the email sent to Xavier was very much like what they explain about "consistency score". In that forum, he was also showing an answer he received from "community manager":

  "There is no specific timeline for the measures you mention, meaning that a translator who receives two scores under 7 has high probabilities of being demoted on the spot. However, in cases like those you mention (when those scores happen months apart), this is unlikely to apply. The keyword here, and what we are really looking for is "consistency". So we are looking closely at score fluctuations. Think for example of someone who receives a 10 today, a 5 tomorrow, a 7 the next day, and a 3 next... obviously this person's quality is pretty inconsistent and varies from GoCheck to GoCheck - these are the kind of people who are most likely to get demoted after two scores below 7. On the other hand, if you consistently score above 7, but you receive two lower scores, months apart, when we look at the fluctuation we would see that those are isolated cases and your score is actually pretty consistent."

  It is getting more and more like "consistency score". Only difference is that there was no mentioning about the term "consistency score" in his comment. I wonder when they come up with the idea...

  There is also one more thing I wish to comment about this topic: I have not received any such e-mail received by Xavier and other translators. Maybe these e-mails were only sent to EN>FR translator, but who knows.
  
  
  Anyway, what mentioned in Xavier's comment was that two lower score will make you revoked of your qualification.

  On the other hand, Antje was saying to have received only one lower score and revoked of qualification.

  My conclusion: Gengo is currently developing new evaluation system and things are changing quite rapidly, perhaps even now. That is the reason why they cannot show their new evaluation method on their support page.
  
  
  In my previous comments, what I have been asking was this: please give us more information about this new evaluation method.

  It seems like I need to do this once again. If no new information can be given, at least, I wish to know more inforamtion about their schedule.

  (Ex. When can we find more information in our support page?)

  The problem is, I have not received an answer to my previous request from Gengo support (regarding this topic) for quite a long time by now. Maybe that they have been discussing about what to do with the matter, but again, who knows?

  Anyway, hopefully this new information can be any help to you all!

• 0
  
  Lara Fernandez 2018年10月23日 18:14

  @jwcpetty: I'm glad you liked the totebag! And no worries about long posts. I'm here to read you all :) I do understand your frustration and the fact that you feel that taking more jobs is a risk. However, one thing that I would like to remind you and everyone following this thread is that, even if we weren't actively sending warnings before, taking a job should always demand the same amount of responsibility from a translator. We are all human, and sometimes we make a mistake and we pick up jobs that we don't completely understand (such as the emoji one you mentioned), maybe because there's nothing else available, or maybe because we find it challenging, but the fact remains that, ideally, that shouldn't happen, and you should only be picking up jobs that you are confident you can translate with good quality. 

  @Antje: I had a look at your language pairs and found that, as you stated, you had only 1 extremely low score in the language pair where you had your qualifications revoked. I did also notice that your scores fluctuated significantly more in that language pair than in the one you still hold qualifications for. Please allow me to double check with our Quality Team on this and get back to you.

  Also, no, you will not get a bad score on a review just because a customer complained about a job. What actually happens is that, if you receive a low scoring customer feedback or if a job of yours is rejected, the job automatically triggers a GoCheck review, which is conducted objectively by an LS. 

  @亮: What Xavier wrote on that thread almost a month ago regarding the two low scores rule, applied to his language pair, almost a month ago. Other translators have not received that email because it didn't apply to them. Translators are being sent the emails that apply to their specific situation.

  Thank you all for your understanding!

• 8
  
  masanpra 2018年10月25日 16:21

  Sometimes Language Specialists commit big mistakes. How does gengo control the quality of the internal reviews? How many "major correction mistakes" are they allowed to make before being warned by the Quality Control Team? If a Language Specialist were consistently corrected by a second Language Specialist, would gengo take action? Should translators report major review mistakes to the Quality Control Team besides asking for a re-review?

• 1
  
  Bobby Knuckles 2018年11月01日 13:51

  Anyone have an answer for @masanpra? Reviewers can blatantly refuse admitting to a mistake, and you don't get a re-review. It seems like there's nothing you can do but accept your faith (and low score, and losing your qualification eventually). 

• 0
  
  Xavier 2018年11月03日 08:16

  Hi guys! I'm happy to see other people questioning Gengo's latest "improvement" of policy.

  There have been some developments since the post mentioned by 亮.

  Like Antje, my professional qualification was revoked after one low score. What happened was: one customer wasn't satisfied and gave me a 1/5. That low score triggered automatically a Go-check, which was below 7. That job was an AI post-edition, and I failed to correct a mistranslation, so the score was fair, no complaint there. However, because of the low score from the customer and from the Go-check, all my recent work was reviewed. I then received an email quoting some minor mistakes I made across a few recent jobs (without receiving a score for each though, so for all I know these jobs were all still above 7), and I lost my qualification, despite having an average of 8.8 currently (it was 9 at the time).

  I already had a long conversation with Lara, both on the forum and by email. My feedback was that it is unfair that two low scores from the customer and Go-check on one single job cause a global review, since the former already triggers the latter. But, of course, nothing can be done, and I just have to wait patiently until I can retake the test for pro jobs... 

• 1
  
  亮 2018年11月03日 08:40

  To Xavier,

  This is the OP. I feel very sorry for what had been happening to you.

  I have been asking questions to Gengo's support since posting my last comment. What they have been saying is that Gengo takes "qualifty fluctuation" as the important part of their evaluation of translator's qualification, and they also said that they actually have not changed "expectation" to translators (minimmum 7.0 for each job). But, for instance, have you noticed that they used to expect Pro level translators to score above 8.0?

  Their answer does not satisfy me, so I am still asking questions. To be more precise, I feel that their answer seems not to fully explan the situation of you (Xavier) and other translators who receive above 7.0 for scorecard, but receive warning or even worse, revoked of his/her qualification.
  Support team really does not give me clear explanation about it, and sometime they goes around and around the same answer again and again. But I will continue
  Hopefully I can get better answer from them soon.

• 1
  
  Chris 2018年11月04日 17:48

  @Xavier:

  I don't think a global review should be a problem. It seems like an appropriate way to see if a low score is an outlier or not. In fact it sounds much better than the scenario of getting your qualification revoked because of one bad score. The bad score might be the trigger, but the reason would be multiple translations. However, this global review should of course be done according to the exact same standards as usual quality reviews (it sounds like you feel that didn't happen in your case).

• 0
  
  Xavier 2018年11月14日 06:46

  There was a message posted by "Michael" this morning/last night that has since disappeared. Maybe he changed his mind or got contacted by the Support team/someone from Gengo.

  I had a question related to what he was saying though. It sounded like he lost his pro qualification a while ago, got it back in August, and then was under review again for a low score a few days ago, but this time he lost both his pro and his standard qualification? That seems surprising but I would like to be sure.

  Lara, can you confirm what happens in such a case? If someone gets their pro qualification back, and then lose it once more because of a low score, they would still be able to access at least the standard jobs, right? And they could retake the pro test after 30 days? It's not like we have only one chance of getting it back before having to start again from scratch?

  (and he mentioned another good thing I was afraid of, the tests are not always open when there are too many translators in that language pair. the pro ones usually are I guess, but if someone loses their standard qualification as well, they might not be able to retake the test anytime soon. shouldn't there be an exception for already active translators who lost their qualification?)

  Xavierにより編集されました 2018年11月14日 06:46
• 0
  
  Michael 2018年11月14日 09:31

  Yes, both my Pro and Standard qualifications were revoked. 

  As for my previous post, I edited it because there was a typo, but after I edited it, it was marked "pending approval," which just made me want to delete it for some reason. Plus I just didn't like how I sounded. I'd been trying to think about how to moot this issue for awhile, and it was hard to figure out how to say what I wanted to say without just rambling on and on.

  And I did originally post it hoping that someone who could do something about it would see it, but I decided to e-mail Support instead. It seems like they have someone doing another audit consisting only of my recent work. But I'm probably toast because I just got a 0.64/10.00 on a job for which the customer specifically requested speed at the expense of quality. (I just filled out a form contesting a few of the marks on those grounds, plus I just disagree with one of the things marked "major error," but even if they accept it, which they probably won't, a couple of the actual mistakes seem like they'd probably be difficult to overlook.)

  And I didn't even want to submit that job. I really wanted to drop it halfway through. But I also thought I basically had a serviceable translation by the end, and I didn't want to be the reason the customer didn't get what they asked for on time.

  So it looks like I'll probably just be waiting as planned.

• -1
  
  Lara Fernandez 2018年11月15日 09:00

  @Xavier - We never censor the forums, and in the event that we had to delete or make edits to a post (due to inappropriate content, etc) you will never see us remove the post in its entirety. This means that if for example we at Gengo had had to delete Michael's post, instead of removing it, we would have deleted its content and replaced it with something along the lines of [Post removed due to (reason) - (name of person who removed the post)]. You may have already seen this in certain other posts where I have removed customer names due to privacy reasons. It does look like Michael himself removed the post though, as I can't find it in the Pending Approval list anymore, either.

  If you ever need to go back to a post that has been deleted, though, you can probably still find the full post in the email notification you received when it was first posted, if you're subscribed to this thread.

  On to your questions:

  It sounded like he lost his pro qualification a while ago, got it back in August, and then was under review again for a low score a few days ago, but this time he lost both his pro and his standard qualification? That seems surprising but I would like to be sure.

  The removal of his Pro qualifications in August, and the removal of both his qualifications this time around are two separate events. His qualifications weren't removed due to one low score. The low score triggered an internal quality audit of a greater number of his jobs. In an event like this, we don't just go back and look at jobs that have been GoChecked, but we take a look at more jobs that have gone unchecked and take action based on our findings. I notified the Quality Team about Michael's post yesterday, and requested they double check his situation. Several more of his jobs have been checked, and unfortunately, the team's final decision is that his qualifications will stay revoked.

  Lara, can you confirm what happens in such a case? If someone gets their pro qualification back, and then lose it once more because of a low score, they would still be able to access at least the standard jobs, right? And they could retake the pro test after 30 days? It's not like we have only one chance of getting it back before having to start again from scratch?

  As you can infer from the above, the circumstances under which Michael lost his qualifications in August and now are completely different, so they're not to be compared. Whether you lose your Pro qualifications or both Standard and Pro, you're always welcome to take the test after 30 days, as long as they're available.

  @Michael - I'm sorry to hear about your experience. As I explained above while answering Xavier's questions, I did see your post, and I brought it up with our Quality Team. I just didn't post here at the time because I generally prefer to wait until I have a better insight on the issue. I hope that my answers to Xavier can give you a better understanding of the process through which your qualifications were removed.

  I also wanted to bring up one thing that you mentioned in your original post, please allow me to quote:

  "I received a 3.67/10.00 for that job, which seems fair aside from the fact that getting a GoCheck score below 7.0 is a mortal sin now."

  This is a very problematic statement for a variety of reasons, but I would like to highlight the fact that GoCheck scores are not only "for the translator" in the same way in which you'd think of a score for a test at school or college. These scores are our way to check the quality that you're delivering to the end customer, who is paying for your work and has received your translation. I think it's important that we all take a moment to reflect upon what these scores really mean in a work environment, and how much fluctuations affect any translator's reliability, both within Gengo, and also outside of Gengo, as consistency is not really a "Gengo-only" or "translation-only" expectation.

  Thanks,

  Lara

• 0
  
  Xavier 2018年11月15日 09:35

  Hi Lara!

  Thank you for answering quickly, I got it now!

  Sorry for the misunderstanding, I didn't mean to imply he had been censored, I meant that maybe he got an answer directly from Gengo and didn't feel the need to keep his comment on this thread :)

  Xavierにより編集されました 2018年11月15日 09:36
• 2
  
  亮 2018年11月30日 02:59

  To all,

  This is the OP. In case if anyone is reading this topic right now, I wish to tell you this: Gengo has changed its scoring&qualification process so drastically.

  https://support.gengo.com/hc/en-us/articles/231441287-How-Gengo-measures-quality

  As all of you might have seen the above-mentioned URL already, fron now on, the "consistency score" has become the standard way for reviewing Gengo's translator. My score has become significantly lower than before (but somehow manage to maintain my qualification). But in the very least, now that we all know how we are judged.

  Since the announcement of "consistency score", I have been confused by Gengo's attitiude. It was unkonwn to me whether they were using "average score" or "consistency score" for judgement, because they did not put any explanation about consistency score on their website, and instead told us about "consistency score" only with their e-mail. I have been requesting them to make their explanation on their website and...now that they do, and thier explanation is so detailed one.

  As a translator, I believe I should be satisifed with this. Personally, I still feel that their judgemetn criteria is too severe, and I am not sure how many translator can actually survivie as the Gengo's translator after this change in judgement criteria. But that is not my business.

  Well, good luck folks. Hopefully you can make it.

   

• 22
  
  Nelson Bras 2018年12月01日 13:39

  Dear Gengo,

   

  I usually don't have time to participate in the forums, but this is far too serious to be ignored.

   

  Let's start by saying that my score went from 9.8 (last week) to 8.8 (this week), after 975,000 words translated for Gengo and 90 GoChecks. Nevertheless, this is not bad, according to Gengo, right? I have to disagree!

  1 - The score is the first thing a client looks at when he checks our profiles. My 8.8 score will be immediately compared to the score of someone who has just started and displays a perfect 10, just because he/she didn't have the chance to undergo 90 GoChecks, as I did. On the other hand, this is not the kind of score I would proudly share on any media, because it does not reflect my real, present performance… it means nothing!

  2 - I am selling my current quality, which derives, among other things, from the know-how obtained translating almost a million words here. What the client and Gengo are buying is my quality of today, not my performance from 4 years ago. The fact that a certain translator can have the same "consistency" effect on his/her score if he/she was bad at the beginning and now is good, as another translator who was good at the beginning and now is bad… doesn't make any sense!

  3 - Since I am a very active translator and Gengo is not my only place of work, my time is very limited, so I have ignored most of the reviewing issues… assuming that, after 3 reviews, the problem would be solved (in terms of scorecard). However, not only this will no longer be true in the future (you should prepare yourselves to receive a lot more feedback on the reviews), but also, I am going to be "punished" according to the new "law" for "crimes" for which the consequences were very different in the past, according to the "old law". Should I have known that you would be introducing such a severe rule with retroactive effects in the future… of course, I would have acted differently, notably, in terms of "appealing to the Supreme Court"!

  Why am I using legal terms? Because, if this rule was something "official", it would be illegal in almost every jurisdiction worldwide. There is a good reason why the "Principle of non-retroactivity of the criminal law" exists: http://users.cecs.anu.edu.au/~James.Popple/publications/articles/retroactive/2.shtml ///  https://en.wikipedia.org/wiki/Ex_post_facto_law

  The "consistency rule" should enter into force at the date of its publication, with no retroactive effects.

  The "consistency rule" should only cover a certain reasonable number of the latest GoChecks, in order to ensure equal treatment for every translator (newcomer or otherwise) and to promote an incentive for them to get better and see their efforts reflected in the following period.

  The consistency score should never be made public - the clients will always tend to interpret it as if they were looking at the usual scores they can see everywhere in the market. This is bad for both Gengo and the translator.

  I rest my case.

   

  Best regards,

  Nelson Brás

  Nelson Brasにより編集されました 2018年12月01日 17:55
• 19
  
  Nelson Bras 2018年12月01日 14:01

  By the way, the whole "consistency" system is based on a GoCheck procedure that has never been perfect and, I dare to say, still isn't (you should GO and Check how many changes have been made to the senior translator's platform since the early days), along with the less than perfect intervention (they are only human, after all) from senior translators. However, the retroactivity considered under the "consistency" rule assumes that everything is perfect from the beginning.
  After 975,000 words translated for Gengo, of course, I have already faced a lot of the following situations:

  a) The senior translator was absolutely right;

  b) The senior translator was convinced that every grammarian in the world is wrong and should learn with Gengo's senior translators (e.g. https://ciberduvidas.iscte-iul.pt/consultorio/perguntas/ligacao-a-vs-ligacao-com/34828 - by the way, this was never corrected);

  c) The senior translator considered a minor mistake as a major mistake;

  d) The senior translator marked the same error twice;

  e) Only one segment with a misplaced tag was reviewed, out of a job where the total word count was several hundreds of words - note: the mistake would be as visible to me as it was to the senior translator if we were using the same platform!

  f) The senior translator considered style differences (subjective) as a mistake;

  g) The senior translator considered that a certain word (synonym) was more frequently used than the one I chose… in a language (Portuguese) so rich in synonyms that we pride ourselves (it is even an established good writing rule!) to avoid using the same word twice in a sentence! A mistake? Really?

  h) A senior translator considers something wrong, while another senior translator considers it right in a different job;

  i) A senior translator considers something as being an error, just because he/she lacks the knowledge in a certain field, which makes him/her trust in his/her common sense + Wikipedia, even in the presence of a scientific substantiation that he/she does not understand;

  j) A certain word can be found in Microsoft's glossary… and the next day the same word is not there anymore, making it impossible to prove why that specific term was chosen;

  k) An error is detected seconds after the job was submitted - as the jobs cannot be edited anymore, the mistake will be there, no matter what. I would suggest the introduction of a time limit (say… 10 minutes?) to make any corrections before delivering the job to the client (most of the jobs are finished before time, anyway).

  l) After requesting a correction to Gengo's support, along with the correction came an error made by the person who changed the text… my mistake… I should have checked the whole translation before telling the nice lady from Gengo's support that "everything is OK now. Thanks";

  m) A translation was submitted before time, by mistake;

  n) After facing certain types of issues (power failures, hurricanes, technical problems, either from my side or Gengo's side, etc.), with not much time left to deliver (say… seconds) I only had 2 options: submit the translation without proofreading… or lose several hours of dedication and the corresponding payment, by declining the collection and offering the work already done to a different translator, who would not have to make any special effort to earn 400/500 USD in 10 minutes.

  o) Etc.

  These are all issues that should be addressed by Gengo before adopting such a drastically retroactive measure (consistency score).
  Note: I understand that senior translators are also human beings. They make mistakes and have bad days, like everybody else. So, I don't blame them. What I mean is that these issues must be taken into consideration, to avoid building a new system on the sand, instead of solid rock. Gengo, however, seems not to share this perception, since the new system allocates a stronger weight to every decision taken so far, as well as future decisions, by the senior translators... as if they were rock-solid and error-free.

  Best regards,

  Nelson Brás

• 5
  
  Xavier 2018年12月02日 03:57

  Thank you for raising those points Nelson!

  I thought the consistency score was only based on the last 10 jobs, so I didn't think much of it. But after seeing my score drop from 9 to 6.7 suddenly, and reading your comment, I realized that that score is based on the "entire checked job history".

  As you said it, that seems rather unfair for old translators! We improve with experience and practice, the scores we got a year ago or two years ago shouldn't be taken into account anymore! And if we had known about this before, maybe many of us would have stayed away from jobs that were difficult, in a poor source language, with tight deadlines, etc. We are now penalized for something that, before, could only at worst give us a low score that would disappear after 10 more reviews... (and I am with you on the LS comment, they do a great job, but mistakes can happen. It is already hard enough to plea your case and get a re-review, the scores have too much power now).

  In my opinion, Gengo should reset all the translator scores before implementing this new harsh system :(

   

   

• 7
  
  Sara 2018年12月02日 08:33

  Nelson said: "Since I am a very active translator and Gengo is not my only place of work, my time is very limited, so I have ignored most of the reviewing issues… assuming that, after 3 reviews, the problem would be solved (in terms of scorecard). However, not only this will no longer be true in the future (you should prepare yourselves to receive a lot more feedback on the reviews), but also, I am going to be "punished" according to the new "law" for "crimes" for which the consequences were very different in the past, according to the "old law". Should I have known that you would be introducing such a severe rule with retroactive effects in the future… of course, I would have acted differently, notably, in terms of "appealing to the Supreme Court"!"

  I definitely agree with that. There were definitely reviews I didn't agree with, and while I appealed some, I was busy and my average was usually high enough that I figured my average would sort itself out after a couple more checks. If I had known consistency checking was going to be a factor, then I would have appealed every 'questionable' review.

  I appreciate that the job of the Gengo reviewers is not easy, as I have done reviews for 2 other companies. One thing I have always tried to do when faced with stylistic differences as a reviewer is to indicate in my feedback what I thought would be a better sounding option, but not mark it as an error if it were truly a matter of stylistic preference. Looking back, I feel that I was unfairly penalised at times as a Gengo translator for making a different stylistic choice than the reviewer, and now I am sorely regretting failing to appeal those reviews. I also had a least one low score due to a technical error. Once again, I just let the situation go because I thought my average would quickly improve itself. I regret that as well. I am really not a fan of the new system.

  Saraにより編集されました 2018年12月04日 12:00