15

Hi,

This is my first time posting on the forums as far as I can recall, it could have been on better omens I guess, but I guess I'm more of a "no news good news" type.

So, I must say that I woke up whith a severe (and figurative) hangover when I logged in to work today and discovered my new translator score. Now, I'll get into the details later and, while I'm really not happy about it, this here post is not about ranting as much as it is about laying some hopefully constructive remarks that I hope can be backed up by my peers, or even rebuted by my colleagues in which case I would at least have learned something today.

First of all, the elephant in the room: while I have never (ever) seen my translator score dislpayed in red during the 4+ years I've worked with Gengo, I now start up and see a dreadful 6.6 on my scoreboard. That's quite a shock to be honest, I checked the mail we received explaining how all the previous scores are now taken into account, which I can understand, even if it raises a lot of questions I'll get to later (still, quite a shock). I don't know if I'm an isolated case, ending up with the lowest rating I ever had after all my work is taken into account instead of just the last 10. Now, while my score was above 8 during the past weeks, I still found that the job supply was quite weak despite the fact that it was announced that 8s and over would get jobs otherwise innaccessible for lower score holders. Now that I'm where I'm at, I must say I feel quite uneasy at the perspective of even less jobs (or even withdrawal of my qualifications).

 

Now, one can easily respond to all this: "get better". It's true, and I will, because it won't be said that I'm above getting better at my job. However, and this is my main gripe right here and now: the scoring system is flawed –not its very existence, mind you, just the way it is/was up to now; and I'd like to have the opinion of both my fellow translators and Gengo on what I'm about to develop.

  • Fisrt off: as translators might get inconsistent with the quality of their work (guilty as charged, obviously) this is also true for the reviewers and their reviews. I've been more than once appalled by some reviews I got. Rarely because they saw flaws where I (still) didn't – althought it happened in a few rare cases, but mainly by the way it impacted the ratings in the end. I once had the displeasure of being at a variance on wordings with the reviewer (i.e: nothing so objectively penalisable as a typo or a grammatical mistakes), variance I would still debate to this day btw (having asked several colleagues for input on the matter afterwards), all of this resulting on a score of 3 or worse; not gonna lie, tough pills to swallow every time this occurs. Twice I tried to signal Gengo about this, knowing –and accepting and understanding, that there was no appeal on the matter, but I did it hoping that this could find an echo in the future or that I could get further explanations on why I could merely be wrong on the matter. Twice I got cookie cutter answers, "reviews are unalterable, please check the forum/FAQs if you want more details" (paraphrasing here). I reluctantly decided that the least worst course of action then was getting my head down, waiting for other jobs, getting other jobs done good and reviewed accordingly, hoping said reviews occur quickly enough to finally expunge the "dud score(s)" from my last 10 reviewed jobs and go back to a more, shall I say, "representative" scoreboard. As I said earlier, I always managed to maintain a score of at least 7 with all that, which leads me to my next point.
  • Next point being: if an average of a translator's "career" is obviously more representative of the quality of their work on paper than just their last 10 jobs, and thus is ultimately a good idea for Gengo, its customers and even its tranlsators imo, it presupposes that the rating and review system is –and more importantely: always was– absolutely bulletproof. And I'll say it: it's not. Not enough to take such a gamble anyways. As it stands, you place the burden of every single quirk or mood from a reviewer (and I say this with all the more confidence, having had the occasion of working on both sides of the fence, and for Gengo to boot) and the burden of every potential fluke on the shoulders of a translator, and permanently so. I remember seeing in Gengo communication something along the lines of "at Gengo, we understand that a translator can have a bad day/phase and offer them a chance to win their momentum back and prove that a string of bad results was only momentary". Well, two remarks on this: firstly, by this logic, you have to take into account that the reviewers cans suffer from the same ordeal (and in my opinion, you don't), secondly, with the new system, this is utterly thrown down the sink.
  • I wholeheartedly subscribe to the fact that a translator with a string of 7s is more reliable and sounder from a business standpoint than one with several 10s and a few 3s thrown in the mix, as exemplified by your "how it works" scoring page. But to hammer it on: you then have to make really sure that the 3s really are worth 3s. And by that I'm not necessarely saying "it should be 10 or 3", but rather: "3 obviously implies mistakes, but do those mistakes really account for such a plummet in ratings?". If not, you run the risk of having otherwise pretty decent translators working with a sword of Damocles above their heads every time they submit a translation, this is both unfair and detrimental to the business and the overall quality of the work. It's a lose/lose/lose situation for Gengo, the customers, and the translators. I get that Gengo is a business, I get that running it implies handling both carrot and stick with your contractors, but there's simply too much stick here, or rather, to be more precise: handling of said stick is carried on too haphazardly.
  • Finally, I find it inconsistent –if not incoherent, that Gengo should within two days announce a revamping of its reviewing system on one hand, which could imply a change of scoring scale (be it minute or of significance, I can't say as I didn't experience the new system first hand yet), and on the other hand announce (and enforce) on the morrow the taking into account of the whole results of the older system, effective immediately. Consider this solely for the sake of argument–as I obviously lack any data on the matter: if the new system proves to be more forgiving than the previous one, then the translators present up until this point are at a huge disadvantage compared to anyone who integrates the Gengo ecosystem afterwards, and are so on criteria that are not relevant with their inherent qualities (or lack thereof) as translators. The opposite could be quite true as well, if the new system is harsher then newcomers could be at a disadvantage independently of their talent. This creates unfairness on one hand, and could fail on a systemic level at doing what it's purposed to do on the other: evaluate the qualities of every single translator, compare them to those of their peers (which works insofar as their respective evaluations are on the same scale, if you catch my drift) and then decide who to dish what jobs to. To reiterate: if that fails, then everybody accross-the-board suffers from it.

 

On a unrelated note: I suggest that you get rid of any MT stuff like those I've seen appear lately. Don't get me wrong: I really commend Gengo for not going the usual route of "here's a translation that has been MT'd beforehand, you thus have less work to do, we thus pay you less". Nothing's further from the truth, and that's why I am grateful. But please allow me to explain how it's "so far from the truth":

EDIT: Sadly, you can scratch that "grateful" part. Because after examination with other translators, it appears that Gengo indeed uses the excuse of a job that has been machine translated beforehand to scratch 25% off of a translator's fee on Pro level works. Standard jobs fees, on the other hand, remain untouched. Now, before this addendum I was about to explain why this was preposterous, so:

You have two courses of action with a machine translation that you're supposed to "smooth around the edges" (wich is often an understatement, might I had): you either try to effectively "smoothen" the existing text, which often implies mental gymnastics of epic proportions which in turn a) ends up taking more time to complete than a translation from the ground up and b) considerably increases the risks of incoherences due to careless mistakes. Or, you just delete all the text and translate everything from the ground up which, as explained in "a)" above, takes less time anyway.

Both options effectively render Machine Translation moot as a tool. Don't get me wrong, MT has its uses, especially when no translator is around, but considering the business Gengo and its contractors are in, it's moot.

Now this is going to sound like an outlandish exaggeration, but I'll stand by it for argument's sake because that remains true in absolute terms: either Gengo is developping a "learning" MT tool that will eventually be capable of effectively (and efficiently) replacing translators (I'm not judging: if it ends up working properly it's fair game), in which case: forget I said anything and please do carry on. Or Gengo is not, and it is just a tool whose sole purpose is and will be to aid translators in their work, in which case I'll refer you to the two previous paragraphs and let you draw your own conclusions.

 

I think I've said everything I needed now. Thank you for bearing through this and as I said you're welcome to chime in or answer anything that transpired in this post, I would just humbly ask that yout do so by setting out arguments regardless of the level of agreement or disagreement that you would experience.

 

Everybody have a nice one.

41 comments

  • 1
    Avatar
    Chris

    A couple of thoughts regarding machine translation:

    It won't be terribly long until machine translation for certain kinds of jobs will reach a level that will require very little human intervention.
    Some of the jobs submitted to Gengo today will go away one way or another. 
    It does make sense for Gengo to try to keep those jobs and tap into that market as early as possible, but for translators it will be generally detrimental, especially in the long run (but again: one way or another).

    During this transition you can decide if you want to make money off of these kind of jobs by teaching "the machine" or not. Now that's nothing new, basically pretty much any translation you do on an online platform will eventually be used to feed translation memories and AI, but e. g. translating close to a MT suggestion will speed up that process.

    Questions I would ask are if these MTs are supplied by the customer (that might indeed be additional jobs) and if e. g. the Google Translate API feeds directly back into Google Translate. 

    At the end of the day translators are offering a service, but you can't be forced to take certain jobs.
    I would agree that MT translations currently need too much work to warrant any reward reduction in most cases and there are jobs that simply aren't suited for them (and won't be in the foreseeable future).
    And I would also appreciate if reduced rewards would be clearly indicated with numbers for each collection. I realize this might require changes for the current platform while a new interface might be in the works, but e. g. there are those red "+ x%" bonus indicators which maybe can be repurposed to show deductions as well (though I can understand why one would want to avoid that).

    On the other hand I don't blame anyone for editing MTs either. I'd simply give it a try and see if it's worth my time and decide if that's really what I want to be doing.

     

     

  • 1
    Avatar
    Xavier

    Yeaaah, un autre français :p

    That project is indeed TM-enabled. They also provide machine translation. I let you imagine how useful that MT can be on the broken English often used by Amazon... 

    Since Gengo is now super strict with reviews and that those reviews impact which jobs I can access, I am not sure I will ever touch that project again. It is too easy for me to make mistakes, between the broken English and the weird product descriptions without pictures or so.

    I wonder how long it is going to take for Gengo to see jobs staying on the Dashboard for days because of the source language is of poor quality and the reward is not worth the risk... The Sony project is already affected for example, I see some jobs not being taken by anybody until Gengo adds 30% or so to the reward... 

    Edited by Xavier
  • 0
    Avatar
    Lara Fernandez

    @Bobby Knuckles - Your account will not be terminated on the basis that you used MT, as long as you consistently deliver good quality. That is to say, when a job that you have translated by editing machine translation is picked up for GoCheck review, it will be subject to the same quality standards as any other job. 

    @AlexF @juan.garcia.heredero @gamesofhands - Thanks for bringing that bug to my attention. I hadn't heard of it, so I will follow-up internally to ensure that our Engineering team is aware of it and can work on a fix.

    @draper - Thank you for bringing up the discrepancy between rewards/unit vs. total rewards in TM jobs. I agree that, for TM jobs, the reward/unit should be clearly displayed with the applied discount. I wasn't aware before today that this was happening on the dashboard, so I will bring this up internally as well.

    @Nelson Bras - At the moment there is no opt-out option to block out certain jobs from showing on your dashboard. This is an interesting request, though, and I will make sure to take note of it and share with the team. 

  • 0
    Avatar
    Lara Fernandez

    @Antoine,

    Thank you for your kind words, I truly appreciate them.

    There's a reason why the PEMT feature with a reward discount has been implemented only in 6 language pairs so far. This is because we are only willing to implement it in those cases in which we have actually obtained results that prove an increased efficiency. This is not to say that MT is always perfect, and that a translator will never encounter a poorly translated MT target text. Translators coming across bad MT are encouraged to actually flag and report it, so that we can investigate and improve.

    Of course, we are aware that MT is not always 100% perfect (and hence it needs human editing), and we are also aware that it does better with some language pairs than with others.

    We actually first implemented PEMT without a discount back in January. For months now, we've been analyzing, experimenting, studying, and measuring the results for the PEMT jobs that go through our system. What we found was that, for the language pairs where we've recently implemented a reward discount, there was a significant decrease in turn around time and no negative impact on quality scores, proving that PEMT was actually contributing to the efficiency of the translators working on those jobs. It is based on these findings that the decision to implement the discount was made.

    I hope that this explanation better answers your questions, and clarifies the process that led to the implementation of the new reward rates for these jobs.

    Last but not least, it's important to remember that PEMT is not applied to all jobs, and only to some of them. And, as always, that no translator is forced to work on a job that they're not interested in.

    Thanks,

    Lara

  • 0
    Avatar
    AlexF

    I would like to notify Gengo and other fellow translators working on the Amazon project of another annoying situation (bug?) : do the maths, all jobs are at a different reward per word, whereas they all have the heading 0.035$/word…

     

     

  • 0
    Avatar
    Lara Fernandez

    Hi AlexF,

    All those jobs have the "TM" icon next to the LP, so I'd assume they are TM-enabled.

    Please see this Support article for more details on TM and how it affects your rewards.

    Thanks,

    Lara

  • 0
    Avatar
    Lara Fernandez

    @Xavier — I think you’re mixing two different features here.
    TM is translation memory, which is what’s enabled for the jobs AlexF showed :)
    TM is not new and it’s been at least 2 years since we’ve implemented it.
    MT is machine translation.

  • 0
    Avatar
    Xavier

    Hm good point Lara. I know TM have been used for a while, and I don't mind them much (apart from the fact that we can't edit them and that some of them have mistakes, but that's a different debate).

    This project uses both TM and machine translations provided by the client (or Gengo? I'm not sure), and I was referring to the latter ones being useless. They might not be affecting the rewards though, I will edit my previous comment, thanks!

     (Screenshot removed due to including Client Material - Lara)

  • 0
    Avatar
    Lara Fernandez

    @Xavier - It doesn't look in your screenshot that MT has been prefilled into the target area (which is empty). It does look like the customer provided MT for reference in a comment, but please rest assured that when this is the case, said MT does not affect your rewards in any way. MT can only affect your rewards when it has been prefilled into the target field of the workbench.

    Also, please note that I will be removing your screenshot after posting this comment, to protect our customer's privacy as per Section 1 of our Translator Agreement, which requires all customer material to be treated as confidential. This would include not only customer ID numbers and such, but also their source texts. 

    Thanks!
    Lara

  • -6
    Avatar
    Lara Fernandez

    Hi all, 

    Thanks for this discussion.

    The implementation of the PEMT feature with a discounted reward rate was announced officially via email on October 26th to active translators in the applicable language pairs, namely EN<>ES, EN<>FR and EN<>IT.  

    The subject of said email was "Our PEMT feature is growing!" and the sender was Gengo Translator Team. Depending on your email settings, it's possible that it ended up in your Spam folder or that, if you have unsubscribed from email communications from Gengo (such as our newsletter), it couldn't be delivered. For the record, I'm pasting the content of the email below.

    Thanks,

    Lara

     

     

    Our PEMT feature is growing!

    Hi (name),

    We’re thrilled to be reaching out today to let you know that big changes are coming to your language pair.

    Our efforts to create more job opportunities are paying off, and as you have already read in other announcements and newsletters, in the past few months we’ve began to pave the way towards our long term goal of diversifying our offerings beyond translation, including AI-related projects, and other language related services, such as our Transcription and Edit services.

    Among these services, PEMT is one of the features that we implemented last January as a way to leverage technology to make the translation process more efficient while increasing consistency and quality. For a while now you may have been seeing and possibly have worked on PEMT (post-editing machine translation) collections in your language pair. These are jobs where the target section is pre-filled with a machine suggestion, which we expect you to use as a reference to make appropriate edits when necessary. If you have never worked on a PEMT job yet, or are not familiar with these types of jobs, please take a look at this guide to PEMT that we’ve put together with the help of our LSs.

    As demand for PEMT continues to grow, not only at Gengo but within the translation industry as a whole, we’re aiming to be able to bring on more new clients by extending the PEMT feature to a greater number of jobs ordered in your language pair. We expect that extending the PEMT feature to more collections will continue to help you make better use of your time, enabling you to access and pick up more work. Accordingly, we’re adjusting rewards for these jobs, which will now be paid at 75% of the rate per unit for non-PEMT jobs.

    As always, please don’t hesitate to reach out to Support with any questions or quality concerns. We are working on creating a feature that will make the process of flagging poor suggestions smoother. However, for the time being, we’d appreciate your help in letting us know if you experience any MT, PEMT or Edit-related issues.

    We continue to be hard at work to bring in more content and new customers – and now you can help too! A great way to do this is to follow us on our SNS: FacebooktwitterLinkedInInstagram, and YouTube, and to share our content with your family and friends, along with your brand new Translator Profile!

    Happy translating!
    —Gengo Translator Team
  • -6
    Avatar
    Lara Fernandez

    @Bobby Knuckles - As the email states, we are increasingly receiving demand for PEMT from customers, and this is not only Gengo but a common trend in the translation industry. Technically, you could say that yes, things have changed as the translation industry as a whole continues to evolve, and as Gengo needs to stay competitive to continue to be able to provide work for our translators. We originally implemented PEMT in a few selected language pairs (without a discounted reward) back in January (if your LP was affected, you should have received the relevant email) and upon analyzing the results of said implementation, proceeded to expand the feature as per our email of October 26th.

    Edited by Lara Fernandez
Please sign in to leave a comment.