Machine translation (MT) systems are now ubiquitous
This ubiquity is due to a combination of increased need for translation in today’s global marketplace. And an exponential growth in computing power that has made such systems viable. And under the right circumstances MT systems are a powerful tool. They offer low quality translations in situations where low quality translation is better than no translation at all. Or where a rough translation of a large document delivered in seconds or minutes is more useful than a good translation delivered in three weeks’ time.
Unfortunately despite the widespread accessibility of MT. It is clear that the purpose and limitations of such systems are frequently misunderstood. And their capability widely overestimated. In this Article Scroll i want to give a brief overview of how MT systems work and thus how they can be put to best use. Then I’ll present some data on how Internet based MT is being used right now. And show that there is a chasm between the intended and actual use of such systems and that users still need educating on how to use MT systems effectively.
How machine translation works
You might have expected that a computer translation program would use grammatical rules of the languages in question, combining them with some kind of in-memory “dictionary” to produce the resulting translation. And indeed, that’s essentially how some earlier systems worked. But most modern MT systems actually take a statistical approach that is quite “linguistically blind”. Essentially, the system is trained on a corpus of example translations. The result is a statistical model that incorporates information such as:
Given a huge body of such observations
The system can then translate a sentence by considering various candidate translations– made by stringing words together almost at random (in reality, via some ‘naive selection’ process)– and choosing the statistically most likely option.
On hearing this high-level description of how MT works, most people are surprised that such a “linguistically blind” approach works at all. What’s even more surprising is that it typically works better than rule-based systems.
This is partly because relying on grammatical analysis itself introduces errors into the equation (automated analysis is not completely accurate, and humans don’t always agree on how to analyse a sentence). And training a system on “bare text” allows you to base a system on far more data than would otherwise be possible: corpora of grammatically analysed texts are small and few and far between; pages of “bare text” are available in their trillions.
However, what this approach does mean
Is that the quality of translations is very dependent on. If you accidentally type he will returned or vous avez demander instead. Such as will returned are unlikely to have occurred many times in the training corpus or worse. May have occurred with a completely different meaning as in they needed his will returned to the solicitor. And since the system has little notion of grammar to work out for example that returned is a form of return and the infinitive is likely after he will it in effect has little to go on. Article Scroll
Similarly, you may ask the system to translate a sentence that is perfectly grammatical and common in everyday use but which includes features that happen not to have been common in the training corpus. Such as technical or business documents or transcripts of meetings of multilingual parliaments and conferences.
This gives MT systems a natural bias towards certain types of formal or technical text. The grammar of everyday speech such as using tú instead of usted in spanish or using the present tense instead of the future tense in various languages may not.
MT systems in practice
Researches and developers of computer translation systems have always been aware. That one of the biggest dangers is public misperception of their purpose and limitations. Somers (2003)[1], observing the use of MT on the web and in chat rooms, comments that: “This increased visibility of MT has had a number of side effets. […] Observing MT in use in 2009, there’s sadly little evidence that users’ awareness of these issues has improved.
As an illustration I’ll present a small sample of data from a Spanish-English MT service. That i make available at the EspañolInglés web site. The service works by taking the user’s input applying some cleanup processes such as correcting. Some common orthographical errors and decoding common instances of SMS speak and then looking for translations in a bank of examples from the site’s Spanish English dictionary.
And a MT engine. The figures I present here are from an analysis of 549 Spanish English queries presented to the system from machines in Mexico other words. We assume that most users are translating from their native language.
First what are people using the MT system for? For each query. I attempted a best guess at the user’s purpose for translating the query. In many case, the purpose is quite obvious in a few cases there is clearly ambiguity. With that caveat i judge that in about 88% of cases the intended use is fairly clear cut and categorise these uses as follows:
- Looking up a single word or term: 38%
- Translating a formal text: 23%
- Internet chat session: 18%
- Homework: 9%
A surprising if not alarming observation is that in such a large proportion of cases users are using the translator to look up a single word or term. In fact 30% of queries consisted of a single word. The finding is a little surprising given that the site in question also has a Spanish English dictionary. And suggests that users confuse the purpose of dictionaries and translators.
Perhaps as a consequence of student over drilling on dictionary usage. We see for example a query for cuarto para quarter to followed immediately by a query for a number. There is clearly a need to educate students and users in general on the difference between. The electronic dictionary and the machine translator in particular. That a dictionary will guide the user to choosing the appropriate translation given the context.
I estimate that in less than a quarter of cases users are using the MT system for its trained for purpose of translating or gisting a formal text and are entering an entire sentence or at least partial sentence rather than an isolated noun phrase.
The online chat context poses particular problems for MT systems. Since features such as non standard spelling lack of punctuation. And presence of colloquialisms not found in other written contexts are common.
It’s not too surprising that students are using MT systems to do their homework. But it’s interesting to note to what extent and how. In fact use for homework incudes a mixture of fair use understanding an exercise with an attempt. To get the computer to do their homework with predictably dire results in some cases. Queries categorised as homework include sentences which are obviously instructions to exercises. Plus certain sentences explaining trivial generalities that would be uncommon in a text or conversation. But which are typical in beginners homework exercises.
Whatever the use, an issue for system users and designers alike is the frequency of errors in the source text which are liable to hamper the translation. In fact, over 40% of queries contained such errors, with some queries containing several.
- Missing accents: 14% of queries
- Missing punctuation: 13%
- Other orthographical error: 8%
- Grammatically incomplete sentence: 8%
Bearing in mind that in the majority of cases users where translating from their native language. Users appear to underestimate the importance of using standard orthography to give the best chance of a good translation. More subtly users do not always understand that the translation of one word can depend on another and that the translator’s job is more difficult if grammatical constituents are incomplete. So that queries such as hoy es día de are not uncommon. Such queries hamper translation because the chance of a sentence in the training corpus with say a dangling preposition like this will be slim.
Lessons to be learnt…?
At present, there’s still a mismatch between the performance of MT systems and the expectations of users. I see responsibility for closing this gap as lying in the hands both of developers and of users and educators. Users need to think more about making their source sentences MT friendly and learn. How to assess the output of MT systems and developers including myself need to think about how we can make the tools we offer better suited to language users’ needs.
