Free/Open-Source technologies
for Turkic languages
developed in the Apertium project

Jonathan N. Washington¹ · Ilnar Salimzianov² · Francis M. Tyers³
Memduh Gökırmak⁴ · Sardana Ivanova⁵ · Oğuzhan Kuyrukçu⁶

  1. Swarthmore College, USA
  2. Kazan, Tatarstan
  3. Indiana University, USA; Высшая Школа Экономики, Москва
  4. Univerzita Karlova, Praha
  5. Helsingin yliopisto, Helsinki
  6. Boğaziçi Üniversitesi, İstanbul

Overview

We present a collection of Free/Open-Source
morphological transducers and machine translation systems
for Turkic languages.
  • Motivation for:
    • Turkic language technology
    • Symbolic approaches
    • Free/Open-Source Software
  • Morphological transducers for Turkic languages
  • Machine translation systems for Turkic languages
  • Future work:
    • Morphological transducers
    • Machine translation systems

Motivation: Turkic language technology

  • Official status ≠ future survival
  • Language technology necessary for a language to survive digital age
    • "only" ⅓ of 7000 languages are endangered, but only 5% have sufficient level of access to language technology (Kornai, 2013)
  • Situation of Turkic languages:
    • ⅓ of languages (13 of 39) officially considered endangered (Ethnologue, 2019)
    • Very little support for language technology
  • What can we do about the current situation?
    • "We live during a brief period of overlap between the mass extinction of the world’s languages and the advent of the digital age." — Bird (2009)

Motivation: Symbolic approaches

  • Symbolic vs. corpus-based (neural, statistical)
  • Symbolic approaches encode linguistic generalisations
  • Advantages (+):
    • No requirement for giant corpora
    • More precise than corpus-based
    • Easy to fix errors and expand coverage
    • Lots of "free rides" for related languages
    • Only need knowledge of language, not mathematics and programming
  • Disadvantages (-):
    • ...Need knowledge of language, not mathematics and programming
    • Takes time to develop
  • I.e., community-maintainable

Motivation: Free/Open-Source Software

Free = бесплатно + свободно
Open = открыто + доступно
  • Available for anyone to use or modify
  • Support from developers
  • Robust community (Apertium)
  • Able to easily compare to other systems
  • I.e., accessible to community

Morphological transducers: Uses

Example:
Input / Output:Алмасы
Output / Input: алмаnpx3spnom
алмасnpx3spnom
Алмаnpantfpx3spnom
Алмасnpantmpx3spnom
алvtvneggpr_futsubstpx3spnom
алvauxneggpr_futsubstpx3spnom
алvtvnegger_futpx3spnom
алvauxnegger_futpx3spnom

Main uses:
  • Spell checkers
  • Component of machine translation systems
  • Language-learning systems (e.g., Revita, revita.cs.helsinki.fi)
  • Whatever else you can imagine

Morphological transducers: Languages

Production-level
92%-98% coverage
(Tatar, Kazakh, Turkish, Kyrgyz, Crimean Tatar, Tuvan)
Working
88-93% coverage
(Bashqort, Chuvash, Uzbek, Qaraqalpaq, Uyghur, Sakha, Karachay-Balkar, Gagauz, Kumyk)
Prototype
<80% coverage
(Azerbaycani, Turkmen, Noghay, Khakas, Altay, Ottoman)

Morphological transducers: extra features

  • Multiple orthographies — Crimean Tatar, Qaraqalpaq
  • Spell checking — Word, LibreOffice for Kazakh, Kyrgyz, Tuvan, Tatar
  • Segmenter — Kazakh

Machine translation systems

  • Main uses:
    • Saving translators time: MT then post-editing
    • Assisting a speaker of one langage to understand material written in another language
    • Making production-ready legal documents, literature, marketing materials, etc.
  • Examples:
    • Crimean Tatar speaker wants to read Turkish news website
    • Amateur translator wants to quickly translate Turkish Wikipedia articles for Crimean Tatar Wikipedia

Machine translation pairs

Turkic-Turkic    
tatbak57510
chvtur31946
chvtat29512
uzbkaa19369
uigtur10194
kaztat9956
turaze8211
kazkir8177
turkir7824
kaztur7636
crhtur7140
kazkaa5425
turuzb4359
turtat4272
kazsah2890
kazuig2727
kazkum563
kiruzb323
kaztyv159
with non-Turkic languages
engkaz33002
kazrus29777
tateng14985
tatrus6036
engkir368
khkkaz140

Machine translation systems: How

  • Standard Apertium RBMT pipeline
  • Any element may be
    • augmented by statistical methods
    • replaced with corpus-based model (statistical, neural)

Machine translation systems: example

  • input: yahşı vaqıt edi
  • morphological analysis:
    ^yahşı/yahşıadj/yahşıadjsubstnom/yahşıadj+ecopaorp3sg/yahşıadjsubstnom+ecopaorp3sg$ ^vaqıt/vaqıtnnom/vaqıtnattr/vaqıtnnom+ecopaorp3sg$ ^edi/ecopifip3sg/evauxifip3sg$^./.sent$
  • disambiguation:
    ^yahşı/yahşıadj$ ^vaqıt/vaqıtnnom$ ^edi/ecopifip3sg$^./.sent$
  • lexical transfer:
    ^yahşıadj/iyiadj$ ^vaqıtnnom/vakitnnom/sürennom$ ^ecopifip3sg/icopifip3sg$^.sent/.sent$
  • lexical selection:
    • ^yahşıadj/iyiadj$ ^vaqıtnnom/vakitnnom$ ^ecopifip3sg/icopifip3sg$^.sent/.sent$
  • structural transfer:
    ^iyiadj$ ^vakitnnom+icopifip3sg$
  • morphological generation: iyi vakitti

Lexical selection: example

  • vaqıtcrh → vakittur time
  • bir vaqıtcrh → bir süretur (for) some time

Lexical transfer:

<e><p><l>vaqıt<s n="n"/></l><r>vakit<s n="n"/></r></p></e>
<e><p><l>vaqıt<s n="n"/></l><r>süre<s n="n"/></r></p></e>

Lexical selection:

<rule weight="1.2">
  <match lemma="bir"/>
  <match lemma="vaqıt" tags="n.*">
    <select lemma="süre" tags="n.*"/>
  </match>
</rule>

<rule weight="1.0">
  <match lemma="vaqıt" tags="n.*">
    <select lemma="vakit" tags="n.*"/>
  </match>
</rule>

Structural transfer: example

  • barğan soñcrh → gittikten sonratur after going
  • ^bar<v><iv><ger_past><nom>$ ^soñ<post>$crh →
    ^git<v><iv><ger_past><abl>$ ^sonra<post>$tur
<rule comment="REGLA: GAn soñ ">
  <pattern>
    <pattern-item n="ger_past"/>
    <pattern-item n="soñ"/>
  </pattern>
  <action>
    <call-macro n="f_strip_tags"><with-param pos="1"/></call-macro>
    <let>
      <clip pos="1" side="tl" part="a_cas"/>
      <lit-tag v="abl"/>
    </let>
    <out><chunk name="n" case="caseSecondWord">
      <tags><tag><lit-tag v="SP"/></tag></tags>
      <lu><clip pos="1" side="tl" part="whole"/></lu>
      <b/>
      <lu><clip pos="2" side="tl" part="whole"/></lu>
    </chunk></out>
  </action>
</rule>

Future work: transducers

  • Multiple script support, esp. Uyghur, Uzbek, Azerbaycani, Kazakh
  • Spell-checkers automatically generated for each language
  • More languages, e.g. Dolgan, Qashqai, Shor, Salar, etc.

Future work: translation systems

  • Revised Apertium RBMT pipeline
  • Additional optional elements:
    • Multiwords and separable expressions (2017)
    • Anaphora resolution (2019)
    • Recursive structural transfer (2019)

Sağ oluñız! / Сагъ олунъыз!

Special thanks to:

  • Hèctor Alòs i Font, Kantörö Erkulov, Gianluca Grossi, Sharapat Kalabaev, Mansur Saykhunov, Beknazar Abdikamalov, Akın Dalkı, Ağarahim Sultanmuradov, Tolgonay Kubatova, Remziye Berberova, Darya Kavitskaya, and Nick Howell
    (contributed to the development of these systems)
  • the Google Summer of Code program
  • the Google Code-In program
  • the Apertium open source community