Supporting minoritised languages through language technology: what's needed (for Turkic) and why bother?

The language vitality crisis

Languages everywhere are losing vitality

Linguistic loss of vitality

The "breakdown" of intergenerational transmission

• Sometimes due to:
• population displacement, genocide, etc.
• oppressive policies (e.g., educational)
• Happens even without explicity oppressive actions or policies
• But is always due to oppression and marginalisation

At the core

If there are disincentives to using a language, people will stop using it

Turkic languages by number of speakers

>200 million speakers total

remainder: ~31 languages

Why is loss of language vitality bad?

• For individual community members:
• Weakened connection to cultural knowledge
• Loss of cognitive, academic, emotional benefits of multilingualism
• Impact on families
• Impact on identity
• For the community and wider world:
• Loss of environmental, medical, and historical knowledge
• Loss of a whole way of understanding the world
• Loss of a linguistic (cognitive and social) system...
• Loss of diversity

How to counter loss of vitality?

• Fight oppressive policies and actions
• Support language maintenance and revitalisation efforts
• Easy, right?

What is language technology?

anything linguistic that involves interacting with [digital] technology

• input methods (keyboards, mobile keyboards, speech recognition)
• localisation (e.g., text displayed in a program)
• screen readers (text-to-speech)
• voice assistants
• machine translation
• search engines

Why is language technology important?

• Language mediates most interaction with digital technology
• Digital technology is hugely pervasive

What does language technology do for a language?

• add to ease with which a language can be used in digital contexts
• increases range of uses of language
• raises perceptions of language

What happens without language technology

"Digital language death" (i.e., loss of intergenerational transmission)

Mobile keyboards

Machine translation

Text-to-speech support

ISO 639 language codes

(baseline for any hope of language technology)

None currently for: Iraqi Türkman, Fu-yü Gyrgys, Dukhan

CLDR

(baseline localisation data used widely)

The short answer

We can't.

A slightly longer answer

By convincing each company that each language will bring in more money than they will spend to add it.

(except it probably won't, and they know that)

Some possibilities

• (Re)submit proposals for ISO codes
  (Dukhan rejected, Iraqi Türkman pending for 2 years)
• Contribute to CLDR and other open localisation efforts
• …
• Develop our own language technology?

Example:kaz

“Downstream tasks” (uses):

• Spell checkers
• Component of machine translation systems
• Computer-assisted language-learning (CALL)
• Script conversion
• Other NLP tasks and research
• Whatever else you can imagine

Symbolic models (compared to corpus-based)

• Advantages:
• No requirement for giant corpora
• No powerful computers required
• More precise than corpus-based
• Easy to fix errors and expand coverage
• (In MT:) lots of "free rides" for related languages
• Support for multiple orthographies ~trivial (Washington et al. 2020, Washington et al. 2021)
• (Essentially) single development cycle (Tyers et al. 2015, Butt 2020)
• Only need knowledge of language, not mathematics and programming
• Disadvantages:
• ...Need knowledge of language, not mathematics and programming
• Takes time to develop (human time, not computing time)
• i.e., good for community partnerships:
• Engagement between different types of experts, as equals
• Less prone to extractive & other marginalising approaches

Free/Open Source

What this means (ideally / in the case of Apertium):

• Available for anyone to use, modify, or build on
• Robust community, support from developers
• Able to easily compare to other systems
• Abandoned projects can be resumed or repurposed
• Standards / tools for wider integration/use (e.g., Apertium: website platform, integration into OmegaT & Wikipedia Content Translation tool)

→ i.e., maximally accessible to language communities

The real challenges

• Localisation: much software is not localisable independently of the producer of the software
• Closed interfaces: language-related programming interfaces are often closed to developers
• Closed resources: language resources are closed and not accessible to the public
• Disregarded standards: language-related international standards are not respected or fully implemented