#underrepresented languages

An article in BBC Future about linguistic representation on the internet. 

The Kaqchikel Mayan community from Guatemala includes more than half a million speakers. Miguel Ángel Oxlaj Kumez is part of it and was one of the organisers of the first Latin American Festival of Indigenous Languages on the Internet, held in 2019.

“When I get on the internet I find more than 90% of the content in English and hence a significant percentage in Spanish and other languages,” he says. “So what I have to do is to move to another language, and that favours the displacement of my own language.”

“It discredits my own language, because – as it is not on the internet – then it is not valid, then it does not work, therefore why am I going to continue learning it? Why am I going to teach it to my children if, when I turn on the internet or television, I cannot find it there?”

Oxlaj Kumez is working with other activists to create a version of Wikipedia in Kaqchikel Mayan, as well as a translated version of Mozilla’s Firefox web browser. His dream is to be able to have a “digital life in my own language, and when I decide to move to another language that it will be my decision”.

He is not the only one with that dream. In 2003, Unesco adopted a recommendation to promote the use of multilingualism online. Ever since, the organisation has been pushing for universality on the internet, with a special focus on indigenous languages.

Two recorded talks from typography conferences about increased cross-linguistic font support, from Zachary Scheuren, one of the people working on the Google Noto font which aims to support all the writing systems of the world. Description of the first talk, Invisible Letters, Invisible Languages: 

If we can’t see a language, does it exist? Of course it does, but in an increasingly computerized world, many languages remain invisible. Some languages have no writing system to be seen. Others are not necessarily accessible in the digital realm. What are the implications for these languages in light of today’s technologies? Computing in general is grounded in the assumption that spoken language can be transcribed and manipulated. But who are we leaving behind and why should we care? The benefits of digital representation aren’t limited to the locale of a particular language, script, or culture. The possibilities range from linguistic analysis to all forms of cross-referencing across disciplines like linguistics, history, archaeology, literature, sociology, economics, anthropology, and more. This presentation will look at a number of historic and minority writing systems and consider what can be gained by digitizing them and making them accessible to the world.

Watch on youtube. 

Here’s what the second talk (Writing Systems of the World) is about: 

There are hundreds of writing systems in the world, some no longer in use, some in use by small groups, and some (like the words you are reading now) in use by larger groups. Many writing systems have been encoded in various ways for use in computing, but encoding is just the first part of the process. Without fonts and proper support in applications a writing system might not be usable at all. The current state of technology allows us to do so many amazing things yet many writing systems are not adequately implemented for use in computing. How can it be that some scripts are so difficult to use? What is standing in the way? Why are shaping engines so complicated? (They aren’t). Why is dealing with text encodings so hard? (It shouldn’t be). Why are there little to no fonts for some writing systems? What can we do to move forward? 

 This presentation will provide an overview of world scripts and the common problems that keep some scripts from being used. We will then look at what needs to be done to remedy this and how everyone can help.

A post on the PanLex blog about the steps involved in creating digital language tools for under-supported languages, including the Zero to Digital guide that breaks down in detail exactly what steps are involved. Excerpt: 

Many of us who use a well-supported language online may not realize how many layers of technology underpin the implementation of language on our devices. When we buy a new phone or laptop, we barely notice these layers, because our language usually functions and displays seamlessly on the device without hiccup. But that’s not the experience for speakers of the under-resourced languages, even including some with millions of speakers. For example, Lahnda (Pakistan) with 93M speakers and Wu Chinese with 81M are not well supported.

Here are some of the foundational layers that must be implemented behind the scenes for a language to be digitally supported.

 - A writing system (such as Latin, Devanagari, or a newly created system) and orthography (spelling rules) must be identified, chosen, or developed for the language.  - Each letter or character of the writing system (like capital B, lowercase c, or !) must exist in Unicode, giving each one a standard numeric code point.  - A font that displays all letters and characters in the writing system must be developed.  - Using typography design software, font designers identify and create all necessary glyphs, or graphic shapes that represent the letters and characters of the writing system.  - The font designers write rules to handle any required ligatures and other complex cases in order to combine, stack, or connect the glyphs properly, in accordance with the writing system’s conventions. (See graphic below.) This step can be labor intensive for complex writing systems.  - The font must be available on a device at the moment the user needs it. Keyboards must be created so that text can be input in a language. These can be either on-screen keyboards or mappings from common physical keyboard layouts such as QWERTY.

An article in Slate about the role of Wikipedia in creating language tools. Excerpt: 

Although Alexa still does not speak or understand Welsh, the Celtic language’s presence in tech has increased dramatically within a short period. Google announced in February that it had expanded its offerings in Docs, Sheets, Slides, and Drive to include Welsh. And Google Translate—infamous since 2009 for its Scymraeg, or scummy Welsh—has, according to the BBC, recently taken a great leap forward in terms of the accuracy and quality of its Welsh translations. Morlais and others attribute this in part to the fact that there are now more than 100,000 articles on the Welsh version of Wikipedia, known as Wicipedia.

Like other language editions, Wicipedia is a separate website with its own content, not simply a translation of English Wikipedia, a distinction that matters for both users and big tech companies. Back in 2017, Morlais observed, “There appears to be an indication that there is a link between the languages with the most Wikipedia articles or pages and the languages that are supported by the digital giants.” Google Translate and other technologies use artificial neural networks to learn from example, training themselves with language data from rich internet sources like Welsh Wikipedia.

The Welsh community is not alone in using wiki-technology to promote its language. This year’s Celtic Knot conference in Cornwall, England, included several indigenous languages with their own Wikipedia editions. The original idea, as the name suggests, was to focus on Celtic languages, including Irish, Scots, Breton, Welsh, and Cornish, which was declared extinct merely a decade ago. But as word got out about a Wikipedia minority language conference, others began to join, representing, for example, the Sámi language spoken in parts of Norway, Finland, Sweden, and Russia; the Berber family of languages spoken in Northern Africa; and the Basque and Catalan communities. (In his 2017 presentation, Morlais noted that Catalan was one of the few minority languages supported by Google search, an accomplishment he linked to the fact that Catalan already had more than 500,000 articles on its language edition of Wikipedia.)

Masakhane is an interesting project to make machine translation more available in languages of Africa. From the description: 

We need African researchers from ACROSS the continent to join our effort in building translation models for African languages. Masakhane means "We Build Together" in isiZulu and was inspired by the Deep Learning Indaba theme for 2018.

Phase 1

We want to develop baseline machine translation models from English-to-Target African languages, with publicly available code and data. This problem is part data gathering, part developing the translation model, and part error analysis to understand what issues the models have

Phase 2

Once we, as a collaborative African NLP team, have trained baseline translation models for many of our languages, we combine our datasets and do transfer learning with fine-tuning across the languages.

Phase 3

Write & submit a paper, with all of our work, to a top-tier NLP conference and in doing so, once and for all put Africa on the NLP map

A paper from the Workshop on the Use of Computational Methods in the Study of Endangered Languages (ComputEL-3) about making better tech tools for linguistic annotation, featuring many ideas that I wish I’d had in grad school! 

Abstract: 

To reduce the annotation burden placed on linguistic fieldworkers, freeing up time for deeper linguistic analysis and descriptive work, the language documentation community has been working with machine learning researchers to investigate what role technology can play, with promising early results. This paper describes a number of potential follow-up technical projects that we believe would be worthwhile and straightforward to do. We provide examples of the annotation tasks for computer scientists; descriptions of the technological challenges involved and the estimated level of complexity; and pointers to relevant literature. We hope providing a clear overview of what the needs are and what annotation challenges exist will help facilitate the dialogue and collaboration between computer scientists and fieldwork linguists.