#panlex

A post on the PanLex blog about the steps involved in creating digital language tools for under-supported languages, including the Zero to Digital guide that breaks down in detail exactly what steps are involved. Excerpt: 

Many of us who use a well-supported language online may not realize how many layers of technology underpin the implementation of language on our devices. When we buy a new phone or laptop, we barely notice these layers, because our language usually functions and displays seamlessly on the device without hiccup. But that’s not the experience for speakers of the under-resourced languages, even including some with millions of speakers. For example, Lahnda (Pakistan) with 93M speakers and Wu Chinese with 81M are not well supported.

Here are some of the foundational layers that must be implemented behind the scenes for a language to be digitally supported.

 - A writing system (such as Latin, Devanagari, or a newly created system) and orthography (spelling rules) must be identified, chosen, or developed for the language.  - Each letter or character of the writing system (like capital B, lowercase c, or !) must exist in Unicode, giving each one a standard numeric code point.  - A font that displays all letters and characters in the writing system must be developed.  - Using typography design software, font designers identify and create all necessary glyphs, or graphic shapes that represent the letters and characters of the writing system.  - The font designers write rules to handle any required ligatures and other complex cases in order to combine, stack, or connect the glyphs properly, in accordance with the writing system’s conventions. (See graphic below.) This step can be labor intensive for complex writing systems.  - The font must be available on a device at the moment the user needs it. Keyboards must be created so that text can be input in a language. These can be either on-screen keyboards or mappings from common physical keyboard layouts such as QWERTY.

A blog post about an interesting digitization project for Balinese palm-leaf manuscripts (lontars) and the writing system on them. Excerpt: 

The Balinese language is still commonly spoken, but the ability to read and write literary works in the Balinese script has largely been lost. Since much of Bali’s culture and history is told in written manuscripts called lontars, the Internet Archive and the linguists at PanLex are teaming up with a group of local Balinese supporters to build new technologies and tools to keep their script and literary culture alive.

Unfortunately, these ancient teachings—which were created by etching script into dried palm leaves and blackening the words with soot—were in danger of being lost forever due to humidity and time. And although they contain vital pieces of Bali’s rich cultural heritage, the lontars are unreadable for most Balinese who conduct their modern lives more and more in Indonesian.

So in 2011, the Internet Archive launched a project with the Culture Office of Bali to photograph and upload to archive.org some 3,000 lontar manuscripts made up of 130,000 palm leaves—making up “90% of Bali’s literature,” according to Bali’s Minister of Culture. The Internet Archive has preserved these texts in the Balinese Digital Library collection, but they realized that simply digitizing the lontars was not enough, as the resulting images were not easy to share or understand.

This year, the Internet Archive began working with PanLex, an organization dedicated to keeping the world’s languages alive, to engineer methods to transcribe some 3,000 palm leaves online. The team discovered that keyboards do not easily support Balinese script and that there was no complete font for the language, so PanLex worked with a font designer to create new fonts and created a new Keyman keyboard that enables users to type Unicode Balinese script on standard keyboards. These new tools empower more people to participate in transcription and makes it easier to use another tool that auto-generates a Romanized version of Balinese. Transforming the Balinese lontars from lifeless PDFs to machine-readable text means the rich cultural information contained within the lontars will be easier to read, format, and share.

Something you writers may find useful: