A new set of OpenType shaping rules for Malayalam script

TLDR; research and development of a completely new OpenType layout rules for Malayalam traditional orthography.

Writing OpenType shaping rules is hard. Writing OpenType shaping rules for advanced (complex) scripts is harder. Writing OpenType shaping rules without causing any undesired ligature formations is even harder.

Background

The shaping rules for SMC fonts abiding v2 of Malayalam OpenType speification (mlm2 script tag) were polished in large part by me over many years, fixing shaping errors and undesired ligature formations. It still left some hard to fix bugs. Driven by the desire to fix such difficult bugs in RIT fonts and the copyright fiasco, I have set out to write a simplified OpenType shaping rules for Malayalam from scratch. Two major references helped in that quest: (1) a radically different approach I have tried few years ago but failed with mlym script tag (aka Windows XP era shaping); (b) a manuscript by R. Chithrajakumar of Rachana Aksharavedi who devised the ‘definitive character set’ for Malayalam script. The idea of ‘definitive character set’ is that it contains all the valid characters in a script and it doesn’t contain any (invalid) characters not in the script. By the definition; I wanted to create the new shaping rules in such a way that it does not generate any invalid characters (for e.g. with a detached u-kar).

Fig. 1. Samples of Malayalam definitive character set listing by R. Chithrajakumar, circa 1999. Source: K.H. Hussain.

“Simplify, simplify, simplify!”

Henry David Thoreau

It is my opinion that a lot of complexity in the Malayalam shaping largely comes from Indic OpenType shaping specification largely follows Devanagari, which in turn was adapted from ISCII, which has (in my limited understanding) its root in component-wise metal type design of ligature glyphs. Many half, postbase and other shaping rules have their lineage there. I have also heard similar concerns about complexity expressed by others, including Behdad Esfahbod, FreeFont maintainer et al.

Implementation

As K.H. Hussain once rightly noted, the shaping rules were creating many undesired/unnecessary ligature glyphs by default, and additional shaping rules (complex contextual lookups) are written to avoid/undo those. A better, alternate approach would be: simply don’t generate undesired ligatures in the first place.

“Invert, always invert.”

Carl Gustav Jacob Jacobi

Around December 2019, I set out to write a definitive set of OpenType shaping rules for traditional script set of Malayalam. Instead of relying on many different lookup types such as pref, pstf, blwf, pres, psts and myriad of complex contextual substitutions, the only type of lookup required was akhn — because the definitive character set contains all ligatures of Malayalm and those glyphs are designed in the font as a single glyph — no component based design.

The draft rules were written in tandem with RIT-Rachana redesign effort and tested against different shaping engines such as HarfBuzz, Allsorts, XeTeX, LuaHBTeX and DirectWrite/Uniscribe for Windows. Windows, being Windows (also being maintainers of OpenType specification), indeed did not work as expected adhering to the specification. Windows implementation clearly special cased the pstf forms of യ (Ya, 0D2F) and വ (Va, 0D35). To make single set of shaping rules work with all these shaping engines, the draft rules were slightly amended, et voila — it worked in all applications and OSen that use any of these shaping engines. It was decided to drop support for mlym script which was deprecated many years ago and support only mlm2 specification which fixed many unfixable shortcomings of mlym. One notable shaping engine which doesn’t work with these rules is Adobe text engine (Lipika?), but they have recently switched to HarfBuzz. That covers all major typesetting applications.

Testing fonts developed using this new set of shaping rules for Malayalam indeed showed that they do not generate any undesired ligatures in the first place. In addition, compared to the previous shaping rules, it gets rid of 70+ lines of complex contextual substitutions and other rules, while remaining easy to read and maintain.

Old vs new shaping rules in Rachana
Fig. 3. Old vs new shaping rules in RIT Rachana.

Application support

This new set of OpenType layout rules for Malayalam is tested to work 100% with following shaping engines:

  1. HarfBuzz
  2. Allsorts
  3. DirectWrite/Uniscribe (Windows shaping engine)

And GUI toolkits/applications:

  1. Qt (KDE applications)
  2. Pango/GTK (GNOME applications)
  3. LibreOffice
  4. Microsoft Office
  5. XeTeX
  6. LuaHBTeX
  7. Emacs
  8. Adobe InDesign (with HarfBuzz shaping engine)
  9. Adobe Photoshop
  10. Firefox, Chrome/Chromium, Edge browsers

Advantages

In addition, the advantages of the new shaping rules are:

  1. Adheres to the concept of ‘definitive character set’ of the language/script completely. Generate all valid conjunct characters and do not generate any invalid conjunct character.
  2. Same set of rules work fine without adjustments/reprogramming for ‘limited character set’ fonts. The ‘limited character set’ may not contain conjunct characters as extensive in the ‘definitive character set’; yet it would always have characters with reph and u/uu-kars formed correctly.
  3. Reduced complexity and maintenance (no complex contextual lookups, reverse chaining etc.). Write once, use in any fonts.
  4. Open source, libre software.

This new OpenType shaping rules program was released to public along with RIT Rachana few months ago, and also used in all other fonts developed by RIT. It is licensed under Open Font License for anyone to use and integrate into their fonts, please ensure the copyright statements are preserved. The shaping rules are maintained at RIT GitLab repository. Please create an issue in the tracker if you find any bugs; or send a merge request if any improvement is made.

Panmana: new Malayalam body text font

Rachana Institute of Typography starts the new year 2021 with the release of a new body-text Malayalam Unicode font named ‘Panmana’.

Fig. 1: ‘Panmana’ font specimen.

The font is named after and dedicated to Prof. Panmana Ramachandran Nair who steadfastly voiced for the original script of Malayalam. It is designed by K.H. Hussain with inputs from Ashok Kumar and CVR and font engineering by Rajeesh (your correspondent); maintained by RIT.

‘Panmana’ is released under Open Font License, free to use and share. Truetype and Web font can be downloaded from the website. A flyer about the font is available. If you spot any issues, please report those in the source repository.

RIT Rachana: a classic typeface reimagined

It was around 2006 I started reading and writing Malayalam (my native language) text widely on the computer, thanks to Unicode and proliferation of Malayalam blogs. It was also at the same time that I noticed Malayalam text was not ‘shaped’ correctly in many cases on my primary operating system — GNU/Linux. A number of Unicode fonts were available under libre license, of which I liked Rachana the most.

Cut to chase: few years later, I ended up co-maintaining Rachana, trying to fix all the known bugs and succeeded to a large extent; among many other things.

In 2020, with new insights into the design metrics of Malayalam fonts, the designer of Rachana — KH Hussain redrew all the glyphs of Rachana, completely overhauled Bold variant and freshly designed Italic & BoldItalic styles. All fonts in the new typeface contain more than 1100 glyphs with entire Malayalam characters encoded in Unicode version 13.0 and all conjuncts/ligatures in the definitive character set of Malayalam traditional orthography. The Latin glyphs are adapted from TeX Gyre Schola with express permission from GUST. These make the font suitable to typeset contemporary text, novels, poetry, scholarly works, Sanskrit text, Bible, archaic books and everything in between.

Fig. 1: RIT Rachana glyph redesign samples.

Not satisfied with solutions on how to fix some remaining shaping bugs in Rachana, I have researched and ventured to try radically different approach to complex advanced text shaping rules for traditional Malayalam script fonts. Following the v2 version of Indic OpenType specification (mlm2), a completely new set of shaping rules were written from the scratch. Though it was bit of a struggle to get Uniscribe/Windows with its idiosyncrasies to shape correctly, and Adobe InDesign need this fix, it proved to be a great success. The new set of rules fix all known shaping bugs to my knowledge. The development of this shaping rule program is a blog entry for another time.

A comparison of problematic shaping combinations can be found in Fig. 2. Note that two “സ്വാതന്ത്ര്യം” differ in code points in their order of “ര്യ” and “യ്ര” and their shaping should be different.

Fig. 2: RIT Rachana improved shaping of problematic conjuncts.

In the process, the build script and test cases were also written from scratch.

The result is a new font named RIT Rachana, released under libre Open Font License free to download and use by individuals, designers, organizations, institutions, government departments and media houses.

Fig. 3: RIT Rachana variants/styles.

All four variants of RIT Rachana can be downloaded from the website for desktop and web usage. If you notice any issues, report them at the source repository.

The typeface is a fruit of months of labour of many, including the designers and developers, early users and testers and feedbacks from those: especially Ashok Kumar, CVR, Sayahna typesetters and sysadmins.