SMC mentoring for GSoC 2014 too

Swathanthra Malayalam Computing is a free software collective engaged in language computing, development, localization, standardization and popularization of various Free and Open Source Softwares in Malayalam language. SMC developers have contributed to various Indian language computing efforts including fonts, spell checkers, hyphenation patterns (used by TeX, Libreoffice, Firefox), input methods etc. Last year, SMC was selected as a mentor for Google Summer of Code program and we successfully mentored 3 student projects – a web application to store and process bibliography data of books with i8n support, port SILPA into Flask application and restructure into standalone modules and Automated Shaping&Rendering testing, primarily for HarfBuzz.

Together with Santhosh, I have mentored the Automated Shaping&Rendering testing framework which we use to test Malayalam font changes against HarfBuzz. It can also be used to test Uniscribe shaping engine if compiled in Windows, or used against HarfBuzz with Uniscribe backend.

SMC is selected as an organization to mentor for GSoC again this year. If you are a student who wants to work on interesting problems, look at our project ideas. One of the problems I am particularly interested and to mentor is adding Indic shaping support to ConTeXt. Apart from the listed ideas, you can propose other ideas as well. Read the FAQ, you can reach us by mailing list or via IRC #smc-project on freenode.net.

SMC Malayalam fonts updated versions released

The Unicode fonts for Malayalam maintained by Swathanthra Malayalam Computing were last updated almost 2 years ago. They all were supporting just the v1 Indic opentype spec. But there were rendering problems with the fonts under Harfbuzz.

I was fortunate to attend the Open Source Language Summit 2012 (last year!) organized by Wikimedia Foundation and Red Hat (thank you, guys!) where many of the Indic language experts came together to work on issues at hand. The 2-days workshop helped me greatly to get much more insight into fonts, opentype spec and Harfbuzz in general. Since then I have been spending a lot of effort in updating and fixing the Malayalam fonts and also testing git snapshots of Harfbuzz and reporting issues to harfbuzz development list.

In the meantime, Harfbuzz matured enough and fixed many rendering issues. Thanks to the last Udupi hackfest by Behdad and Jonathan Kew, all known issues with Malayalam shaping has been addressed. And we were busy updating the fonts, opentype lookup rules and fixing bugs to work with old shapers (old pango, Qt, ICU Layout Engine, Windows XP) as well as the new ones (Harfbuzz, Uniscribe, Adobe). The v1 Indic opentype spec was a mess due to ‘undesirable’ Halant reordering (Consonant+Halant forms were ligated while it should have been Halant+Consonant). It has caused a lot of grief on the font developers and shaping engine developers side. With the v2 spec (mlm2 script tag for Malayalam), this has been changed and there is no need to perform Halant shifting anymore by shaping engines. I was leading the effort of porting to mlm2 spec of Malayalam fonts. We could port only Meera and Rachana for now, and RaghuMalayalam taken care by a few sed scripts.

During the 12th anniversary celebrations of Swathanthra Malayalam Computing group, the new version of fonts (5.1 supporting old shapers and 6.0 supporting new shapers) were released. See the email to smc-discuss for details. Remaining fonts also need to be updated, there is interest from community to collaborate on that. The new release will show up in Fedora 20.

In the process, I have learned quite some intricacies of the Indic opentype spec and would try to document them in a series of posts.

Adventues in upgrading to Fedora 19 using ‘fedup’

So… after upgrading Fedora 18 installation to Fedora 19 (using DVD ISO image) via 'fedup' tool which was quite event-less* (not quite, read below); decided to upgrade second laptop running Fedora 17 to 19 directly using network (note that F17->F19 upgrade is not supported, IIRC).

Downloaded the latest f18 fedup noarch rpm (hey, it’s python!) from Koji directly and installed on F17 system. Did a 'fedup-cli --verbose --network 19'. After half an hour and around 2.0 GB of package downloads including third party ones, new ‘Upgrade’ boot entry is set up and reboot into it.
Boots into F19 upgrade plymouth screen… nothing happens – instead of progressing upgrade, system simply reboots after few minutes. After some head-scratching and trying to capture systemd errors, found the culprit to be a mount point entry in /etc/fstab with ntfs-3g file system – mount attempt was timing out and boot fails. Commented out that part, and then upgrade ran successfully. Reboot again, new Fedora 19 boot menu appears, boot into it. The reboot saga continues, this time with different systemd errors. Then recollect that I probably did something stupid while fedup was running (probably installing another kernel), looking at the grub entry shows that the boot target is still 'system-upgrade.target' which does not exist anymore after the upgrade. Manually correct the grub entry and voila – booting succeeds. Fixed the screw-up later by regenerating grub config – 'grub2-mkconfig -o /boot/grub2/grub.cfg' and later re-enable ntfs-3g mount point in /etc/fstab which was commented.

(*) Coming back to the first upgrade of Fedora 18 to Fedora 19 using ISO image – I had a weird setup – that is, the F18 installation was on an external USB hard disk and I boot from USB (it suffices to say there are non-technical reasons why I can’t install Linux on that laptop). But the internal hard disk of the laptop is faster and I tend to keep data there. As someone following ‘diskless install’ for a few years now and FAT filesystem cannot hold the DVD images, I was always interested if the NTFS filesystem was supported for install from. The DVD ISO image was kept in the ‘internal’ hard disk with NTFS partition  but 'fedup-cli --verbose --iso /path/to/Fedora-19-i386-DVD.iso' loudly complains that the ISO file is on ‘external’ hard disk (which is true because the OS is running from USB disk and thus the ‘internal’ hard disk is considered ‘external’) and it is not supported at the moment.
The next option to try was 'fedup-cli --device' method. For this, mount the NTFS partition which has DVD ISO image, then loop-mount the ISO image in another mount point – add these in /etc/fstab, assuming that packages need to be taken from loop-mounted location during actual upgrade run. Run  'fedup-cli --device /media/Fedora' which prepares the upgrade (this option does not copy the rpm packages into /var/lib/fedora-upgrade as the mounted device already have them – a sane thing to do). Reboot into the ‘Upgrade’ boot entry but this quickly failed and resulting in cleaning up the boot entry followed by a reboot – the error scrolled down were all related to not finding the rpm files. I didn’t investigate the issue further then, but now realize that it must be related to the mount points added in /etc/fstab failing.
In the end I copied the DVD ISO image into slow external USB hard disk itself  with Ext4 partition (from which the OS runs) followed by a 'fedup-cli --verbose --iso /home/rajeesh/Fedora-19-i386-DVD.iso' which worked flawlessly.

This is the first time I tried 'fedup' tool  and it is remarkably good – stable, fast, polished and easy. What was really impressive is that the well thought and sane behaviours – if you upgrade via network and something went wrong on the first try, the packages already downloaded to /var/lib/fedora-upgrade were kept and not downloaded again – huge win on usability and bandwidth. Also the ‘Device’ method do not copy packages unnecessarily either. What would I love to see in the future? Support for keeping ISO images in ‘external’ hard disks, and of course for NTFS partitions. Also, even packages excluded from update in yum.conf were also considered for upgrade and downloaded – perhaps a feature request for fedup to skip them?

fedup is a great tool, I want to thank the developers for making it so.

Yum

Many years ago, when I first saw this thing called Linux and found that I could use it everyday (in the college Lab), it intrigued me so much that I spent days and nights with it. Learning new things every day.

I remember this particular story – trying to get MPlayer work on my friend’s desktop running RedHat 9. Only the college lab had internet connection, I was downloading the RPM, finds that it is too big to fit in a Floppy disk, so I cut it into smaller KB files, doing round trips from Lab to hostel room, finally stitch them together and try to install it. Then I got into fighting the ‘dependency hell’ – MPlayer had a lot of dependencies, so I have to then search for the dependencies individually in rpm.pbone.net and download all these RPMS, copy them into the floppy and try to install them all together – of course using ‘rpm -ivh‘. That, then would result into a new level of dependency, missing dozens of libxyz.so files. The end of the story is that I did manage to install MPlayer and play videos.

And then Fedora Core emerged. With it, we found Yum as the package manager and instantly found that it can solve the dependencies for me! Ever since then, the one single command I have run most would be “yum”. Over the years it gained a lot of new features and stability.

I have recently learned the tragic demise of Seth Vidal who developed yum; and though I never knew him personally; he has touched someone’s life at the other end of the world. Thank you, Seth.

Meiga is updated for Rawhide/Fedora 19

Recent mass rebuild of packages for Fedora 19 resulted in Meiga (lightweight content sharing tool) failing to build from source. A bug report has been filed which pointed at the build log. Build failure is caused by failing to link to libgssdp properly and my attempts to fix the issue didn’t succeed. Enrique was quick enough to produce a new release fixing the issue and Meiga has been updated to the new version in Rawhide as of last week. Packages can be found here.

Fedora 18 : Install from ISO file

I can’t remember the last time I burned a DVD when a new Fedora version releases. The preferred way of installing a new version is the ‘diskless’ install method provided the computer already is running one or the other version of Fedora. For guidelines, see the installation guide: http://docs.fedoraproject.org/en-US/Fedora/18/html/Installation_Guide/ap-medialess-install.html

As of Fedora 17, the Boot Options have been changed, and the “askmethod” parameter is deprecated, it is no longer recognized by the installer. Instead you need to use “inst.repo” parameter with appropriate syntax to specify install from network (http/ftp), nfs, hard disk et al.

I have obtained the Fedora 18 DVD ISO file, mounted it loopback, and copied the vmlinuz and initrd.img from isolinux folder to /boot/ and added a custom menu entry in /etc/grub.d/40_custom with the repo method as found in the documentation. Installer starts, but unfortunately fails, probably at stage2, with a dracut message saying “Cannot boot” and “Root device not found”. There’s no helpful error message or warning as to what could cause this.

Finally figured out that the repo parameter was missing a colon after the hard disk device. This had hit me during the Fedora 17 install too, so let it be documented for future reference. The proper parameter would be something like:

linux /boot/vmlinuz-fedora18-install repo=hd:/dev/sda2:/home/rajeesh/

Where the Fedora DVD ISO is placed at /home/rajeesh. Once that is fixed, installer boots into the graphical mode just fine. The new installer seems quite unstable – it crashed 4 times before I could complete installation – at disk probing, when switching to other terminals using Alt+Ctrl+F2, Alt+Ctrl+F3 etc. But once that phase is passed, the package installation is swift. There’s no way to customize and tweak package selection during the installation, which is quite limiting. That said, once the installation is done, the desktop is quite solid and polished.