Smarter tabular editing with Vim

I happen to edit tabular data in LaTeX format quite a bit. Being scientific documents, the table columns are (almost) always left-aligned, even for numbers. That warrants carefully crafted decimal and digit alignment on such columns containing only numbers.

I also happen to edit the text (almost) always in Vim, and just selecting/changing a certain column only is not easily doable (like in a spreadsheet). If there are tens of rows that needs manual digit/decimal align adjustment, it gets even more tedious. There must be another way!

Thankfully, smarter people already figured out better ways (h/t MasteringVim).

With that neat trick, it is much more palatable to look at the tabular data and edit it. Even then, though, it is not possible to search & replace only within a column using Visual Block selection. The Visual Block (^v) sets mark on the column of first row till the column on last row, so any :<','>s/.../.../g would replace any matching text in-between (including any other columns).

To solve that, I’ve figured out another way. It is possible to copy the Visual Block alone and pasting any other content over (though cutting it and pasting would not work as you think). Thus, the plan is:

• Copy the required column using Visual Block (^v + y)
• Open a new buffer and paste the copied column there
• Edit/search & replace to your need in that buffer, so nothing else would be unintentionally changed
• Select the modified content as Visual Block again, copy/cut it and come back to the main buffer/file
• Re-select the required column using Visual Block again and paste over
• Profit!

Here’s a short video of how to do so. I’d love to hear if there are better ways.

Demo of column editing in Vim

Advertisements

Powerline git dirty status without powerline_gitstatus

With git-prompt it is possible to display the dirty state (when a tracked file is modified) by setting the env variable GIT_PS1_SHOWDIRTYSTATE=true. Powerline can display the status of a git repository, such as number of commits ahead/behind, number of modified files etc. using the powerline_gitstatus module. Unfortunately, Fedora doesn’t have it packaged. I did some digging in, and found that there’s colour highlighting for branch_dirty and powerline.segments.common.vcs.branch function (which displays the current branch name) takes 2 parameters  to modify its behaviour. Modify the shell theme /etc/xdg/powerline/themes/shell/default.json under the left segment (because only left works in shell) then as follows:

...
    {   "function": "powerline.segments.common.vcs.branch",
        "args": {"ignore_statuses": ["U"], "status_colors": true},
        "priority": 20
    }
...

The branch will now be highlighted if a tracked file is modified (ignore_statuses = ["U"] causes untracked files to be ignored). Clean repository:

Clean repo

Once a tracked file is modified:

Dirty repo

Adventures in upgrading to Fedora 27/28 using ‘dnf system-upgrade’

[This post was drafted on the day Fedora 27 released, about half a year ago, but was not published. The issue bit me again with Fedora 28, so documenting it for referring next time.]

UPDATE: The issue occurred in Fedora 28 because I had exclude=grub2-tools in /etc/dnf/dnf.conf which is the reason error “nothing provides grub2-tools” was coming up. Removing that previously added and then forgotten line fixes the issue with updating grub2 packages.

With fedup and subsequently dnf improving the upgrade experience of Fedora for power users, last few system upgrades have been smooth, quiet, even unnoticeable. That actually speaks volumes of the maturity and user friendliness achieved by these tools.

Upgrading from Fedora 25 to 26 was so event-less and smooth (btw: I have installed and used every version of Fedora from its inception and the default wallpaper of Fedora 26 was the most elegant of them all!).

With that, on the release day I set out to upgrade the main workstation from Fedora 26 to 27 using dnf system-upgrade as documented. Before downloading the packages, dnf warned that upgrade cannot be done because of package dependency issues with grub2-efi-modules and grub2-tools.

Things go wrong!

I simply removed both the offending packages and their dependencies (assuming were probably installed for the grub2-breeze-theme dependency, but grub2-tools actually provides grub2-mkconfig) and proceeded with dnf upgrade --refresh and dnf system-upgrade download --refresh --releasever=27. If you are attempting this, don’t remove the grub2 packages yet, but read on!

Once the download and check is completed, running dnf system-upgrade reboot will cause the system reboot to upgrade target and actual upgrade happen.

Except, I was greeted with EFI MOK (Machine Owner Key) screen on reboot. Now that the grub2 bootloader is broken thanks to the removal of grub2-efi-modules and other related packages, a recovery must be attempted.

Rescue

It is important to have a (possibly EUFI enabled) live media where you can boot from. Boot into the live media and try to reinstall grub. Once booted in, mount the root filesystem under /mnt/sysimage, and EFI boot partition at /mnt/sysimage/boot/efi. Then chroot /mnt/sysimage and try to reinstall grub2-efi-x64 and shim packages. If there’s no network connectivity, don’t despair, nmcli is to your rescue. Connect to wifi using nmcli device wifi connect <ssid> password <wifi_password>. Generate the boot configuration using grub2-mkconfig -o /boot/efi/EFI/fedora/grub.cfg followed by actual install grub2-install --target=x86_64-efi /dev/sdX (the –target option ensures correct host installation even if the live media is booted via legacy BIOS). You may now reboot and proceed with the upgrade.

But this again failed at the upgrade stage because of grub package clash that dnf warned earlier about.

Solution

Once booted into old installation, take a backup of the /boot/ directory, remove the conflicting grub related packages, and copy over the backed up /boot/ directory contents, especially /boot/efi/EFI/fedora/grubx64.efi. Now rebooting (using dnf system-upgrade reboot) had  the grub contents intact and the upgrade worked smoothly.

For more details on the package conflict issue, follow this bug.

Sundar — a new traditional orthography ornamental font for Malayalam

There is a dearth of good Unicode fonts for Malayalam script. Most publishing houses and desktop publishing agencies still rely on outdated ASCII era fonts. This not only causes issues with typesetting using present technologies, it makes the ‘document’ or ‘data’ created using these fonts and tools absolutely useless — because the ‘document/data’ is still Latin, not Malayalam.

Rachana Institute of Typography (rachana.org.in) has designed and published a new traditional orthography ornamental Unicode font for Malayalam script, for use in headings, captions and titles. It is named after Sundar, who was a relentless advocate of open fonts, open standards and open publishing. He dreamed of making available several good quality Malayalam fonts, particularly created by Narayana Bhattathiri with his unique calligraphic and typographic signature, freely and openly to the users. The font is licensed under OFL.

The font follows traditional orthography for Malayalam, rather than the unpleasing reformed orthography which was solely introduced due to the technical limitations of typewriters in the ’70s. Such restrictions do not apply to computers and present technology, so it is possible to render the classic beauty of Malayalam script using Unicode and Opentype technologies.

‘Sundar’ is designed by K.H. Hussain — known for his work on Rachana and Meera fonts which comes pre-installed with most Linux distributions; and Narayana Bhattathiri — known for his beautiful calligraphy and lettering in Malayalam script. Graphic engineers of STM Docs (stmdocs.in) did the vectoring and glyph creation. Yours truly took care of the Opentype feature programming. The font can be freely downloaded from rachana.org.in.

The source code of ‘Sundar’, licensed under OFL is available at https://gitlab.com/rit-fonts/Sundar.

Switching Raspbian to Pixel desktop

Official Raspbian images based on Debian Stretch by default has the Pixel desktop environment and will login new users to it. But if you have had a Raspbian installation with another DE (such as LXDE), here are the steps to install and login to the Pixel desktop.

apt-get install raspberrypi-ui-mods
sed -i 's/^autologin-user=pi/#autologin-user=pi/' /etc/lightdm/lightdm.conf
update-alternatives --set x-session-manager /usr/bin/startlxde-pi
sed -i 's/^Session=.*/Session=lightdm-xsession/' ${USER}/.dmrc

Make sure the user’s ‘.dmrc’ file is updated with the new startlxde-pi session as that is where lightdm login manager looks to decide which desktop should be launched.

Improvement in converting video/audio files with VLC

VLC Media Player has the ability to convert video/audio files into various formats it supports, since a long time. There is a dedicated “Convert/Save” menu for converting single or multiple files at once into a different format, with limited ‘editing’ features such as specifying a start time, caching options etc. It is quite useful for basic editing/cropping of multimedia files.

As an example, one of the easiest ways to create a custom iPhone ringtone is to create a “.m4r” (AAC format) file exactly 40 seconds long. It is a matter of selecting your favourite music file and doing a “Convert/Save” with appropriate “Profile”. A “Profile” specifies the video/audio encoding to be used, which can be easily customized by selecting different audio and video codecs.

The options “Caching time”, “Play another media synchronously” (think adding different sound track to a video clipping) and a “Start time” etc can be specified under “Show more options” button and even more advanced functionality is available by making use of the “Edit Options” line. Internally, all the options specified at this line are passed to the converter.

There was one thing lacking in this “Convert/Save” dialog though – there was no possibility to specify a “Stop Time” akin to the “Start Time”, in the GUI (although it can be manually specified in the “Edit Options”, but you need to calculate the time in milliseconds). VLC 2.x series convert looks like as follows – notice the lack of “Stop time”:

Being bugged by this minor annoyance, I set out to add the missing “Stop-time” functionality. Going through the codebase of VLC, it was relieving to see that the converter backend already supports “:stop-time=” option (akin to “:start-time=”). It was then a matter of adding “Stop Time” to the GUI and properly updating the “Edit Options” when user changes the value.

A working patch was then sent to vlc-devel mailing list for review and feedback. After 5 rounds of review and constructive feedback from Filip Roséen the code was cleaned up (including existing code) which is now committed to the master branch. This functionality should be available to users in the upcoming 3.0 release. Screenshot below:

 

Convert iPhone contacts to vCard

On a recent troubleshooting attempt, I lost all the contacts in my Android phone. It had also received a recent update which took away the option to import contacts from another phone via bluetooth.
I still had some contacts in the old iPhone, but now that mass transfer via bluetooth is gone, it was a question of manually sending each contact in vCard format to the Android phone. That means I should probably find a less dreadful way to get the contacts back.

Here is one way to extract contacts en-masse from iPhone into popular vCard format. The contact and address details in iOS are stored by AddressBook application in a file named ‘AddressBook.sqlitedb’ which is an sqlite database. The idea is to open this database using sqlite, extract the details from a couple of tables and convert the entries into vCard format.

Disclaimer: the iPhone is an old 3GS running iOS 6 and it is jailbroken. If you attempt this, your mileage would vary. Required tools/softwares are usbmuxd (especially libusbmuxd-utils) and sqlite, with the prerequisite that openssh server is running on the jailbroken iPhone.

1. Connect iPhone via USB cable to the Linux machine. Run iproxy 2222 22 to connect to the openssh server running on the jailbroken phone. iproxy comes with libusbmuxd-utils package.
2. Copy the addressbook sqlite database from phone:scp -P 2222 mobile@localhost:/var/mobile/Library/AddressBook/AddressBook.sqlitedb .Instead of steps 1 and 2 above, it might be possible to copy this file using Nautilus (gvfs-afc) or Dolphin (kio_afc) file manager, although I’m not sure if the file is accessible.
3. Extract the contact and address details from the sqlite db (based on this forum post):sqlite3 -cmd ".out contacts.txt" AddressBook.sqlitedb "select ABPerson.prefix, ABPerson.first,ABPerson.last,ABPerson.organization, c.value as MobilePhone, h.val ue as HomePhone, he.value as HomeEmail, w.value as WorkPhone, we.value as WorkEmail,ABPerson.note from ABPerson left outer join ABMultiValue c on c.record_id = ABPerson.ROWID and c.label = 1 and c.property= 3 left outer join ABMultiValue h on h.record_id = ABPerson.ROWID and h.label = 2 and h.property = 3 left outer join ABMultiValue he on he.record_id = ABPerson.ROWID and he.label = 2 and he.property = 4 left outer join ABMultiValue w on w.record_id = ABPerson.ROWID and w.label = 4 and w.property = 3 left outer join ABMultiValue we on we.record_id = ABPerson.ROWID and we.label = 4 and we.property = 4;"
4. Convert the extracted contact details to vCard format:cat contacts.txt | awk -F\| '{print "BEGIN:VCARD\nVERSION:3.0\nN:"$3";"$2";;;\nFN:"$2" "$3"\nORG:"$4"\nEMAIL;type=INTERNET;type=WORK;type=pref:" $9"\nTEL;type=CELL;type=pref:"$5"\nTEL;TYPE=HOME:"$6"\nTEL;TYPE=WORK:"$8"\nNOTE:"$9"\nEND:VCARD\n"}' > Contacts.vcf
5. Remove the empty content lines if some contacts do not have all the different fields:sed -i '/.*:$/d' Contacts.vcf

Now simply transfer the Contact.vcf file containing all the contact details to Android phone’s storage and import contacts from there.