Adventures in upgrading to Fedora 27/28 using ‘dnf system-upgrade’

[This post was drafted on the day Fedora 27 released, about half a year ago, but was not published. The issue bit me again with Fedora 28, so documenting it for referring next time.]

UPDATE: The issue occurred in Fedora 28 because I had exclude=grub2-tools in /etc/dnf/dnf.conf which is the reason error “nothing provides grub2-tools” was coming up. Removing that previously added and then forgotten line fixes the issue with updating grub2 packages.

With fedup and subsequently dnf improving the upgrade experience of Fedora for power users, last few system upgrades have been smooth, quiet, even unnoticeable. That actually speaks volumes of the maturity and user friendliness achieved by these tools.

Upgrading from Fedora 25 to 26 was so event-less and smooth (btw: I have installed and used every version of Fedora from its inception and the default wallpaper of Fedora 26 was the most elegant of them all!).

With that, on the release day I set out to upgrade the main workstation from Fedora 26 to 27 using dnf system-upgrade as documented. Before downloading the packages, dnf warned that upgrade cannot be done because of package dependency issues with grub2-efi-modules and grub2-tools.

Things go wrong!

I simply removed both the offending packages and their dependencies (assuming were probably installed for the grub2-breeze-theme dependency, but grub2-tools actually provides grub2-mkconfig) and proceeded with dnf upgrade --refresh and dnf system-upgrade download --refresh --releasever=27. If you are attempting this, don’t remove the grub2 packages yet, but read on!

Once the download and check is completed, running dnf system-upgrade reboot will cause the system reboot to upgrade target and actual upgrade happen.

Except, I was greeted with EFI MOK (Machine Owner Key) screen on reboot. Now that the grub2 bootloader is broken thanks to the removal of grub2-efi-modules and other related packages, a recovery must be attempted.

Rescue

It is important to have a (possibly EUFI enabled) live media where you can boot from. Boot into the live media and try to reinstall grub. Once booted in, mount the root filesystem under /mnt/sysimage, and EFI boot partition at /mnt/sysimage/boot/efi. Then chroot /mnt/sysimage and try to reinstall grub2-efi-x64 and shim packages. If there’s no network connectivity, don’t despair, nmcli is to your rescue. Connect to wifi using nmcli device wifi connect <ssid> password <wifi_password>. Generate the boot configuration using grub2-mkconfig -o /boot/efi/EFI/fedora/grub.cfg followed by actual install grub2-install --target=x86_64-efi /dev/sdX (the –target option ensures correct host installation even if the live media is booted via legacy BIOS). You may now reboot and proceed with the upgrade.

But this again failed at the upgrade stage because of grub package clash that dnf warned earlier about.

Solution

Once booted into old installation, take a backup of the /boot/ directory, remove the conflicting grub related packages, and copy over the backed up /boot/ directory contents, especially /boot/efi/EFI/fedora/grubx64.efi. Now rebooting (using dnf system-upgrade reboot) had  the grub contents intact and the upgrade worked smoothly.

For more details on the package conflict issue, follow this bug.

Advertisements

Attaching debugger and ptrace_scope

In Fedora 22, if you try to attach debugger to a running process, even if by the same user, gdb will politely refuse with error message:

ptrace: Operation not permitted.

The reason is a newly enabled security feature YAMA to specifically restrict inspecting memory of other programs. See the RH bug 1196825 for original discussion. This information is yet to reflect on the Fedora security features matrix.  On why this restriction is a good thing, read the Linux kernel documentation. Note that this restriction doesn’t affect program started by debugger, such as “gdb myprogram”. To enable debugging running programs, as root do:

echo 0 > /proc/sys/kernel/yama/ptrace_scope
To enable that permanently, do:
echo kernel.yama.ptrace_scope = 0 > /etc/sysctl.d/10-ptrace.conf

Libinput support added to Touchpad KCM

libinput is a library to handle input devices in Wayland compositors and to provide a generic X.Org input driver. It provides device detection, device handling, input device event processing and abstraction so minimize the amount of custom input code compositors need to provide the common set of functionality that users expect.

libinput is expected to replace input drivers such as synpatics in the future, and there’s already a drive to move input stack in Fedora 22 to libinput as can be seen in the discussion here. Most of the software would transparently work, except – as quickly noted in that discussion – the KDE touchpad configuration. Tochpad KCM exposes almost every knob present in the synaptics driver. In contrast, libinput exposes very few options to configure – such as tap-to-click and keeps most of the other options enabled by default – such as disable touchpad while typing. This is more sensible and efficient to do – when using synaptics driver, disabling touchpad while typing was accomplished by the synclient daemon.

As an aftermath of the discussion in Fedora, libinput maintainer Peter Hutter contacted KDE developers, including yours truly who is guilty of porting the kcm-touchpad to KDE Frameworks 5. As I know nothing about input stack or touchpads in general (phew), Peter was kind enough to step up, clone the kcm-touchpad and add support for libinput in addition to (existing) synaptics driver. All I had to do then, is to port it again to Frameworks 5.

Tochpad KCM running on libinput
Tochpad KCM running on libinput

As of last week, the review request to do that has been reviewed by Martin Gräßlin and David Edmundson and merged into the master branch – in the meantime I have obtained KDE developer commit access. In other words, Touchpad KCM now supports both libinput and synaptics drivers! If both are installed, libinput is preferred and exposes only the relevant options. As the KCM user interface exposes a lot of options, most of them remain disabled. Ideally, it should be re-designed – for example the GNOME mouse/touchpad configuration (including GNOME Tweak tool) exposes very few options. Alexander Mezin, kcm-touchpad maintainer has mentioned plan to rewrite it, I hope he will be able to find some time.

The updated kcm-tochpad packages for Fedora 20, 21 and Rawhide (what to become F22) are available for testing in the copr in Fedora repositories. For some caveats such as edge scrolling available only on single-touch touchpads, see the discussion and associated bug here. There’s still some issue with edge scrolling setting, I’m investigating it.

Paratype PT Serif and PT Mono fonts are now available in Fedora

Paratype has a set of nice Latin/pan-Cyrillic typefaces including sans-serif, serif and monospace fonts. The sans-serif typeface, PT Sans, released in 2010 has been part of Fedora for a long time and it is the default font for Cyrillic/Russian. It is a nice font for display in desktop, documents and web.

In 2011, PT Serif and PT Mono were added to the collection. They both are nice looking and very good quality fonts. All the fonts are also made available under OFL (Open Font License) and all it needed was someone to package them for Fedora. Something in my todo list for a long time, couple of weeks ago I have leveraged the spec file of paratype-pt-sans-fonts and packaged the serif and monospaced fonts. Paratype distributes the source tar balls separately for each set and Fedora mandates to create individual packages in such cases. Thanks to the review and comments from Fedora Fonts-SIG, especially Parag Nemade, two new font packages – paratype-pt-serif-fonts and paratype-pt-mono-fonts are now in Fedora repositories.

Obligatory screenshot of both fonts:

Paratype-PT-Serif-Mono-fonts

Adventures in upgrading to Fedora 19 using ‘fedup’

So… after upgrading Fedora 18 installation to Fedora 19 (using DVD ISO image) via 'fedup' tool which was quite event-less* (not quite, read below); decided to upgrade second laptop running Fedora 17 to 19 directly using network (note that F17->F19 upgrade is not supported, IIRC).

Downloaded the latest f18 fedup noarch rpm (hey, it’s python!) from Koji directly and installed on F17 system. Did a 'fedup-cli --verbose --network 19'. After half an hour and around 2.0 GB of package downloads including third party ones, new ‘Upgrade’ boot entry is set up and reboot into it.
Boots into F19 upgrade plymouth screen… nothing happens – instead of progressing upgrade, system simply reboots after few minutes. After some head-scratching and trying to capture systemd errors, found the culprit to be a mount point entry in /etc/fstab with ntfs-3g file system – mount attempt was timing out and boot fails. Commented out that part, and then upgrade ran successfully. Reboot again, new Fedora 19 boot menu appears, boot into it. The reboot saga continues, this time with different systemd errors. Then recollect that I probably did something stupid while fedup was running (probably installing another kernel), looking at the grub entry shows that the boot target is still 'system-upgrade.target' which does not exist anymore after the upgrade. Manually correct the grub entry and voila – booting succeeds. Fixed the screw-up later by regenerating grub config – 'grub2-mkconfig -o /boot/grub2/grub.cfg' and later re-enable ntfs-3g mount point in /etc/fstab which was commented.

(*) Coming back to the first upgrade of Fedora 18 to Fedora 19 using ISO image – I had a weird setup – that is, the F18 installation was on an external USB hard disk and I boot from USB (it suffices to say there are non-technical reasons why I can’t install Linux on that laptop). But the internal hard disk of the laptop is faster and I tend to keep data there. As someone following ‘diskless install’ for a few years now and FAT filesystem cannot hold the DVD images, I was always interested if the NTFS filesystem was supported for install from. The DVD ISO image was kept in the ‘internal’ hard disk with NTFS partition  but 'fedup-cli --verbose --iso /path/to/Fedora-19-i386-DVD.iso' loudly complains that the ISO file is on ‘external’ hard disk (which is true because the OS is running from USB disk and thus the ‘internal’ hard disk is considered ‘external’) and it is not supported at the moment.
The next option to try was 'fedup-cli --device' method. For this, mount the NTFS partition which has DVD ISO image, then loop-mount the ISO image in another mount point – add these in /etc/fstab, assuming that packages need to be taken from loop-mounted location during actual upgrade run. Run  'fedup-cli --device /media/Fedora' which prepares the upgrade (this option does not copy the rpm packages into /var/lib/fedora-upgrade as the mounted device already have them – a sane thing to do). Reboot into the ‘Upgrade’ boot entry but this quickly failed and resulting in cleaning up the boot entry followed by a reboot – the error scrolled down were all related to not finding the rpm files. I didn’t investigate the issue further then, but now realize that it must be related to the mount points added in /etc/fstab failing.
In the end I copied the DVD ISO image into slow external USB hard disk itself  with Ext4 partition (from which the OS runs) followed by a 'fedup-cli --verbose --iso /home/rajeesh/Fedora-19-i386-DVD.iso' which worked flawlessly.

This is the first time I tried 'fedup' tool  and it is remarkably good – stable, fast, polished and easy. What was really impressive is that the well thought and sane behaviours – if you upgrade via network and something went wrong on the first try, the packages already downloaded to /var/lib/fedora-upgrade were kept and not downloaded again – huge win on usability and bandwidth. Also the ‘Device’ method do not copy packages unnecessarily either. What would I love to see in the future? Support for keeping ISO images in ‘external’ hard disks, and of course for NTFS partitions. Also, even packages excluded from update in yum.conf were also considered for upgrade and downloaded – perhaps a feature request for fedup to skip them?

fedup is a great tool, I want to thank the developers for making it so.

Yum

Many years ago, when I first saw this thing called Linux and found that I could use it everyday (in the college Lab), it intrigued me so much that I spent days and nights with it. Learning new things every day.

I remember this particular story – trying to get MPlayer work on my friend’s desktop running RedHat 9. Only the college lab had internet connection, I was downloading the RPM, finds that it is too big to fit in a Floppy disk, so I cut it into smaller KB files, doing round trips from Lab to hostel room, finally stitch them together and try to install it. Then I got into fighting the ‘dependency hell’ – MPlayer had a lot of dependencies, so I have to then search for the dependencies individually in rpm.pbone.net and download all these RPMS, copy them into the floppy and try to install them all together – of course using ‘rpm -ivh‘. That, then would result into a new level of dependency, missing dozens of libxyz.so files. The end of the story is that I did manage to install MPlayer and play videos.

And then Fedora Core emerged. With it, we found Yum as the package manager and instantly found that it can solve the dependencies for me! Ever since then, the one single command I have run most would be “yum”. Over the years it gained a lot of new features and stability.

I have recently learned the tragic demise of Seth Vidal who developed yum; and though I never knew him personally; he has touched someone’s life at the other end of the world. Thank you, Seth.