Letsencrypt certificate renewal: Nginx with reverse-proxy

Let’s Encrypt revolutionized the SSL certificate management for websites in a short span of time — it directly improved the security of users of the world wide web by: (1) making it very simple to deploy SSL certificates to websites by administrators and (2) make the certificates available free of cost. To appreciate their efforts, compare to what hoops one had to jump through to obtain a certificate from a certificate authority (CA) and how much money and energy one would have to spend on it.

I make use of letsencrypt in all the servers I manitain(ed) and in the past used the certbot tool to obtain & renew certificates. Recent versions of certbot are only available as a snap package, which is not something I’d want to or able to setup in many cases.

Enter acme. It is shell script that works great. Installing acme will also setup a cron job, which would automatically renew the certificate for the domain(s) near its expiration. I have recently setup dict.sayahna.org using nginx as a reverse proxy to a lexonomy service and acme for certificate management. The cron job is supposed to renew the certificate on time.

Except it didn’t. Few days ago received a notification from about imminent expiry of the certificate. I have searched the interweb quite a bit, but didn’t find a simple enough solution (“make the proxy service redirect the request”…). What follows is the troubleshooting and a solution, may be someone else find it useful.

Problem

acme was unable to renew the certificate, because the HTTP-01 authentication challenge requests were not answered by the proxy server where all traffic was being redirected to. In short: how to renew letsencrypt certificates on an nginx reverse-proxy server?

Certificate renewal attempt by acme would result in errors like:

# .acme.sh/acme.sh --cron --home "/root/.acme.sh" -w /var/www/html/
[Sat 08 May 2021 07:28:17 AM UTC] ===Starting cron===
[Sat 08 May 2021 07:28:17 AM UTC] Renew: 'my.domain.org'
[Sat 08 May 2021 07:28:18 AM UTC] Using CA: https://acme-v02.api.letsencrypt.org/directory
[Sat 08 May 2021 07:28:18 AM UTC] Single domain='my.domain.org'
[Sat 08 May 2021 07:28:18 AM UTC] Getting domain auth token for each domain
[Sat 08 May 2021 07:28:20 AM UTC] Getting webroot for domain='my.domain.org'
[Sat 08 May 2021 07:28:21 AM UTC] Verifying: my.domain.org
[Sat 08 May 2021 07:28:24 AM UTC] my.domain.org:Verify error:Invalid response from https://my.domain.org/.well-known/acme-challenge/Iyx9vzzPWv8iRrl3OkXjQkXTsnWwN49N5aTyFbweJiA [NNN.NNN.NNN.NNN]:
[Sat 08 May 2021 07:28:24 AM UTC] Please add '--debug' or '--log' to check more details.
[Sat 08 May 2021 07:28:24 AM UTC] See: https://github.com/acmesh-official/acme.sh/wiki/How-to-debug-acme.sh
[Sat 08 May 2021 07:28:25 AM UTC] Error renew my.domain.org.

Troubleshooting

The key error to notice is

Verify error:Invalid response from https://my.domain.org/.well-known/acme-challenge/Iyx9vzzPWv8iRrl3OkXjQkXTsnWwN49N5aTyFbweJiA [NNN.NNN.NNN.NNN]

Sure enough, the resource .well-known/acme-challenge/… is not accessible. Let us try to make that accessible, without going through proxy server.

Solution

First, create the directory if it doesn’t exist. Assuming the web root as /var/www/html:

# mkdir -p /var/ww/html/.well-known/acme-challenge

Then, edit /etc/nginx/sites-enabled/my.domain.org and before the proxy_pass directive, add the .well-known/acme-challenge/ location and point it to the correct location in web root. Do this on both HTTPS and HTTP server blocks (otherwise it didn’t work for me).

 6 server {
 7   listen 443 default_server ssl;
...
43   server_name my.domain.org;
44   location /.well-known/acme-challenge/ {
45     root /var/www/html/;
46   }
47  
48   location / {
49     proxy_pass http://myproxyserver;
50     proxy_redirect off;
51   }
...
83 server {
84   listen 80;
85   listen [::]:80;
86 
87   server_name my.domain.org;
88 
89   location /.well-known/acme-challenge/ {
90     root /var/www/html/;
91   }
92 
93   # Redirect to HTTPS
94   return 301 https://$server_name$request_uri;

Make sure the configuration is valid and reload the nginx configuration

nginx -t && systemctl reload nginx.service

Now, try to renew the certificate again:

# .acme.sh/acme.sh --cron --home "/root/.acme.sh" -w /var/www/html/
...
[Sat 08 May 2021 07:45:01 AM UTC] Your cert is in  /root/.acme.sh/my.domain.org/dict.sayahna.org.cer 
[Sat 08 May 2021 07:45:01 AM UTC] Your cert key is in  /root/.acme.sh/my.domain.org/my.domain.org.key 
[Sat 08 May 2021 07:45:01 AM UTC] v2 chain.
[Sat 08 May 2021 07:45:01 AM UTC] The intermediate CA cert is in  /root/.acme.sh/my.domain.org/ca.cer 
[Sat 08 May 2021 07:45:01 AM UTC] And the full chain certs is there:  /root/.acme.sh/my.domain.org/fullchain.cer 
[Sat 08 May 2021 07:45:02 AM UTC] _on_issue_success

Success.

Panmana: new Malayalam body text font

Rachana Institute of Typography starts the new year 2021 with the release of a new body-text Malayalam Unicode font named ‘Panmana’.

Fig. 1: ‘Panmana’ font specimen.

The font is named after and dedicated to Prof. Panmana Ramachandran Nair who steadfastly voiced for the original script of Malayalam. It is designed by K.H. Hussain with inputs from Ashok Kumar and CVR and font engineering by Rajeesh (your correspondent); maintained by RIT.

‘Panmana’ is released under Open Font License, free to use and share. Truetype and Web font can be downloaded from the website. A flyer about the font is available. If you spot any issues, please report those in the source repository.

RIT Rachana: a classic typeface reimagined

It was around 2006 I started reading and writing Malayalam (my native language) text widely on the computer, thanks to Unicode and proliferation of Malayalam blogs. It was also at the same time that I noticed Malayalam text was not ‘shaped’ correctly in many cases on my primary operating system — GNU/Linux. A number of Unicode fonts were available under libre license, of which I liked Rachana the most.

Cut to chase: few years later, I ended up co-maintaining Rachana, trying to fix all the known bugs and succeeded to a large extent; among many other things.

In 2020, with new insights into the design metrics of Malayalam fonts, the designer of Rachana — KH Hussain redrew all the glyphs of Rachana, completely overhauled Bold variant and freshly designed Italic & BoldItalic styles. All fonts in the new typeface contain more than 1100 glyphs with entire Malayalam characters encoded in Unicode version 13.0 and all conjuncts/ligatures in the definitive character set of Malayalam traditional orthography. The Latin glyphs are adapted from TeX Gyre Schola with express permission from GUST. These make the font suitable to typeset contemporary text, novels, poetry, scholarly works, Sanskrit text, Bible, archaic books and everything in between.

Fig. 1: RIT Rachana glyph redesign samples.

Not satisfied with solutions on how to fix some remaining shaping bugs in Rachana, I have researched and ventured to try radically different approach to complex advanced text shaping rules for traditional Malayalam script fonts. Following the v2 version of Indic OpenType specification (mlm2), a completely new set of shaping rules were written from the scratch. Though it was bit of a struggle to get Uniscribe/Windows with its idiosyncrasies to shape correctly, and Adobe InDesign need this fix, it proved to be a great success. The new set of rules fix all known shaping bugs to my knowledge. The development of this shaping rule program is a blog entry for another time.

A comparison of problematic shaping combinations can be found in Fig. 2. Note that two “സ്വാതന്ത്ര്യം” differ in code points in their order of “ര്യ” and “യ്ര” and their shaping should be different.

Fig. 2: RIT Rachana improved shaping of problematic conjuncts.

In the process, the build script and test cases were also written from scratch.

The result is a new font named RIT Rachana, released under libre Open Font License free to download and use by individuals, designers, organizations, institutions, government departments and media houses.

Fig. 3: RIT Rachana variants/styles.

All four variants of RIT Rachana can be downloaded from the website for desktop and web usage. If you notice any issues, report them at the source repository.

The typeface is a fruit of months of labour of many, including the designers and developers, early users and testers and feedbacks from those: especially Ashok Kumar, CVR, Sayahna typesetters and sysadmins.