Recently I have seen an Ubuntu 16.04 server producing segfaults errors in kern.log and dmesg. It was traced to a process calls nfsidmap

May 22 03:36:56 penguinoserver1 kernel: [241227.277964] nfsidmap[28966]: segfault at 0 ip 00007f2471ff6ad3 sp 00007ffc480fa048 error 4 in libc-2.23.so[7f2471f6d000+1c0000]
May 22 03:36:56 penguinoserver1 kernel: [241228.117058] nfsidmap[28970]: segfault at 0 ip 00007f044aa31ad3 sp 00007fff80f0a998 error 4 in libc-2.23.so[7f044a9a8000+1c0000]
May 22 03:41:59 penguinoserver1 kernel: [241530.495847] nfsidmap[29502]: segfault at 0 ip 00007f43660a7ad3 sp 00007fff98ec35a8 error 4 in libc-2.23.so[7f436601e000+1c0000]
May 22 03:41:59 penguinoserver1 kernel: [241530.686111] nfsidmap[29504]: segfault at 0 ip 00007f16e26f1ad3 sp 00007fff1f439398 error 4 in libc-2.23.so[7f16e2668000+1c0000]

Later I found that in /var/log/apport.log the offending command line appears.

ERROR: apport (pid 49878) Thu May  3 03:10:44 2018: executable: /usr/sbin/nfsidmap (command line "nfsidmap -t 600 480996374 uid:")

Running the command directly does replicate the problem

[email protected]:/etc/request-key.d# nfsidmap -t 600 692838889 uid:
Segmentation fault (core dumped)

I read that nfsidmap above is triggered by the file id_resolver.conf. But I do not know exactly how and why.

[email protected]:/etc/request-key.d# cat id_resolver.conf
create  id_resolver     *       *       /usr/sbin/nfsidmap -t 600 %k %d

More reading from the website

http://vcheru.blogspot.co.uk/2014/11/nfsv4-idmap-in-kernel-keyring-issues.html

reveals that those parameters were obtained from the keyring. Looking at the output, my keyring file contains invalid characters with the id_resolve entries.

[email protected]:/etc# cat /proc/keys

To resolve the issue, I use the nfsidmap clear command

[email protected]:/etc# nfsidmap -c

And those invalid entries disappeared. Afterwards I no longer saw those nfsidmap segfault errors.

[email protected]:/etc# cat /proc/keys
02b5a634 I--Q---     4 perm 1f3f0000  1002 65534 keyring   _uid.1002: empty
05c39a2a I--Q---     1 perm 1f3f0000     0 65534 keyring   _uid_ses.0: 1
0c5b9a13 I------     1 perm 1f0f0000     0     0 keyring   .ima_mok: empty
1770fbca I------     1 perm 1f0f0000     0     0 keyring   .ima: empty
179c7797 I------     1 perm 1f030000     0     0 keyring   .id_resolver: empty
24fb29d1 I------     1 perm 1f030000     0     0 keyring   .system_blacklist_keyring: empty
25ff868d I------     1 perm 1f0f0000     0     0 keyring   .ima_blacklist: empty
2817244a I------     1 perm 1f030000     0     0 keyring   .dns_resolver: empty
2b8807e2 I------     1 perm 1f030000     0     0 asymmetri Build time autogenerated kernel key: c9042d3cb5a38ebe2ba353e1bda9e4db9d7e4919:: X509.rsa 91bbe527 []
309aff75 I--Q---     2 perm 1f3f0000     0 65534 keyring   _uid.0: empty
3a12c6aa I--Q---    18 perm 3f030000  1002  1002 keyring   _ses: 1
3c8c2650 I------     1 perm 1f0b0000     0     0 keyring   .system_keyring: 1

This seems to be a bug in Ubuntu when nfsidmap has read an invalid key entry, then it would cause a segfault and core dump.