{"id":1252,"date":"2018-05-23T19:29:22","date_gmt":"2018-05-23T19:29:22","guid":{"rendered":"http:\/\/www.penguino.co.uk\/?p=1252"},"modified":"2018-05-23T19:33:09","modified_gmt":"2018-05-23T19:33:09","slug":"nfsidmap-segfault","status":"publish","type":"post","link":"http:\/\/www.penguino.co.uk\/computing\/nfsidmap-segfault","title":{"rendered":"nfsidmap segfault"},"content":{"rendered":"\n\n\r\n\n\n\n

Recently I have seen an Ubuntu 16.04 server producing segfaults errors in kern.log and dmesg. It was traced to a process calls nfsidmap<\/p>\n

May 22 03:36:56 penguinoserver1 kernel: [241227.277964] nfsidmap[28966]: segfault at 0 ip 00007f2471ff6ad3 sp 00007ffc480fa048 error 4 in libc-2.23.so[7f2471f6d000+1c0000]\r\nMay 22 03:36:56 penguinoserver1 kernel: [241228.117058] nfsidmap[28970]: segfault at 0 ip 00007f044aa31ad3 sp 00007fff80f0a998 error 4 in libc-2.23.so[7f044a9a8000+1c0000]\r\nMay 22 03:41:59 penguinoserver1 kernel: [241530.495847] nfsidmap[29502]: segfault at 0 ip 00007f43660a7ad3 sp 00007fff98ec35a8 error 4 in libc-2.23.so[7f436601e000+1c0000]\r\nMay 22 03:41:59 penguinoserver1 kernel: [241530.686111] nfsidmap[29504]: segfault at 0 ip 00007f16e26f1ad3 sp 00007fff1f439398 error 4 in libc-2.23.so[7f16e2668000+1c0000]<\/pre>\n
Later I found that in \/var\/log\/apport.log the offending command line appears.<\/p>\n
ERROR: apport (pid 49878) Thu May  3 03:10:44 2018: executable: \/usr\/sbin\/nfsidmap (command line \"nfsidmap -t 600 480996374 uid:\")<\/pre>\n
Running the command directly does replicate the problem<\/p>\n
root@penguinoserver1:\/etc\/request-key.d# nfsidmap -t 600 692838889 uid:\r\nSegmentation fault (core dumped)<\/pre>\n
I read that nfsidmap above is triggered by the file id_resolver.conf. But I do not know exactly how and why.<\/p>\n
root@penguinoserver1:\/etc\/request-key.d# cat id_resolver.conf\r\ncreate  id_resolver     *       *       \/usr\/sbin\/nfsidmap -t 600 %k %d<\/pre>\n
More reading from the website<\/p>\n
http:\/\/vcheru.blogspot.co.uk\/2014\/11\/nfsv4-idmap-in-kernel-keyring-issues.html<\/a><\/p>\n
reveals that those parameters were obtained from the keyring. Looking at the output, my keyring file contains invalid characters with the id_resolve entries.<\/p>\n
root@penguinoserver1:\/etc# cat \/proc\/keys<\/pre>\n
\"\"<\/a><\/p>\n
To resolve the issue, I use the nfsidmap clear command<\/p>\n
root@penguinoserver1:\/etc# nfsidmap -c<\/pre>\n
And those invalid entries disappeared. Afterwards I no longer saw those nfsidmap segfault errors.<\/p>\n
root@penguinoserver1:\/etc# cat \/proc\/keys\r\n02b5a634 I--Q---     4 perm 1f3f0000  1002 65534 keyring   _uid.1002: empty\r\n05c39a2a I--Q---     1 perm 1f3f0000     0 65534 keyring   _uid_ses.0: 1\r\n0c5b9a13 I------     1 perm 1f0f0000     0     0 keyring   .ima_mok: empty\r\n1770fbca I------     1 perm 1f0f0000     0     0 keyring   .ima: empty\r\n179c7797 I------     1 perm 1f030000     0     0 keyring   .id_resolver: empty\r\n24fb29d1 I------     1 perm 1f030000     0     0 keyring   .system_blacklist_keyring: empty\r\n25ff868d I------     1 perm 1f0f0000     0     0 keyring   .ima_blacklist: empty\r\n2817244a I------     1 perm 1f030000     0     0 keyring   .dns_resolver: empty\r\n2b8807e2 I------     1 perm 1f030000     0     0 asymmetri Build time autogenerated kernel key: c9042d3cb5a38ebe2ba353e1bda9e4db9d7e4919:: X509.rsa 91bbe527 []\r\n309aff75 I--Q---     2 perm 1f3f0000     0 65534 keyring   _uid.0: empty\r\n3a12c6aa I--Q---    18 perm 3f030000  1002  1002 keyring   _ses: 1\r\n3c8c2650 I------     1 perm 1f0b0000     0     0 keyring   .system_keyring: 1<\/pre>\n
This seems to be a bug in Ubuntu when nfsidmap has read an invalid key entry, then it would cause a segfault and core dump.<\/p>\n
 <\/p>\n\n\n\r\n\n\n\n","protected":false},"excerpt":{"rendered":"
Recently I have seen an Ubuntu 16.04 server producing segfaults errors in kern.log and dmesg. It was traced to a process calls nfsidmap May 22 03:36:56 penguinoserver1 kernel: [241227.277964] nfsidmap[28966]: segfault at 0 ip 00007f2471ff6ad3 sp 00007ffc480fa048 error 4 in libc-2.23.so[7f2471f6d000+1c0000] May 22 03:36:56 penguinoserver1 kernel: [241228.117058] nfsidmap[28970]: segfault at 0 ip 00007f044aa31ad3 sp 00007fff80f0a998 […]<\/p>\n","protected":false},"author":1,"featured_media":1256,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_s2mail":"yes","ngg_post_thumbnail":0},"categories":[3,14,13],"tags":[],"_links":{"self":[{"href":"http:\/\/www.penguino.co.uk\/wp-json\/wp\/v2\/posts\/1252"}],"collection":[{"href":"http:\/\/www.penguino.co.uk\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/www.penguino.co.uk\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/www.penguino.co.uk\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/www.penguino.co.uk\/wp-json\/wp\/v2\/comments?post=1252"}],"version-history":[{"count":0,"href":"http:\/\/www.penguino.co.uk\/wp-json\/wp\/v2\/posts\/1252\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"http:\/\/www.penguino.co.uk\/wp-json\/wp\/v2\/media\/1256"}],"wp:attachment":[{"href":"http:\/\/www.penguino.co.uk\/wp-json\/wp\/v2\/media?parent=1252"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/www.penguino.co.uk\/wp-json\/wp\/v2\/categories?post=1252"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/www.penguino.co.uk\/wp-json\/wp\/v2\/tags?post=1252"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}