Some observations about glibc GHOST vulnerability patching in the field

I currently work as a Linux System Engineer for an open source software service company that provides manpower for various Managed Service Providers (MSP) serving the Korean telecom industry. I spend most of my days out of the office on service calls ranging from basic server installation, system monitoring, to troubleshooting. In the normal course of my work I come into contact with sysadmins with varying levels of Linux familiarity. About 50% of the time the sysadmins I meet are more familiar with Windows or various flavors of Unix like HPUX, AIX, Solaris, etc. Of course, there are some sysadmins who are quite adept at Linux, too.

Thanks to an itinerant work arrangement in which I visit different sites every day, I have some perspective on the diverse ways that IT staff maintain their Linux servers. 2014 and early 2015 have been a busy time for patching servers with bugs found in Bash, OpenSSL and glibc. Although the glibc GHOST vulnerability was announced in Jan 2015, some of our clients still haven't completed patching all of their servers. Several sysadmins have asked me about the proper rpm commands for upgrading glibc-related packages on RHEL or CentOS. I was a bit worried when they said they used commands like

rpm -Uvh --nodeps pkgName

rpm -i --force --nodeps pkgName

as hacky workarounds for dependency errors that popped up when they tried to update glibc.

For the record, at my employer Growin, we recommend the following steps for upgrading glibc.

1) First identify relevant glibc packages installed on your RHEL/CentOS system

rpm -qa | grep -E "glibc|nscd" | grep -v "compat"

This will return a list of glibc-related packages on your system with the exclusion of compat- packages which don't need to be upgraded. For why compat-glibc doesn't need to be patched, see the Redhat Solutions post Is compat-glibc affected by GHOST, glibc vulnerability CVE-2015-0235? (login required). I quote,

The dynamic libraries provided by the compat-glibc package are not vulnerable because they do not provide runtime code...

2) Although there are more than 16 packages related to glibc that you could install, most probably you don't have glibc-debug* or the glibc-utils packages installed on your system. If these packages did not appear in the results from step 1, do not try to install them (doing so can lead to dependency errors)! Prepare only the updated rpm's that you need for an upgrade in a separate directory and then

rpm -Uvh glibc*

should do the trick. If your system has nscd installed, the above command would become:

rpm -Uvh glibc* nscd*


It is also possible to skip step 1 if you use the rpm -F flag (--freshen), which, according to man rpm:

will upgrade packages, but only ones for which an earlier version is installed

Desktop Linux users of Fedora, CentOS, or other rpm-based distros might wonder why sysadmins go to all the trouble of manually upgrading packages using rpm -Uvh pkgName when they could just do a yum update and upgrade all packages at one go.

There are several good reasons for not using yum update in an enterprise environment. First, some servers are only connected to an internal network. Second, custom applications created by developers may be compiled against a certain version of the C libraries in glibc, so doing a yum update runs the risk of breaking applications. Mainly for the second reason, it is not uncommon to see production servers running really old kernels like 2.6.18 (RHEL 5.X), and I have heard horror stories from coworkers about companies that use even older kernels!