RHEL 7.X DBus.Error.AccessDenied caused by permissions problem on root partition

At a client site, I have 6 nodes running Openstack Mitaka on top of RHEL 7.2. After rebooting one node, however, the networking configs in /etc/sysconfig/network-scripts were not being loaded (specifically, the OVS bridges necessary for Openstack to run, i.e. br-ex, br-int). When I attempted to manually load the network settings with

systemctl start network

I was told that the systemd unit file network.service does not exist! This file is normally generated automatically by systemd-sysv-generator at boot from legacy SystemV scripts in /etc/init.d/ and written to /run/systemd/generator.late/network, but for some reason this was not happening.

Because RHEL 7.2 was not reading my network config files, I decided to manually create the OVS bridges using the following commands:

ovs-vsctl add-br br-ex
ip link set br-ex up
ovs-vsctl add-port br-ex eno1

To use OpenVSwitch, however, the systemd's openvswitch.service must be running. When I tried to invoke the service using systemctl start openvswitch, I got the following error:

DBus.Error.AccessDenied: An SELinux policy prevents this sender from sending this message to this recipient

Also the journalctl log showed tons of auditd errors that continued to print every 3 seconds or so.

It turns out that this is a permissions problem on /! According to Redhat, the proper permissions on the root partition is 555, or r-x r-x r-x. After changing the permissions and rebooting, I no longer get the DBus.Error.AccessDenied error message. I don't know why the perm's on / have to be set as 555 (on a personal Archlinux installation without SELINUX, the perms on / are 755 rwx r-x r-x). Furthermore I don't know how the perms got changed to 555. I checked the history log on the affected nodes and there is no record of anyone changing permissions on the root partition.




References:
https://access.redhat.com/solutions/1990203 (you must register in order to access this Knowledge Base solution)

EFI multiboot for Ubuntu 15.10 Wily and Archlinux

At work I recently had the opportunity to install Ubuntu 15.10 Wily Werewolf (released on Oct 22, 2015) on a company laptop using EFI boot instead of legacy BIOS. Ubuntu installs just fine on post-2011 hardware that has UEFI boot enabled, but note that Ubuntu uses the grub2 bootloader on top of EFI, whereas other Linux distributions use bootctl from systemd-boot or other EFI boot managers.

The company laptop initially had Archlinux installed with an encrypted root inside LVM on a LUKS partition with lots of free space left over for the installation of Ubuntu 15.10. Installing Ubuntu as the second OS on an EFI boot machine ran into two problems:

1. If you choose to install Ubuntu 15.10 into a new LUKS partition on a disk already containing other LUKS partitions, the Debian Installer will erase all other LUKS headers when it creates the new LUKS partition. In fact, this is a known issue and the developer of cryptsetup warns (section 1.2 WARNINGS) against installing Ubuntu with LUKS on a disk that already contains other LUKS partition:

https://gitlab.com/cryptsetup/cryptsetup/wikis/FrequentlyAskedQuestions

UBUNTU INSTALLER: In particular the Ubuntu installer seems to be quite willing to kill LUKS containers in several different ways. Those responsible at Ubuntu seem not to care very much (it is very easy to recognize a LUKS container), so treat the process of installing Ubuntu as a severe hazard to any LUKS container you may have...the installer offers to create LUKS partitions in a way that several people mistook for an offer to activate their existing LUKS partition. The installer gives no or an inadequate warning and will destroy your old LUKS header, causing permanent data loss

2. Ubuntu can fail to install grub2 into the ESP (EFI System Partition) if another boot manager has already been installed there. I think this occurs because the path /boot/efi doesn't exist, which can happen if another linux distro (like Archlinux) is first installed and a boot manager like bootctl has created the path /boot/EFI on the ESP. The sole difference is EFI in all-caps or not, but this seemingly-minor issue causes problems for the Ubuntu installer.


Workaround

I therefore strongly recommend that you install Ubuntu as your first OS before installing other Linux distros to create a EFI multiboot system. Archlinux plays well with other EFI boot managers installed to the ESP.

During the installation of Archlinux as the second OS on a EFI boot machine, the ESP will be mounted as /boot (actually /mnt/boot before chrooting into the new system). To install the bootctl boot manager, simply invoke

bootctl install

inside the chroot, which will then install bootctl into the ESP.

I created Archlinux as my default boot entry at /boot/loader/entries/arch.conf :

title Arch
linux /vmlinuz-linux
initrd /intel-ucode.img
initrd /initramfs-linux.img
options cryptdevice=UUID=b332de40-afe8-47d5-9512-bf03da8d13cc:ARCH root=/dev/mapper/ARCH-rootvol quiet rw

The example above includes boot options for opening a LUKS partition and accessing '/' inside LVM.

Now you also need to create an entry for ubuntu so that bootctl will load grub2 menu for 15.10 Wily. Let's call it /boot/loader/entries/ubuntu.conf:

title Ubuntu 15.10
efi /EFI/ubuntu/grubx64.efi

Finally, you need to add the Ubuntu 15.10 entry to /boot/loader/loader.conf:

timeout 10
default arch
ubuntu

Now when you get the to EFI boot menu, you will see entries for both Archlinux and Ubuntu.

Troubleshooting a failure of ntpd.service at system startup

Several months ago, I noticed that journalctl was containing messages about ntpd.service failing. systemctl status ntpd also confirmed that systemd failed to load ntpd. A quick and dirty hack (that doesn't solve the underlying problem) is to just run sudo ntpd -qgd to manually load the ntp daemon (and update system time even if there is a difference of over 1000s between local time and ntp server time). This has the effect of making a one-time change to the system clock after ntpd queries the Network Time Protocol servers defined in /etc/ntp.conf

Today I had some free time so I decided to take a closer look at the problem. I discovered several issues:

1. Manually starting ntpd daemon conflicts with starting systemd ntp.service

I know it sounds like common sense, but at times I seem to lack this resource. This problem is characterized by the following error message in journalctl:
...
unable to bind to wildcard address :: - another process may be running - EXITING

By checking running processes, we can see that, sure enough, ntpd is already running:

[archjun@arch ~]$ ps aux | grep ntp
root      1699  0.0  0.3 105200 14588 ?        SLs  10:47   0:00 ntpd
archjun  28055  0.0  0.0  11908  2276 pts/2    S+   11:00   0:00 grep ntp

So problem #1 was solved by doing a kill -15 on pid 1699 shown above.


2. Create user ntp

Invoking systemctl start ntpd still didn't work, however. journalctl -f (equivalent of tail -f /var/log/messages for non-systemd machines) showed the following error:

Nov 18 11:02:59 arch ntpd[1241]: Cannot find user `ntp'
Nov 18 11:02:59 arch systemd[1]: ntpd.service: main process exited, code=exited, status=255/n/a

That's strange. Despite re-installing the ntp package through pacman, user ntp was not created (checked with cat /etc/passwd |grep ntp), although group ntp was created (verified with cat /etc/group |grep ntp).

I tried to create user ntp with a simple useradd ntp, but my system complained that there was already a group with the same name. I thus added user ntp and added them to group ntp all in the same command:

useradd ntp -g ntp

Now when I run systemctl start ntpd everything looks fine when checked with systemctl status ntpd and in journalctl:

Nov 18 11:07:43 arch systemd[1]: Starting Network Time Service...
Nov 18 11:07:43 arch ntpd[11063]: ntpd 4.2.7p465@1.2483-o Sun Sep  7 07:03:04 UTC 2014 (1): Starting
Nov 18 11:07:43 arch ntpd[11063]: Command line: /usr/bin/ntpd -g -u ntp:ntp
Nov 18 11:07:43 arch systemd[1]: Started Network Time Service.
Nov 18 11:07:43 arch ntpd[11064]: proto: precision = 1.047 usec (-20)
Nov 18 11:07:43 arch ntpd[11064]: Listen and drop on 0 v6wildcard [::]:123
Nov 18 11:07:43 arch ntpd[11064]: Listen and drop on 1 v4wildcard 0.0.0.0:123
Nov 18 11:07:43 arch ntpd[11064]: Listen normally on 2 lo 127.0.0.1:123
Nov 18 11:07:43 arch ntpd[11064]: Listen normally on 3 wlp12s0 192.168.0.9:123
Nov 18 11:07:43 arch ntpd[11064]: Listen normally on 4 lo [::1]:123
Nov 18 11:07:43 arch ntpd[11064]: Listen normally on 5 wlp12s0 [fe80::21f:3cff:fe46:6467%3]:123
Nov 18 11:07:43 arch ntpd[11064]: Listening on routing socket on fd #22 for interface updates

Finally, a helpful thread I referred to from the Archlinux forums:

https://bbs.archlinux.org/viewtopic.php?id=155120

Thoughts on working in an Enterprise Linux environment coming from a Desktop Linux background

Linux for me started out as a hobby and only recently has it become my vocation. This means that I am naturally more familiar with desktop Linux environments and this is the background which I carry with me while learning the ropes of the Enterprise Linux (EL) world. I will discuss several of the biggest differences I have encountered between the Desktop and Enterprise domains.

I. Kernels

So far all of the production servers I have worked with are running kernels < linux 3.0
In fact, the most up-to-date kernel I have seen in a production environment is 2.6.32... which is the same version used in Debian Squeeze 6.0 (which I used as my desktop distro in 2010). Kernel 2.6.32... is used in the RHEL/CentOS 6.X series. I also regularly encounter kernel 2.6.18... used in the RHEL/CentOS 5.X series.

II. Packages

Since our customers' production servers use relatively old kernels (compared to the most recent kernel 3.17 which is currently in testing), many of the packages I am used to using in the desktop world are unavailable when I access servers over SSH or the built-in server console.

For example, Python3 is not available from the stock repos for RHEL 5.X/6.X
Also the handy tool lsblk (which lists block devices as well as filesystem info) is only available for kernels 2.6.27 and above, which means this package is not available in RHEL/CentOS 5, which uses kernel 2.6.18... There are countless of other examples.

III. net-tools vs. iproute2

In the EL world, almost all networking tasks are accomplished with utilities from the net-tools package which is becoming deprecated in the desktop Linux world. For example, bleeding-edge distros like Archlinux stopped using net-tools in 2011 in favor of the iproute2 package. Some common syntax differences between similar tasks using utils from net-tools vs. iproute2:

Put up a network interface:

ifup eth0

ip link set eth0 up

Specify an IP for a network interface:

ifconfig eth0 192.168.10.100/24 up

ip addr add 192.168.10.100/24 dev eth0

Note that the ifconfig command above will replace any existing IP addresses assigned to eth0. However the iproute2 command above will add the specified IP address to eth0 in addition to any other IP addresses already assigned to the interface. If you wish to remove any existing addresses before adding a new address using iproute2, you must do the following:

ip addr flush dev eth0

ip addr add 192.168.10.100/24 dev eth0

Of course there are many more examples like netstat being replaced by ss and other utils, etc.

IV. Systemd

In the desktop world, most distros have moved to systemd from init scripts (SysVinit) and upstart. I am now more comfortable with the systemctl action serviceName (i.e. systemctl restart dnsmasq) syntax than with the old service daemonName action (i.e. service dhcpd stop) syntax. In the Enterprise world, servers on kernels < 3.0 do not use systemd.

Many common tasks such as querying the system log are totally different in servers which do not use systemd. The first example shows the non-systemd way of checking the system log, while the second example shows how it's done in systemd:

tail -f /var/log/messages

journalctl -f


I personally think that being exposed to both the Desktop and Enterprise worlds is good for me as a Linux engineer, because I am forced to learn multiple ways of accomplishing the same tasks. When I was just administering a few machines at home, I never used tools like scp, sftp, and other networking-related utils. As a matter of necessity I have also started to write Bash scripts to automate the repetitive and mundane aspects of my job, particularly generating system reports after new Linux installations. Now is an exciting time to be a Linux system engineer!