## 2017년 5월 6일 토요일

### RHEL 7.X DBus.Error.AccessDenied caused by permissions problem on root partition

At a client site, I have 6 nodes running Openstack Mitaka on top of RHEL 7.2. After rebooting one node, however, the networking configs in /etc/sysconfig/network-scripts were not being loaded (specifically, the OVS bridges necessary for Openstack to run, i.e. br-ex, br-int). When I attempted to manually load the network settings with

systemctl start network

I was told that the systemd unit file network.service does not exist! This file is normally generated automatically by systemd-sysv-generator at boot from legacy SystemV scripts in /etc/init.d/ and written to /run/systemd/generator.late/network, but for some reason this was not happening.

Because RHEL 7.2 was not reading my network config files, I decided to manually create the OVS bridges using the following commands:

To use OpenVSwitch, however, the systemd's openvswitch.service must be running. When I tried to invoke the service using systemctl start openvswitch, I got the following error:

DBus.Error.AccessDenied: An SELinux policy prevents this sender from sending this message to this recipient

Also the journalctl log showed tons of auditd errors that continued to print every 3 seconds or so.

It turns out that this is a permissions problem on /! According to Redhat, the proper permissions on the root partition is 555, or r-x r-x r-x. After changing the permissions and rebooting, I no longer get the DBus.Error.AccessDenied error message. I don't know why the perm's on / have to be set as 555 (on a personal Archlinux installation without SELINUX, the perms on / are 755 rwx r-x r-x). Furthermore I don't know how the perms got changed to 555. I checked the history log on the affected nodes and there is no record of anyone changing permissions on the root partition.

References:
https://access.redhat.com/solutions/1990203 (you must register in order to access this Knowledge Base solution)

## 2017년 3월 11일 토요일

### Generate /etc/shadow PW hash from the cli using python2 and 3

In /etc/shadow, hashed and salted passwords are stored together with the user name as follows:

myuser:$6[someSaltedHash]:... where the number following the$ can take the values 1~6 corresponding to the following hash algorithms:

1    md5
2a   Blowfish
2y   Blowfish with correct 8-bit char handling
5    sha-256
6    sha-512

Many How-to's on the Internet recommend using mkpasswd from the expect package, but I find it is much easier to use python2 or python3 to generate the salted hash.

Python 2:

python -c 'import crypt,getpass; print crypt.crypt(getpass.getpass())'

You will then be prompted to enter your plaintext password after which a /etc/shadow compatible hash will be output.

Python 3:

python3 -c 'import crypt; print(crypt.crypt("yourpw", crypt.mksalt(crypt.METHOD_SHA512)))'

In this snippet, you simply enter your plaintext password as an argument and then a /etc/shadow compatible hash will be printed out on the terminal.

You can copy-paste this salted hash into a Kickstart (RHEL and variants) or DI preseed file (Debian and variants) for automated installations.

References:

https://access.redhat.com/solutions/221403 (requires registration)

## 2017년 2월 25일 토요일

### Using ibus in non-GTK/QT apps like Emacs, Java, and Enlightenment/EFL

ibus is a popular Input Method Editor (IME) for Linux which I use for entering Korean and Chinese characters (via ibus-hangul). ibus has good compatibility with apps using the GTK or QT UI frameworks, but ibus sometimes behaves strangely in GUI apps that don't use these frameworks. For example, in Emacs, OmegaT (which uses OpenJDK 7 or 8) and Terminology (a nice terminal that uses the Enlightenment Foundation Libraries), every time I press SPACEBAR after a word, the SPACE character is inserted to the left of the last character. In other words:

"가나다 "

appears as

"가나 다"

To avoid this issue in non-GTK/QT apps on Linux, I launch the ibus daemon as follows:

env IBUS_ENABLE_SYNC_MODE=0 ibus-daemon -rdx

You can read an explanation of the ibus-daemon option flags in a previous post from my blog.

Brandon Schaefer @ Canonical, explains why this error occurs:

The problem seems to be that when IBUS_ENABLE_SYNC_MODE is enabled it pushes all the events through the im engine (such as ibus-hangul) and since it normally only handles Korean text it doesn't know what to do when, say a space is sent through, so it says it didn't handle that space event which then IBus handles it and commits that space BEFORE the preedit.

In addition, my .bashrc contains the following ibus settings:

##### ibus IME settings #####
export GTK_IM_MODULE=ibus
export XMODIFIERS=@im=ibus
export QT_IM_MODULE=ibus
export CLUTTER_IM_MODULE=ibus
export ECORE_IMF_MODULE=xim

ECORE_IMF_MODULE setting is for Enlightenment apps like Terminology terminal.

Once the ibus environment variables are properly set and ibus-daemon is launched with the appropriate options, you will be able to enter Korean and other Asian text into non-QT/GTK apps with ibus.

References:
https://github.com/ibus/ibus/issues/1847

## 2017년 2월 2일 목요일

### Fix black screen in tty mode with Ubuntu 16.04.1 on ASUS Prime Z270-K with KabyLake CPU

After installing Ubuntu 16.04.1 LTS via USB in UEFI mode, I rebooted and was met by a black screen. From another machine on the local network, I was able to ssh into the newly-installed Ubuntu 16.04.1 box and noticed the following in dmesg:

[  +0.000000] Call Trace:
[  +0.000002]  [] dump_stack+0x63/0x90
[  +0.000001]  [] warn_slowpath_common+0x82/0xc0
[  +0.000001]  [] warn_slowpath_fmt+0x5c/0x80
[  +0.000014]  [] __unclaimed_reg_debug+0x80/0x90 [i915_bpo]
[  +0.000002]  [] ? __pm_runtime_resume+0x5b/0x70
[  +0.000016]  [] intel_digital_port_connected+0xf8/0x290 [i915_bpo]
[  +0.000013]  [] ? intel_display_power_get+0x3b/0x50 [i915_bpo]
[  +0.000015]  [] intel_hdmi_detect+0x4b/0x140 [i915_bpo]
[  +0.000003]  [] drm_helper_probe_single_connector_modes_merge_bits+0x235/0x4d0 [drm_kms_helper]

Google reveals many similar issues with i915 Kernel Mode Setting for Intel integrated video on newer intel CPU's (Skylake and beyond).

At the GRUB menu after rebooting, I added the kernel boot parameter nomodeset and booted with F10 after which I was able to boot to a visible login prompt now that Linux is no longer trying to automatically set the video resolution.

To make the kernel boot parameter changes permanent in GRUB menu, I had to edit /etc/default/grub and edited the following:

GRUB_CMDLINE_LINUX_DEFAULT="nomodeset"

I then ran sudo update-grub to update /boot/grub/grub.cfg

Some other hardware info:

$lscpu ... Model name: Intel(R) Core(TM) i7-7700 CPU @ 3.60GHz$ dmidecode -t 2
...
Base Board Information
Manufacturer: ASUSTeK COMPUTER INC.
Product Name: PRIME Z270-K
Version: Rev X.0x

References:

## 2017년 1월 21일 토요일

### Migrate from cinder loopback device to physical block device on RDO Mitaka

In PoC or test installations of RDO Mitaka via Packstack, by default the cinder-volumes LVM volume group is created in the form of a loopback file/device under /var/lib/cinder/

This might be OK for light testing, but if you plan to use cinder volumes in production you need to create the cinder-volumes VG on a real physical device.

After stopping all openstack services with openstack-service stop I used vgs and lvs to take a look at the LVM volume groups and logical volumes on my Openstack storage node (which I separated from the control node using unsupported config options in my Packstack answer file).

Despite stopping Openstack services, when I tried to use lvchange -ay and vgchange -ay to deactivate LV's in the cinder-volumes VG I kept getting error messages that some logical volumes in the VG were still active.

I finally just used gdisk to delete the problematic LVM partition housing cinder-volumes, rebooted and then created a new cinder-volumes VG as detailed in my previous post about setting up Cinder to use a physical block device.

However, after restarting Openstack services, openstack-cinder-volume.service failed to start. I examined the systemd service file for cinder-volume.service in /usr/lib/systemd/system/ and noticed that it contained the line

After = openstack-losetup.service

It turns out that /usr/lib/systemd/system/openstack-losetup.service sets up a loop device to act as a "disk" for storing the cinder-volumes Volume Group. Of course the loop device is just a file and has really bad I/O, so it should only be used for test setups.

I therefore deleted openstack-losetup.service, removed the ...losetup line from the systemd config for cinder-volume.service, and then executed systemctl daemon-reload to reload all systemd service files.

Now openstack-cinder-volume.service starts without any errors and doesn't require a loopback device.

## 2016년 12월 31일 토요일

### Kernel boot parameters for grub and bootctl when using sd-encrypt mkinitcpio hook

In Archlinux, the initial ramdisk which contains kernel modules needed for booting the system is generated by a bash script named mkinitcpio. In /etc/mkinitcpio.conf you specify which kernel modules you need at boot and then you build the initrd image (which Archlinux calls initramfs-linux.img) with mkinitcpio -p linux

mkinitcpio will work OOTB for most standard configurations, i.e. root partition in a regular, non-LVM partition, no encrypted partitions, etc. But in my case, I use LVM2 partitions for my root, home, var and other system partitions with only /boot being a regular partition. In addition, these LVM2 partitions are stored inside of a LUKS encrypted container. The initramfs-linux.img created by default mkinitcpio settings cannot handle such things as encrypted partitions and root partitions located inside LVM2 partitions, so you must customize mkinitcpio.conf accordingly.

mkinitcpio.conf settings for root partition located inside of LVM2 in LUKS (using sd-encrypt hook)

Inside /etc/mkinitcpio.conf locate the line starting with HOOKS=
The default settings as of Dec. 31, 2016 should look something like:

HOOKS="base udev autodetect modconf block filesystems keyboard fsck"

You should edit this line so it looks like:

HOOKS="base systemd sd-encrypt autodetect modconf block sd-lvm2 filesystems keyboard fsck"

Instead of the udev module, you will use systemd, and LUKS partitions will be handled by systemd's sd-encrypt module. Also notice there is a LVM2 kernel module present, but this module is the systemd version sd-lvm2.

Once you have edited mkinitcpio.conf, generate your initial ramdisk with mkinitcpio -p linux

The next step is to add kernel boot options to GRUB (for BIOS machines) or systemd-bootctl (for UEFI machines).

Kernel boot options for sd-encrypt and sd-lvm2 in /etc/default/grub (BIOS)

If you have an older computer that doesn't have UEFI firmware but uses BIOS, Archlinux uses GRUB to boot your system. You have to edit /etc/default/grub with the appropriate options for sd-encrypt and then regenerate your grub configuration in /boot/grub/grub.cfg

Find the line GRUB_CMDLINE_LINUX_DEFAULT= and edit it as follows:

GRUB_CMDLINE_LINUX_DEFAULT="luks.uuid=72369889-3971-4039-ab17-f4db45aeeef2 root=UUID=f4dbfa30-35ba-4d5e-8d43-a7b29652f575 rw"

Note that the luks.uuid is the UUID for your LUKS encrypted partition BEFORE it has been opened with cryptsetup open

You can find the UUID for all partitions with blkid

root=UUID= refers to the UUID of your root partition which resides inside of a LVM partition which itself is inside of a LUKS encrypted container.

If you haven't written the GRUB bootloader to your disk yet, you would then execute:

grub-install --recheck /dev/sda
(assuming sda is the HDD from which you will boot)

grub-mkconfig -o /boot/grub/grub.cfg

Kernel boot options for sd-encrypt and sd-lvm2 in /boot/loader/entries/arch.conf

timeout 5
default arch

This particular menu contains only a single entry, but you can add entries for other Linux distros (systemd-bootctl can chainload to GRUB) and OS'es like Windows.

The OS-specific config files are stored under /boot/loader/entries/ and my arch-specific config is in /boot/loader/entries/arch.conf

title Archlinux
linux /vmlinuz-linux
initrd /intel-ucode.img
initrd /initramfs-linux.img
options luks.uuid=ba04cb93-328c-4f9a-a2a9-4a86cee0f592 luks.name=UUID=ba04cb93-328c-4f9a-a2a9-4a86cee0f592=luks root=/dev/mapper/ARCH-rootvol rw
options acpi_osi=! acpi_osi="Windows 2009"

The kernel boot options appear on the line starting with the keyword options

For luks.uuid you need to specify the UUID of your LUKS partition BEFORE it has been opened by cryptsetup. luks.name is not strictly necessary; it simply assigns an easy-to-read name to your LUKS partition. Finally, root specifies which partition your root partition is located on so that the kernel can proceed with booting. For the root parameter, you can specify the partition using /dev/mapper syntax or UUID. If you use UUID you must use the syntax root=UUID=...

After editing your arch.conf it is not necessary to rebuild any images or regenerate configs as is the case with GRUB.

Warning: Do not mix up the syntax for sd-encrypt and encrypt mkinitcpio HOOKS

Before systemd became popular, the initial ramdisk for Archlinux used busybox. When booting from an encrypted root and LVM2 with busybox, the mkinitcpio HOOKS used are encrypt and lvm2 (as opposed to sd-encrypt and sd-lvm2 for systemd)

An /etc/mkinitcpio.conf customized for LUKS and LVM2 (without systemd) would look like:

HOOKS="base udev encrypt autodetect modconf block lvm2 filesystems keyboard fsck"

Notice that the encrypt hook comes just after udev. If you were using systemd boot, however, udev would be replaced by systemd.

The related settings in /etc/default/grub (BIOS) or /boot/loader/entries/arch.conf (UEFI) would then look something like the following:

/etc/default/grub
GRUB_CMDLINE_LINUX_DEFAULT="cryptdevice=UUID=5a2527d8-8972-43f6-a32d-2c50f734f7a9:ARCH root=/dev/mapper/ARCH-rootvol"

title Arch
linux /vmlinuz-linux
initrd /intel-ucode.img
initrd /initramfs-linux.img
options cryptdevice=UUID=f121bc44-c8c2-4881-b9d5-da4d1b49f3e4:ARCH root=/dev/mapper/ARCH-rootvol rw

The keyword cryptdevice will only work with the encrypt module in your initial ramdisk. If you are using sd-encrypt, however, you must replace cryptdevice=UUID= with luks.uuid=

References:

https://wiki.archlinux.org/index.php/Dm-crypt/System_configuration#mkinitcpio

## 2016년 12월 17일 토요일

### Mirroring PyPI locally

I recently had to set up a local PyPI (Python Package Index) mirror at a client site so that developers could download pip packages locally instead of from the Internet. I used bandersnatch to mirror the packages hosted on PyPI. Bandersnatch can be installed via pip and mirroring requires just two steps.

Customize /etc/bandersnatch.conf

[mirror]
; The directory where the mirror data will be stored.
directory = /MIRROR/pypi

By default, a logfile is specified, but bandersnatch will crash with the error ConfigParser.NoSectionError: No section: 'formatters' if the logfile does not exist. I simply commented out the line to fix this problem:

;log-config = /etc/bandersnatch-log.conf

You can see a sample bandersnatch.conf file at this pastebin:
http://paste.openstack.org/show/586323/

Start the mirroring process

You can start mirroring PyPI with the following command:

sudo bandersnatch mirror

As of Oct. 2016, PyPI takes up 380 GB on disk. Although mirroring requires just two steps, you are not done yet. You still have to make the packages available over your network and you must setup pip clients to use your local mirror.

Make the local mirror available over the network

Once you have mirrored pip packages, the next step is to setup an NFS share to the path where PyPI packages are mirrored or serve the mirror over HTTP using a webserver like Apache, nginx, darkhttpd, etc. I chose to go with darkhttpd because it doesn't require any setup whatsoever. You simply invoke darkhttpd in the path you wish to share over http. If you invoke it as sudo, it will serve files over port 80, but if you invoke it as the regular user, it will serve files over port 8080. You can specify the TCP port used with the --port option.

sudo darkhttpd /MIRROR

Now the Document Root for the webserver is /MIRROR under which the pypi folder is located. I generally place all my local mirrors (i.e. Ubuntu 16.04 repo, CentOS 7 repo, RHEL repo, etc) under the Document Root so that they are all available over http.

Assuming your local mirror machine has the IP address 10.10.10.5, if you navigate to http://10.10.10.5 in your browser (or with cURL) you should be able to see the pypi folder.

Customize pip clients to use local PyPI mirror

On client machines, create the directory ~/.pip and then create ~/.pip/pip.conf (if it doesn't exist already). Now add the following:

[global]
index-url = http://10.10.10.5/pypi/web/simple
trusted-host = 10.10.10.5

This of course assumes that clients are on the 10.10.10.x subnet and that your PyPI mirror is located on 10.10.10.5

In addition, the web server's document root should be /MIRROR as detailed in the previous section.