2014년 11월 18일 화요일

Troubleshooting a failure of ntpd.service at system startup

Several months ago, I noticed that journalctl was containing messages about ntpd.service failing. systemctl status ntpd also confirmed that systemd failed to load ntpd. A quick and dirty hack (that doesn't solve the underlying problem) is to just run sudo ntpd -qgd to manually load the ntp daemon (and update system time even if there is a difference of over 1000s between local time and ntp server time). This has the effect of making a one-time change to the system clock after ntpd queries the Network Time Protocol servers defined in /etc/ntp.conf

Today I had some free time so I decided to take a closer look at the problem. I discovered several issues:

1. Manually starting ntpd daemon conflicts with starting systemd ntp.service

I know it sounds like common sense, but at times I seem to lack this resource. This problem is characterized by the following error message in journalctl:
...
unable to bind to wildcard address :: - another process may be running - EXITING

By checking running processes, we can see that, sure enough, ntpd is already running:

[archjun@arch ~]$ ps aux | grep ntp
root      1699  0.0  0.3 105200 14588 ?        SLs  10:47   0:00 ntpd
archjun  28055  0.0  0.0  11908  2276 pts/2    S+   11:00   0:00 grep ntp

So problem #1 was solved by doing a kill -15 on pid 1699 shown above.


2. Create user ntp

Invoking systemctl start ntpd still didn't work, however. journalctl -f (equivalent of tail -f /var/log/messages for non-systemd machines) showed the following error:

Nov 18 11:02:59 arch ntpd[1241]: Cannot find user `ntp'
Nov 18 11:02:59 arch systemd[1]: ntpd.service: main process exited, code=exited, status=255/n/a

That's strange. Despite re-installing the ntp package through pacman, user ntp was not created (checked with cat /etc/passwd |grep ntp), although group ntp was created (verified with cat /etc/group |grep ntp).

I tried to create user ntp with a simple useradd ntp, but my system complained that there was already a group with the same name. I thus added user ntp and added them to group ntp all in the same command:

useradd ntp -g ntp

Now when I run systemctl start ntpd everything looks fine when checked with systemctl status ntpd and in journalctl:

Nov 18 11:07:43 arch systemd[1]: Starting Network Time Service...
Nov 18 11:07:43 arch ntpd[11063]: ntpd 4.2.7p465@1.2483-o Sun Sep  7 07:03:04 UTC 2014 (1): Starting
Nov 18 11:07:43 arch ntpd[11063]: Command line: /usr/bin/ntpd -g -u ntp:ntp
Nov 18 11:07:43 arch systemd[1]: Started Network Time Service.
Nov 18 11:07:43 arch ntpd[11064]: proto: precision = 1.047 usec (-20)
Nov 18 11:07:43 arch ntpd[11064]: Listen and drop on 0 v6wildcard [::]:123
Nov 18 11:07:43 arch ntpd[11064]: Listen and drop on 1 v4wildcard 0.0.0.0:123
Nov 18 11:07:43 arch ntpd[11064]: Listen normally on 2 lo 127.0.0.1:123
Nov 18 11:07:43 arch ntpd[11064]: Listen normally on 3 wlp12s0 192.168.0.9:123
Nov 18 11:07:43 arch ntpd[11064]: Listen normally on 4 lo [::1]:123
Nov 18 11:07:43 arch ntpd[11064]: Listen normally on 5 wlp12s0 [fe80::21f:3cff:fe46:6467%3]:123
Nov 18 11:07:43 arch ntpd[11064]: Listening on routing socket on fd #22 for interface updates

Finally, a helpful thread I referred to from the Archlinux forums:

https://bbs.archlinux.org/viewtopic.php?id=155120