Setting up mcelog to work with systemd

If you regularly observe system logs such as /var/log/messages, dmesg, or journalctl (systemd) you will eventually encounter a Machine Check Event (mce) which warns you that some kind of hardware error has occurred. For example, a common mce is caused when an incorrect bit is flipped in RAM. For server ECC memory, this is less of a problem because such bit errors can be fixed automatically. When encountering servers in the field with uptime greater than 365 days, it is not hard to find mce errors logged here and there. In the case of RHEL 5/6 machines I encounter in the field, mce errors are logged in the file /var/log/mcelog and the mcelog service runs by default.

Recently I noticed that every few days the kernel ring buffer dmesg on my work laptop gives the following error:

[Jun11 23:49] mce: [Hardware Error]: Machine check events logged

However, when I navigate to /var/log/ I cannot see any file named mcelog. Some old posts floating around the Internet recommend redirecting mcelog to some output file, i.e. /usr/sbin/mcelog > mcelog.out but this didn't work for me. Make sure you have the mcelog package (as it is called in Arch) installed . To enable the daemon in systemd, systemctl enable mcelog. When running mcelog on a Linux machine running systemd instead of the old syslog, you need to make some changes to /etc/mcelog/mcelog.conf

What led me astray was the Archwiki page on MCE Handling, which recommends uncommenting the line

syslog = yes

If  you are running systemd you do NOT want the above setting! The problem is that systemd handles system logging through journalctl. You can follow the other suggestions in the Archwiki to run mcelog as a daemon (daemon = yes), but make sure the syslog lines are commented out. Also you need to specify an output log file for mce errors by uncommenting the following in /etc/mcelog/mcelog.conf:

logfile = /var/log/mcelog

Also uncomment the following in /etc/mcelog/mcelog.conf

run-credentials-user = root

Restart the mcelog service

systemctl restart mcelog

Next time a Machine Check Event occurs, it will be written to /var/log/mcelog. Here is some sample output:

Hardware event. This is not a software error.
MCE 0
CPU 0 BANK 5
MISC b8a0000086 ADDR ffb07500
TIME 1434034181 Thu Jun 11 23:49:41 2015
MCG status:
MCi status:
Error overflow
Uncorrected error
MCi_MISC register valid
MCi_ADDR register valid
Processor context corrupt
MCA: corrected filtering (some unreported errors in same region)
Generic CACHE Level-2 Generic Error
STATUS ee0000000040110a MCGSTATUS 0
MCGCAP c07 APICID 0 SOCKETID 0
CPUID Vendor Intel Family 6 Model 69
Hardware event. This is not a software error.
MCE 0
CPU 0 BANK 5
MISC 78a0000086 ADDR ffb07500

This seems to be indicating a memory error in the CPU cache.

Here is my /etc/mcelog/mcelog.conf file:

	# Example config file for mcelog
	# mcelog is the user space backend that decodes and process machine check events
	# (cpu hardware errors) reported by the CPU to the kernel
	#
	# general format
	#optionname = value
	# white space is not allowed in value currently, except at the end where it is dropped
	#
	# In general all command line options that are not commands work here.
	# See man mcelog or mcelog --help for a list.
	# e.g. to enable the --no-syslog option use
	#no-syslog = yes (or no to disable)
	# when the option has a argument
	#logfile = /tmp/logfile
	# below are the options which are not command line options.
	# Set CPU type for which mcelog decodes events:
	#cpu = type
	# For valid values for type please see mcelog --help.
	# If this value is set incorrectly the decoded output will be likely incorrect.
	# By default when this parameter is not set mcelog uses the CPU it is running on
	# on very new kernels the mcelog events reported by the kernel also carry
	# the CPU type which is used too when available and not overriden.
	# Enable daemon mode:
	daemon = yes
	# By default mcelog just processes the currently pending events and exits.
	# In daemon mode it will keep running as a daemon in the background and poll
	# the kernel for events and then decode them.
	# Filter out known broken events by default.
	filter = yes
	# Don't log memory errors individually.
	# They still get accounted if that is enabled.
	#filter-memory-errors = yes
	# output in undecoded raw format to be easier machine readable
	# (default is decoded).
	#raw = yes
	# Set CPU Mhz to decode uptime from time stamp counter (output
	# unreliable, not needed on new kernels which report the event time
	# directly. A lot of systems don't have a linear time stamp clock
	# and the output is wrong then.
	# Normally mcelog tries to figure out if it the TSC is reliable
	# and only uses the current frequency then.
	# Setting a frequency forces timestamp decoding.
	# This setting is obsolete with modern kernels which report the time
	# directly.
	#cpumhz = 1800.00
	# log output options
	# Log decoded machine checks in syslog (default stdout or syslog for daemon)
	#syslog = yes
	# Log decoded machine checks in syslog with error level
	#syslog-error = yes
	# Never log anything to syslog
	#no-syslog = yes
	# Append log output to logfile instead of stdout. Only when no syslog logging is active
	logfile = /var/log/mcelog
	# Use SMBIOS information to decode DIMMs (needs root).
	# This function is not recommended to use right now and generally not needed.
	# The exception is memdb prepopulation, which is configured separately below.
	#dmi = no
	# When in daemon mode run as this user after set up.
	# Note that the triggers will run as this user too.
	# Setting this to non root will mean that triggers cannot take some corrective
	# action, like offlining objects.
	run-credentials-user = root
	# group to run as daemon with
	# default to the group of the run-credentials-user
	#run-credentials-group = nobody
	[server]
	# user allowed to access client socket.
	# when set to * match any
	# root is always allowed to access.
	# default: root only
	client-user = root
	# group allowed to access mcelog
	# When no group is configured any group matches (but still user checking).
	# when set to * match any
	#client-group = root
	# Path to the unix socket for client<->server communication.
	# When no socket-path is configured the server will not start
	#socket-path = /var/run/mcelog-client
	# When mcelog starts it checks if a server is already running. This configures the timeout
	# for this check.
	#initial-ping-timeout = 2
	#
	[dimm]
	# Is the in memory DIMM error tracking enabled?
	# Only works on systems with integrated memory controller and
	# which are supported.
	# Only takes effect in daemon mode.
	dimm-tracking-enabled = yes
	# Use DMI information from the BIOS to prepopulate DIMM database.
	# Note this might not work with all BIOS and requires mcelog to run as root.
	# Alternative is to let mcelog create DIMM objects on demand.
	dmi-prepopulate = yes
	#
	# Execute these triggers when the rate of corrected or uncorrected
	# Errors per DIMM exceeds the threshold.
	# Note when the hardware does not report DIMMs this might also
	# be per channel.
	# The default of 10/24h is reasonable for server quality
	# DDR3 DIMMs as of 2009/10.
	#uc-error-trigger = dimm-error-trigger
	uc-error-threshold = 1 / 24h
	#ce-error-trigger = dimm-error-trigger
	ce-error-threshold = 10 / 24h
	[socket]
	# Enable memory error accounting per socket.
	socket-tracking-enabled = yes
	# Threshold and trigger for uncorrected memory errors on a socket.
	# mem-uc-error-trigger = socket-memory-error-trigger
	mem-uc-error-threshold = 100 / 24h
	# Trigger script for corrected memory errors on a socket.
	mem-ce-error-trigger = socket-memory-error-trigger
	# Threshold on when to trigger a correct error for the socket.
	mem-ce-error-threshold = 100 / 24h
	# Log socket error threshold explicitely?
	mem-ce-error-log = yes
	# Trigger script for uncorrected bus error events
	bus-uc-threshold-trigger = bus-error-trigger
	# Trigger script for uncorrected IOMCA erors
	iomca-threshold-trigger = iomca-error-trigger
	# Trigger script for other uncategorized errors
	unknown-threshold-trigger = unknown-error-trigger
	[cache]
	# Processing of cache error thresholds reported by Intel CPUs.
	cache-threshold-trigger = cache-error-trigger
	# Should cache threshold events be logged explicitely?
	cache-threshold-log = yes
	[page]
	# Memory error accouting per 4K memory page.
	# Threshold for the correct memory errors trigger script.
	memory-ce-threshold = 10 / 24h
	# Trigger script for corrected errors.
	# memory-ce-trigger = page-error-trigger
	# Should page threshold events be logged explicitely?
	memory-ce-log = yes
	# specify the internal action in mcelog to exceeding a page error threshold
	# this is done in addition to executing the trigger script if available
	# off no action
	# account only account errors
	# soft try to soft-offline page without killing any processes
	# This requires an uptodate kernel. Might not be successfull.
	# hard try to hard-offline page by killing processes
	# Requires an uptodate kernel. Might not be successfull.
	# soft-then-hard First try to soft offline, then try hard offlining
	#memory-ce-action = off|account|soft|hard|soft-then-hard
	memory-ce-action = soft
	[trigger]
	# Maximum number of running triggers
	children-max = 2
	# execute triggers in this directory
	directory = /etc/mcelog

view raw mcelog.conf hosted with ❤ by GitHub