Let's say you have three network cards plugged into your server's PCI expansion slots as well as four LOM (LAN On Motherboard) ports. The default port enumeration on the rear of the server is as follows:
LOM PCI
=== ================
[3] -[8 ]-[4]-[12]-
[2] -[9 ]-[5]-[13]-
[1] -[10]-[6]-[14]-
[0] -[11]-[7]-[15]-
=== ================
You would like to change the port enumeration and set up bonding channels as follows:
LOM PCI
=== ================
[15] -[0]-[4]-[8 ]- bond0: eth0,4
[14] -[1]-[5]-[9 ]- bond1: eth1,5
[13] -[2]-[6]-[10]- bond2: eth2,6
[12] -[3]-[7]-[11]- bond3: eth3,7
=== ================
In the days before RHEL 5.X (kernel 2.6.18.X) and RHEL 6.X (kernel 2.6.32.X) it was customary to explicitly assign network interface names (i.e. eth0, eth1, etc) to MAC addresses or UUID's in the files ifcfg-eth{0..N} in /etc/sysconfig/network-scripts/
For example, here is a sample ifcfg-eth0 that would work reliably pre-RHEL5.X/RHEL6.X:
DEVICE=eth0
ONBOOT=yes
BOOTPROTO=none
HWADDR= 00:1a:2b:3c:4d:5e
MASTER=bond0
SLAVE=yes
USERCTL=no
Regardless of the order in which network interfaces come up and make themselves known to the Linux kernel, since we have assigned the name eth0 to MAC 00:1a:2b:3c:4d:5e the interface will keep this name. If we had not explicitly mapped this HWADDR to eth0, it might for instance change to eth2 if it was the 3rd interface to come up on boot. Obviously port names that change with every boot would be a big headache.
Unfortunatelym explicitly mapping a MAC to a device name within ifcfg-ethX files is no longer guaranteed to work 100% of the time:
https://bugzilla.redhat.com/show_bug.cgi?id=491432
Solution
Because explicitly assigning HWADDR or UUID's to network device names within the files in /etc/sysconfig/network-scripts/ is not reliable, first we will comment out all lines starting with HWADDR or UUID within ifcfg-ethX files. Assuming there are 16 ports to be enumerated (0~15), you could achieve this with the following bash for-loop (assuming you are in the directory /etc/sysconfig/network-scripts):
for i in {0..15}; do
sed -i "s/HWADDR/#HWADDR/g" ifcfg-eth$i
sed -i "s/UUID/#UUID/g" ifcfg-eth$i
done
We will rely exclusively on /etc/udev/rules.d/70-persistent-net.rules to enumerate ports. The udev daemon reads from this file at boot to determine the device names for network ports. Within this file you can either map MAC addresses to ethX device names or PCI bus-info ID's to ethX device names.
Although it has historically been more common to use MAC addresses, I recommend using PCI bus-info ID's for device name mapping in 70-persistent-net.rules. When NIC's crash and you stop (start) the network with service network stop(start), two different device names using the same MAC can appear. For some reason, however, this same issue does not occur when using network cards' PCI bus-info ID's.
1. Find PCI bus-info ID's
You can find the PCI bus-info ID in several ways: from the kernel ring buffer with dmesg, from lspci or from ethtool -i. I will cover the latter two methods as they are easiest to parse.
# lspci -D | grep Solar
41:00.0 Ethernet controller: Solarflare Communications SFC9020 [Solarstorm]
41:00.1 Ethernet controller: Solarflare Communications SFC9020 [Solarstorm]
The -D flag makes sure that PCI bus numbers are printed. From man lscpi:
Always show PCI domain numbers. By default, lspci suppresses them on machines which have only domain 0.In the example above, I found the PCI bus-info ID's for two ports on a 10G Solarflare NIC. Since the first field from the above command will always show PCI bus-info, we can just print this field alone using awk `{print$1}`. We can print just the bus-info ID's for all Ethernet interfaces using the following commands:
lspci -D | grep -i ether | awk '{print$1}'
Note that the -i flag for grep above enables case-insensitive search.
On a machine with two 4-port NIC's plugged into PCI slots and with 3 onboard network ports the output of the above command might look something like the this:
0000:04:00.3
0000:04:00.2
0000:04:00.1
0000:04:00.0
0000:03:00.0
0000:03:00.1
0000:03:00.2
0000:03:00.3
0000:0d:04.2
0000:0d:04.1
0000:0d:04.0
You will notice that ports on the same NIC share the same PCI bus prefix, but differ only in the final number. Therefore we can deduce that 0000:04:00.{0..3} denote 4 ports on one particular PCI network card and that 0000:03:00.{0..3} denote 4 ports on a separate PCI NIC. Also notice that the 3 LOM (LAN On Motherboard) Ethernet ports have totally separate PCI bus-info ID's from the PCI NIC's. lspci | grep -i ether will print the NIC PCI bus ID's in order of network interface, so the first line would be eth0, second line eth1, and so on.
You can also obtain the PCI bus ID for each network port using ethtool -i ethX which returns the following fields:
driver:
version:
firmware-version:
bus-info:
supports-statistics:
supports-test:
supports-eeprom-access:
supports-register-dump:
supports-priv-flags:
Manually typing this command for each interface is not a good idea when you are working on a server with many network ports, so just use a bash for-loop one-liner:
for i in {0..n}; do ethtool -i eth$i; done
2. Map NIC port locations using ethtool -p ethX
In the problem statement at the beginning of this post, we assumed that you already knew the layout of device names to ports (i.e. eth0 is the bottom LOM port, eth12 is the top right-most NIC port). But if you're working on a brand-new server, how would you map out device name/port locations in the first place?
Thanks to ethtool -p ethX you can identify where each network port is located because this command will tell the port to flash it's status LED until the command is terminated. For the command to work, however, the network interface ethX must be up. To quickly put up all the Ethernet interfaces on a system, use the following bash for-loop one-liner (where N is the highest-numbered network port on your server):
for i in {0..N}; do ip link set eth$i up; done
It is better to use ip link set ... up from iproute2 rather than ifup from the venerable net-tools package because the first command is very fast; ifup is 8-10x slower because it tries to bind an IP to each interface when bringing it up, whereas ip link set ethX up simply activates the interface.
Once you have activated all Ethernet interfaces, you can use ethtool -p ethX , pencil and paper to write down the location of each port when it flashes. (Note that some LOM ports do not support ethtool -p)
3. Map PCI bus-info ID to network ifaces in 70-persistent-net.rules
The format for entries in /etc/udev/rules.d/70-persistent-net.rules for RHEL6.X using PCI bus ID's instead of MAC addresses is as follows:
SUBSYSTEM=="net",ACTION=="add",BUS=="pci",KERNEL=="eth*",KERNELS=="0000:01:00.0",NAME="ethX"
...
where ethX is eth0, eth1, etc.
In step 2, you should have found the PCI bus ID for each network port. To change the name of a given port, simply assign a different name to the line containing the relevant PCI bus ID number. The PCI bus-info ID should be entered between double quotes after KERNELS=
RHEL6.X initially generates /etc/udev/rules.d/70-persistent-net.rules automatically from the udev rule /lib/udev/rules.d/75-persistent-net-generator.rules but the network ifaces will not be nicely ordered in the file. By default this file will contain MAC addresses mapped to port names and will look something like this:
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="40:a8:f0:3b:a1:c6", ATTR{type}=="1", KERNEL=="eth*", NAME="eth6"$
$
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="40:a8:f0:3b:a1:c7", ATTR{type}=="1", KERNEL=="eth*", NAME="eth7"$
$
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="28:80:23:a2:d7:c0", ATTR{type}=="1", KERNEL=="eth*", NAME="eth0"$
$
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="40:a8:f0:3b:a1:c5", ATTR{type}=="1", KERNEL=="eth*", NAME="eth5"$
$
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="28:80:23:a2:d7:c2", ATTR{type}=="1", KERNEL=="eth*", NAME="eth2"$
$
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="28:80:23:a2:d7:c1", ATTR{type}=="1", KERNEL=="eth*", NAME="eth1"$
$
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="28:80:23:a2:d7:c3", ATTR{type}=="1", KERNEL=="eth*", NAME="eth3"$
$
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="40:a8:f0:3b:a1:c4", ATTR{type}=="1", KERNEL=="eth*", NAME="eth4"
This is inconvenient because eth6, not eth0, is on the first line, so when we change the port names, it is easy to get confused. It is apparent that the field separator character is comma "," and that the field we wish to sort in ascending order is field 7, "NAME". We can achieve this using the GNU coreutils program sort:
sort -t ',' -k7 -V 70-persistent-net.rules
where -t is the delimiter character, -k7 is for sorting based on the 7th column, and -V (--version-sort) sorts version numbers within text (of the form string+number). Without the -V flag, eth10, eth11 ... would follow eth1 instead of eth9.
If we now look at 70-persistent-net.rules we can see that the file is nicely sorted in ascending order of network ports:
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="28:80:23:a2:d7:c0", ATTR{type}=="1", KERNEL=="eth*", NAME="eth0"
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="28:80:23:a2:d7:c1", ATTR{type}=="1", KERNEL=="eth*", NAME="eth1"
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="28:80:23:a2:d7:c2", ATTR{type}=="1", KERNEL=="eth*", NAME="eth2"
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="28:80:23:a2:d7:c3", ATTR{type}=="1", KERNEL=="eth*", NAME="eth3"
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="40:a8:f0:3b:a1:c4", ATTR{type}=="1", KERNEL=="eth*", NAME="eth4"
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="40:a8:f0:3b:a1:c5", ATTR{type}=="1", KERNEL=="eth*", NAME="eth5"
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="40:a8:f0:3b:a1:c6", ATTR{type}=="1", KERNEL=="eth*", NAME="eth6"
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="40:a8:f0:3b:a1:c7", ATTR{type}=="1", KERNEL=="eth*", NAME="eth7"
Now you have to replace the fields DRIVERS, ATTR, ATTR with the fields BUS and KERNELS and then paste in PCI bus ID's you obtained earlier through lspci -D | grep -i ether or ethtool -i ethX.
For RHEL5.X, however, the key KERNELS is not recognized by udev. You must use the key ID instead (RHEL6.X understands both the keys KERNELS and ID, however):
SUBSYSTEM=="net",ACTION=="add",BUS=="pci",KERNEL=="eth*",ID=="0000:04:00.0",NAME="eth7"
SUBSYSTEM=="net",ACTION=="add",BUS=="pci",KERNEL=="eth*",ID=="0000:04:00.1",NAME="eth6"
SUBSYSTEM=="net",ACTION=="add",BUS=="pci",KERNEL=="eth*",ID=="0000:04:00.2",NAME="eth5"
SUBSYSTEM=="net",ACTION=="add",BUS=="pci",KERNEL=="eth*",ID=="0000:04:00.3",NAME="eth4"
SUBSYSTEM=="net",ACTION=="add",BUS=="pci",KERNEL=="eth*",ID=="0000:03:00.0",NAME="eth0"
SUBSYSTEM=="net",ACTION=="add",BUS=="pci",KERNEL=="eth*",ID=="0000:03:00.1",NAME="eth1"
SUBSYSTEM=="net",ACTION=="add",BUS=="pci",KERNEL=="eth*",ID=="0000:03:00.2",NAME="eth2"
SUBSYSTEM=="net",ACTION=="add",BUS=="pci",KERNEL=="eth*",ID=="0000:03:00.3",NAME="eth3"
(Note that the network ifaces above in 70-persistent-net.rules are not sorted anymore because we have customized the port enumerations.)
Also when creating a port enumeration file for RHEL/CentOS 5.X, beware of existing port enumeration files in /etc/udev/rules.d/
Sometimes a file named 60-net.rules exists in this path in lieu of 70-persistent-net.rules, so if another udev rule file enumerating ports exists, edit that existing file instead of creating a new 70-persistent-net.rules!
For some reason, ATCA blade servers (from Emerson Network Power, Adlink, etc) will accept a shortened format for 70-persistent-net.rules that only contains the fields KERNEL, ID (for RHEL5.X) and NAME as follows:
KERNEL=="eth*", ID=="0000:07:00.0", NAME="ethX"
Note that this format only seems to work for ATCA (Advanced Telecom Computing Architecture) hardware. If you try this on HP Proliant machines, for example, /var/log/messages will complain of invalid udev rules.
4. Apply changes
First shut down network services:
service network stop
Unload all network drivers (this step is not always necessary, but it is good to get into the habit; you can find the network drivers being used by each iface using ethtool -i ethX)
modprobe -r driverName (tg3, igp, bnx, ixgbe, e1000e etc)
...
Note that if your machine uses Solarflare 10G cards, you cannot just remove the driver sfc with modprobe -r or rmmod; You must use a special script called onload_tool provided by Solarflare. From the directory containing the Solarflare scripts:
./onload_tool unload (without any arguments)
To reload the sfc driver , simply replace unload with reload.
Unload bonding module (if you have bonding channels defined in /etc/sysconfig/network-scripts)
modprobe -r bonding
Reload the kernel modules you just removed
modprobe drivername
modprobe bonding
...
Reload udev rules to apply the changes in 70-persistent-net.rules
start_udev
(or udevadm control --reload-rules)
Start network services
service network start
Verify with ethtool -p ethX that your network port enumerations have taken effect.
댓글 없음:
댓글 쓰기