Hierarchical map of key computing elements

Hierarchical map of key computing elements

Trying to optimize network performance by pinning applications to CPU cores with numactl requires a basic understanding of which cores are available and the locality of PCI devices, including network interfaces.

Portable Hardware Locality (hwloc) provides such information and more:

lstopo of Supermicro 5018A-TN4

Via google I found a good introduction article about hwloc on http://www.admin-magazine.com/HPC/: http://www.admin-magazine.com/HPC/Articles/hwloc-Which-Processor-Is-Running-Your-Service

Time to test the tool on a Supermicro 5018A-TN4 server with an Eigh-Core Intel Atom C2750 CPU with a max TDP of only 20W.

# apt-get -y install hwloc

Check out the hwloc(7) man page for a quick intro:

“hwloc provides command line tools and a C API to obtain the hierarchical map of key computing elements, such as: NUMA memory nodes, shared caches, processor sockets, processor cores, and processor ”threads“. hwloc also gathers various attributes such as cache and memory information, and is portable across a variety of different operating systems and platforms.”

The command lstopo(1) shows the topology of the system in various formats, from text over xml to png and pdf:

# lstopo
Machine (31GB)
  Socket L#0
    L2 L#0 (1024KB)
      L1d L#0 (24KB) + L1i L#0 (32KB) + Core L#0 + PU L#0 (P#0)
      L1d L#1 (24KB) + L1i L#1 (32KB) + Core L#1 + PU L#1 (P#1)
    L2 L#1 (1024KB)
      L1d L#2 (24KB) + L1i L#2 (32KB) + Core L#2 + PU L#2 (P#2)
      L1d L#3 (24KB) + L1i L#3 (32KB) + Core L#3 + PU L#3 (P#3)
    L2 L#2 (1024KB)
      L1d L#4 (24KB) + L1i L#4 (32KB) + Core L#4 + PU L#4 (P#4)
      L1d L#5 (24KB) + L1i L#5 (32KB) + Core L#5 + PU L#5 (P#5)
    L2 L#3 (1024KB)
      L1d L#6 (24KB) + L1i L#6 (32KB) + Core L#6 + PU L#6 (P#6)
      L1d L#7 (24KB) + L1i L#7 (32KB) + Core L#7 + PU L#7 (P#7)
  HostBridge L#0
    PCIBridge
      PCIBridge
        PCI 1a03:2000
          GPU L#0 "card0"
          GPU L#1 "controlD64"
    PCIBridge
      PCI 8086:10fb
        Net L#2 "p2p1"
      PCI 8086:10fb
        Net L#3 "p2p2"
    PCI 8086:1f41
      Net L#4 "eth2"
    PCI 8086:1f41
      Net L#5 "eth0"
    PCI 8086:1f41
      Net L#6 "eth1"
    PCI 8086:1f41
      Net L#7 "eth3"
    PCI 8086:1f22
    PCI 8086:1f32
      Block L#8 "sda"
      Block L#9 "sdb"

The lstopo picture at the beginning of this blog was generated with lstopo and selecting PDF or PNG as output format:

# lstopo --of pdf > c2750-lstopo.pdf

This is a single socket server, so all PCI devices are attached to socket #0. To find out which ethernet port is attached to which PCI address, the systool(1) command from the package sysfsutils comes in handy:

# apt-get -y install sysfsutils
# systool -c net
Class = "net"

  Class Device = "eth0"
    Device = "0000:00:14.1"

  Class Device = "eth1"
    Device = "0000:00:14.2"

  Class Device = "eth2"
    Device = "0000:00:14.0"

  Class Device = "eth3"
    Device = "0000:00:14.3"

  Class Device = "lo"

  Class Device = "p2p1"
    Device = "0000:04:00.0"

  Class Device = "p2p2"
    Device = "0000:04:00.1"

The command lspci(8) from the package pciutils shows a description of each PCI address in the server:

# apt-get -y install pciutils
# lspci
00:00.0 Host bridge: Intel Corporation Atom processor C2000 SoC Transaction Router (rev 02)
00:01.0 PCI bridge: Intel Corporation Atom processor C2000 PCIe Root Port 1 (rev 02)
00:02.0 PCI bridge: Intel Corporation Atom processor C2000 PCIe Root Port 2 (rev 02)
00:03.0 PCI bridge: Intel Corporation Atom processor C2000 PCIe Root Port 3 (rev 02)
00:0e.0 Host bridge: Intel Corporation Atom processor C2000 RAS (rev 02)
00:0f.0 IOMMU: Intel Corporation Atom processor C2000 RCEC (rev 02)
00:13.0 System peripheral: Intel Corporation Atom processor C2000 SMBus 2.0 (rev 02)
00:14.0 Ethernet controller: Intel Corporation Ethernet Connection I354 (rev 03)
00:14.1 Ethernet controller: Intel Corporation Ethernet Connection I354 (rev 03)
00:14.2 Ethernet controller: Intel Corporation Ethernet Connection I354 (rev 03)
00:14.3 Ethernet controller: Intel Corporation Ethernet Connection I354 (rev 03)
00:16.0 USB controller: Intel Corporation Atom processor C2000 USB Enhanced Host Controller (rev 02)
00:17.0 SATA controller: Intel Corporation Atom processor C2000 AHCI SATA2 Controller (rev 02)
00:18.0 SATA controller: Intel Corporation Atom processor C2000 AHCI SATA3 Controller (rev 02)
00:1f.0 ISA bridge: Intel Corporation Atom processor C2000 PCU (rev 02)
00:1f.3 SMBus: Intel Corporation Atom processor C2000 PCU SMBus (rev 02)
01:00.0 PCI bridge: ASPEED Technology, Inc. AST1150 PCI-to-PCI Bridge (rev 03)
02:00.0 VGA compatible controller: ASPEED Technology, Inc. ASPEED Graphics Family (rev 30)
03:00.0 USB controller: Renesas Technology Corp. uPD720201 USB 3.0 Host Controller (rev 03)
04:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
04:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Blog at WordPress.com.

Up ↑

%d bloggers like this: