Intel 10GBASE-T X520/X540 driver support?

[CentralWare]

I have a pair of SuperMicro network cards (10-gig 2-port) I'm trying to pair between two linux machines, the TCL box and a Free/TrueNAS box.
TrueNAS sees the cards without effort; TCL I'm uncertain whether we have the same driver support?
Intel's Link and Read Me for ixg* Base Driver (which I have not ventured toward yet! )
SourceForge Link for Intel e1000 which I believe includes X5xx

Code: [Select]

lspci
...
01:00.0 Ethernet controller: Intel Corporation Ethernet Controller 10-Gigabit X540-AT2 (rev 01)
01:00.1 Ethernet controller: Intel Corporation Ethernet Controller 10-Gigabit X540-AT2 (rev 01)
...
I've tried everything I could think of (including i2c-KERNEL) for everything firmware-intel we do support, any ideas of what I might have overlooked?

[Rich]

Hi CentralWare
Does  lsmod  show that the  ixgbe  driver is loaded?

I believe your card is  8086:1528:

Code: [Select]

tc@E310:~$ modinfo ixgbe
filename:       /lib/modules/4.19.10-tinycore/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko.gz
author:         Intel Corporation, <linux.nics@intel.com>
description:    Intel(R) 10 Gigabit PCI Express Network Driver
license:        GPL
parm:           debug:Debug level (0=none,...,16=all)
parm:           allow_unsupported_sfp:Allow unsupported and untested SFP+ modules on 82599-based adapters
parm:           max_vfs:Maximum number of virtual functions to allocate per physical function - default is zero and maximum value is 63. (Deprecated)
version:        5.1.0-k
 ----- Snip -----
alias:          pci:v00008086d00001528sv*sd*bc*sc*i*
 ----- Snip -----
srcversion:     47E398AEA90A36B7EE58800
depends:        mdio,ptp
intree:         Y
vermagic:       4.19.10-tinycore SMP mod_unload 486
tc@E310:~$
You could also check dmesg:

Code: [Select]

dmesg | grep ixgbe
If the driver was not loaded, try modprobing it first, maybe with
the drivers  debug  param.

It might also be useful to look at dmesg for your  Free/TrueNAS  box.
It might reveal a bootcode, firmware, or driver parameter being used.

[CentralWare]

@Rich,

Thanks!  This should be a fun mystery to figure out!

Does  lsmod  show that the  ixgbe  driver is loaded?

Yes, it is/was in fact doing exactly what it was supposed to be doing.
It's loaded, modprobe was unnecessary as dmesg showed me unplugging and reconnecting a cable...  so it's there...  and it says ETH3 was unplugged.

Here's the entire output from lspci:

Code: [Select]

00:00.0 Host bridge: Intel Corporation 4 Series Chipset DRAM Controller (rev 03)
00:01.0 PCI bridge: Intel Corporation 4 Series Chipset PCI Express Root Port (rev 03)
00:06.0 PCI bridge: Intel Corporation 4 Series Chipset PCI Express Root Port (rev 03)
00:1a.0 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #4
00:1a.1 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #5
00:1a.2 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #6
00:1a.7 USB controller: Intel Corporation 82801JI (ICH10 Family) USB2 EHCI Controller #2
00:1c.0 PCI bridge: Intel Corporation 82801JI (ICH10 Family) PCI Express Root Port 1
00:1c.3 PCI bridge: Intel Corporation 82801JI (ICH10 Family) PCI Express Root Port 4
00:1c.4 PCI bridge: Intel Corporation 82801JI (ICH10 Family) PCI Express Root Port 5
00:1c.5 PCI bridge: Intel Corporation 82801JI (ICH10 Family) PCI Express Root Port 6
00:1d.0 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #1
00:1d.1 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #2
00:1d.2 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #3
00:1d.7 USB controller: Intel Corporation 82801JI (ICH10 Family) USB2 EHCI Controller #1
00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 90)
00:1f.0 ISA bridge: Intel Corporation 82801JIR (ICH10R) LPC Interface Controller
00:1f.2 IDE interface: Intel Corporation 82801JI (ICH10 Family) 4 port SATA IDE Controller #1
00:1f.3 SMBus: Intel Corporation 82801JI (ICH10 Family) SMBus Controller
00:1f.5 IDE interface: Intel Corporation 82801JI (ICH10 Family) 2 port SATA IDE Controller #2
01:00.0 Ethernet controller: Intel Corporation Ethernet Controller 10-Gigabit X540-AT2 (rev 01)
01:00.1 Ethernet controller: Intel Corporation Ethernet Controller 10-Gigabit X540-AT2 (rev 01)
02:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Caicos PRO [Radeon HD 7450]
02:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Caicos HDMI Audio [Radeon HD 6450 / 7450/8450/8490 OEM / R5 230/235/235X OEM]
03:00.0 USB controller: VIA Technologies, Inc. VL805 USB 3.0 Host Controller (rev 01)
04:00.0 SATA controller: JMicron Technology Corp. JMB363 SATA/IDE Controller (rev 02)
04:00.1 IDE interface: JMicron Technology Corp. JMB363 SATA/IDE Controller (rev 02)
05:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 02)
06:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 02)
07:00.0 RAID bus controller: Silicon Image, Inc. SiI 3114 [SATALink/SATARaid] Serial ATA Controller (rev 02)

Here's where the fun starts...
1. As shown above, INTEL dual port is seen (twice, of course)
2. 05 and 06 are showing Realtek RTL 8xxx.  The motherboard has two onboard...  PCI32 #2 is also a two channel Realtek 8xxx.

ETH0/1 are onboard NICs bonded to 10.0.0.124/22
ETH2/3 are PCI32 NIC ports bonded to 10.0.4.1/22
Up to this point, everything worked exactly as expected...  then:
INTEL card was installed, anticipating an ETH4 and 5.  [Scratches Head, Instead]

If I pull either cable from ETH0/1 there's NO system message.  If I pull BOTH cables, I'm kicked from SSH (expected) but still no system message.
If I do the same with ETH2/3 there are system messages; connectivity yet to be tested as it's a different network than this workstation has immediate access to.
If I pull either cable from INTEL there are system messages using the identifiers (ETH2/3) which the PCI32 card had been initially assigned.

I'm guessing there's an identifier overlap going on, but it doesn't explain why lspci doesn't list BOTH Realtek networks - which may be the cause of the overlap.
If memory serves, there's an app "ifrename" or similar which given a MAC address allows the identifier to be rewritten to MickeyMouse0 instead of ETH0.  If I pull one of the cards and get MACs from the cards accordingly, that should band-aid the problem, but I'm intrigued what the cause might be!

AMMENDMENT:
As suggested, the TrueNAS box was looked into (yet to look into boot codes) but half-way down (6 seconds into boot) it does what I mentioned above...
renames ETHx to ENP2S0F0/F1.  It also shows me pulling the cable a few times for testing the TCL box.

Code: [Select]

root@TrueNAS1[~]# dmesg | grep ixg
[    1.810957] ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver
[    1.811086] ixgbe: Copyright (c) 1999-2016 Intel Corporation.
[    2.091115] ixgbe 0000:02:00.0: Multiqueue Enabled: Rx Queue count = 8, Tx Queue count = 8 XDP Queue count = 0
[    2.175922] ixgbe 0000:02:00.0: 32.000 Gb/s available PCIe bandwidth (5.0 GT/s PCIe x8 link)
[    2.200365] ixgbe 0000:02:00.0: MAC: 3, PHY: 0, PBA No: 030000-000
[    2.200491] ixgbe 0000:02:00.0: 0c:c4:7a:58:e2:22
[    2.348132] ixgbe 0000:02:00.0: Intel(R) 10 Gigabit Network Connection
[    2.631088] ixgbe 0000:02:00.1: Multiqueue Enabled: Rx Queue count = 8, Tx Queue count = 8 XDP Queue count = 0
[    2.717675] ixgbe 0000:02:00.1: 32.000 Gb/s available PCIe bandwidth (5.0 GT/s PCIe x8 link)
[    2.742522] ixgbe 0000:02:00.1: MAC: 3, PHY: 0, PBA No: 030000-000
[    2.742654] ixgbe 0000:02:00.1: 0c:c4:7a:58:e2:23
[    2.894423] ixgbe 0000:02:00.1: Intel(R) 10 Gigabit Network Connection
[    6.083030] ixgbe 0000:02:00.0 enp2s0f0: renamed from eth2
[    6.115132] ixgbe 0000:02:00.1 enp2s0f1: renamed from eth4
[   46.175464] ixgbe 0000:02:00.1: registered PHC device on enp2s0f1
[   46.488261] ixgbe 0000:02:00.0: registered PHC device on enp2s0f0
[   50.814429] ixgbe 0000:02:00.1 enp2s0f1: NIC Link is Up 10 Gbps, Flow Control: RX/TX
[  239.779626] ixgbe 0000:02:00.1 enp2s0f1: NIC Link is Down
[  282.429763] ixgbe 0000:02:00.1 enp2s0f1: NIC Link is Up 10 Gbps, Flow Control: None
[  403.601818] ixgbe 0000:02:00.1 enp2s0f1: NIC Link is Down
[  408.378783] ixgbe 0000:02:00.1 enp2s0f1: NIC Link is Up 10 Gbps, Flow Control: None
[  467.311084] ixgbe 0000:02:00.1 enp2s0f1: NIC Link is Down
[  471.657811] ixgbe 0000:02:00.1 enp2s0f1: NIC Link is Up 10 Gbps, Flow Control: RX/TX

AMMENDMENT:

WHAT?  IfRename isn't a part of ETHtools?  It's WIFI???
tce-ab...  find ifrename...  install wireless...  copy ifrename binary to /opt...  uninstall wireless/wireless-KERNEL...  all this for a 23kB file

[Rich]

Hi CentralWare
Maybe your  Free/TrueNAS  box is using  udev  to rename
the network device:
https://www.baeldung.com/linux/rename-network-interface#permanent-renaming-network-interface-using-udev-rules

[CentralWare]

Quite possible, especially considering how early in the boot process they get named.  I'll add it to my research list!

UPDATE:
After playing musical "cards" I was able to gather the six MACs needed to make this recipe.
I changed out both of the dual cards with identical models JUST to make sure we weren't dealing with a hardware oddity -- same results.
During the swaps, MACs were recorded while they were "reliably" available.

ifrenme -p probes drivers; this is a "just in case."
-t (takeover) is likely more for when this patch gets added to our tc-config

We then create our config file /etc/iftab where we "name" our "eth#" interfaces with our MickeyMouse# replacements:

Code: [Select]

mickey0  mac AA:BB:CC:DD:EE:F1
mickey1  mac AA:BB:CC:DD:EE:F2
mickey2  mac AA:BB:CC:DD:EE:F3
mickey3  mac AA:BB:CC:DD:EE:F4
mickey4  mac AA:BB:CC:DD:EE:F5
mickey5  mac AA:BB:CC:DD:EE:F6

Afterward, in bootsync we launch ifrename (with no parameters) which does the heavy lifting making our MickeyMouse names a reality.
Finally, in bootlocal we can revert Mickey## back into ETH## by adding:
ifrename -i mickey0 -n eth0
(One line for each interface.)

For sake of covering all bases, it's probably best to ifconfig eth# down after probing, but before reading the config as I remember reading somewhere that rename will choke on an active device.

Now for the WHY...
My "best guess" was decided when I rebooted the server and took a photo of the IRQ assignments BIOS had designated.
The onboard NICs were assigned to 11 and 15.
The INTEL card... ALSO assigned to 15 and 11 (just in reverse order.)
The PCI32 RTL8211 card was assigned 3 and 10.
lspci must have been seeing the PCI32 card and the INTEL card was likely the last device read on those IRQs possibly disregarding the Realtek that was already on the same IRQs.  Hooks assigned for messages from each were also likely given to the "last mac standing."  How ETH2/ETH3 were "re-purposed" onto the INTEL leads to speculation...  it's almost midnight, so I'll leave that for another day!

UPDATE:
MickeyMouse# above worked nicely.
Will look into adding it to uDev tomorrow once I read up on TC's take/rules/etc. on uDev.
Quite Redundant ( eth0 --> mickey0 --> eth0 ) but it gets us past the glitch!

Code: [Select]

tc@TinyNAS1:~$ sudo ethtool eth5
Settings for eth5:
        Supported ports: [ TP ]
        Supported link modes:   100baseT/Full
                                1000baseT/Full
                                10000baseT/Full


[Rich]

Hi CentralWare
If you go the  udev  route, you can trigger it in  bootsync  with:

Code: [Select]

udevadm trigger --type=devices --action=add

[CentralWare]

just tidying up the new udev rule and rebooting...

• Need to determine WHEN the rule must be in /etc/udev in order to be seen prior to devadm re-triggering the list  (Just adding the rule to /etc/udev/rules saved in mydata.tgz doesn't do it early enough.)
• Need to adjust filters to ensure they cover the different hardware (IDs especially) or leave those out
• Need a refill...  coffee pot empty...

Update:
Assuming all interfaces are down when bootsync fires, udevadm trigger (-v) doesn't look as though it sees or cares about the ADD rules, implemented after extraction of mydata.tgz
Then I came across this in a udev manpage reference:

[NAME]
The name to use for a network interface. See systemd.link(5) for a higher-level mechanism for setting the interface name. The name of a device node cannot be changed by udev, only additional symlinks can be created.

Of course, this doesn't mean under certain environments it "cannot" be done; to be universal they likely meant "should not - we won't respond to bug reports if you do!"

All in all, it looks like networking names/rules need to be added to tc-config to make this boot friendly (during hot-swap initialization) OR through ifRename during bootsync/bootlocal.  BUT...  it works!
Turns out two network cards/four network ports, the onboard sound card (which I thought was disabled in BIOS?!) AND PEG2 video are all using the same interrupts...  kernel must love this machine!

[patrikg]

Hello, just a thought... sorry for get more confuse..in my arch dist setup i just add the
kernel command line option, to say that i don't wan't these micke mouse names.
I don't have more then one Ethernet interface, so it is safe... for my part.

Here you have the arg to the kernel command line.

Code: [Select]

net.ifnames=0
Don't know if this is just for systemd or i this is for another init systems or this is a kernel parameter.
I think if you have lot's of interfaces, the best option will be to just let udev do the thing, and like you said get the correct parameters from bios

To know what interface is what.

And then talking about interfaces, I also disable ipv6 in the kernel command line with:

Code: [Select]

ipv6.disable=1

[CentralWare]

mornin' @PatrikG!

ipv6 -- things here for the TCL server turned out to be dependent on v6 even though we don't have USE for it, thus turning that one off may crash our kernel bonding driver.  For most home networks and small business...  the need for v6 is unheard of, so it should be safe otherwise.

ifnames=false/0 I'm thinking is a bypass switch for those who have custom rules set up.
For example, let's say you had two entries in your Arch boot menu where the first one used wireless and the second one used ethernet, but the software running on the machine expects "eth0"...  you could create a rule to rename wlan0 to eth0 and in doing so, the system doesn't know the difference between it being wired...  or on wifi.  For the wired version, you'd use the kernel switch to disregard naming rules.  That's my take anyway, without digging through the manual.

[Rich]

Hi CentralWare

... Need to determine WHEN the rule must be in /etc/udev in order to be seen prior to devadm re-triggering the list  (Just adding the rule to /etc/udev/rules saved in mydata.tgz doesn't do it early enough.)

 ----- Snip -----

Update:
Assuming all interfaces are down when bootsync fires, udevadm trigger (-v) doesn't look as though it sees or cares about the ADD rules, implemented after extraction of mydata.tgz ...

Try making your rules look like this:

Code: [Select]

SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="08:00:27:2e:7a:49", ATTR{type}=="1", NAME="ethN"
If you really need the rules to show up earlier, I think you
could achieve that with a second small  initrd  file:

Code: [Select]

mkdir -p tempdir/etc/udev/rules.d
cd tempdir
cp Path/To/70-persistent-net.rules etc/udev/rules.d/
sudo find . | sudo cpio -o -H newc | gzip > /path/to/new/udevfs.gz
You should then be able to add it to your bootloaders config file
after  corepure64.gz.  Depending on your bootloader, you need
to separate the two initrd filenames with a space or a comma.