MLNX-OS

QSFP Splitting

QSFP Splitting

A nice feature with QSFP (Quad Small Form-factor Pluggable) is that you can turn a single QSFP port into 2 or 4 independent ports using a breakout cable.

In my home lab, I have an Nvidia/Mellanox SN2100 switch running Onyx (MLNX-OS), which I use for my iSCSI network. I also have an HPE MSA 2040 SAN. The HPE MSA 2040 SAN only has 10GB SFP (Small Form-factor Pluggable) ports.

I will use a breakout cable to use the SN2100 with the MSA 2040. The breakout cable will connect to the QSFP port on the SN2100 and give me 4 SFP connections that I will connect to the MSA 2040 SFP ports.

QSFP+ 40GB to SFP+ 10GB DAC cable

In this post, I will show you step-by-step how to split and unsplit an interface on an Nvidia/Mellanox SN2100 switch running Onyx (MLNX-OS).

Split Port

  • SSH into the switch.
  • Enter configuration mode with the command configure terminal
  • Shutdown the interface you want to split out.

The interface I will be splitting is ethernet 1/1. The command I will run is interface ethernet 1/1 shutdown.

  • To split the interface, we will use the command module-type qsfp-split-4 or the command module-type qsfp-split-2

Split 4 will split the QSFP module into four 25 Gbps ports. Split 2 will split the port into two 50 Gbps ports.

…

Read More Read More

Onyx (MLNX-OS) Text Config Export

Onyx (MLNX-OS) Text Config Export

By default, with Onyx (MLNX-OS), when you export the switch configuration, it is exported as a bin file rather than the regular text-based config files we are all used to. Fortunately, there’s a way to export the configuration as a text file.

In this post, I will show you step-by-step how to generate and export a text-based configuration file from a switch running Onyx (MLNX-OS).

The Process

  • SSH into the switch
  • Run the command enable to elevate yourself
  • Enter configuration mode with the command configure terminal

Before we can run the next command, we need to build it out. First, we tell the Onyx switch to generate a text-based version of the configuration. We will do that with the command configuration text generate

The following option tells the Onyx switch what to use to generate the text config file. We can use an already saved configuration file, or we can use the active configuration. I will use the active configuration. To do this, I will use the active parameter.

Now, we need to specify if we want to use the running config that could be unsaved or the saved version of the running config. If the running config isn’t saved, it’s not real, and I don’t care about it, so I will use the saved option.

Next, we decide if we want the configuration’s expanded version. The expanded version will list everything individually and not use ranges. I like ranges, so I won’t be using the expanded parameter.

Ranged config compared to expanded config

We also need to tell the Onyx switch we want to copy the generated text configuration file off of the switch. We will do that with the parameter upload.

…

Read More Read More

ONIE Partition Exists Fix

ONIE Partition Exists Fix

I recently ran into an issue where I had uninstalled Onyx on my switch and I was trying to reinstall it but couldn’t because I was getting an error. Below is an example of the error.

partition exists - please uninstall first
EXT4-fs (sda3): couldn't mount as ext3 due to feature incompatibilities
cp: write error: No space left on device

There is also another variation of this error that I ran into. Below is an example of that error too.

umount: can't unmount /mnt/onie-boot: Invalid argument
/mnt/onie-boot not mounted
Error: You requested a partition from 1000MiB to 2000MiB (sectors 2048000..4095999).
The closest location we can manage is 1000MiB to 1000MiB (sectors 2047999..2047999).

The issue seems to be that somewhere along the lines the NOS uninstall process left behind a partition or two. To fix this we need to delete them. In this post, I’ll show you step-by-step how to delete the leftover partitions with ONIE.

The Fix

This fix works for the Nvidia/Mellanox switches and will likely work on other ONIE switches however, I’ve only tested this on Nvidia/Mellanox switches.

  • Connect to ONIE. I’ll be using a console connection and PuTTY.
  • Enter GNU Parted for the disk by running the command parted /dev/sda

In my case the disk was sda it’s likely the same in most setups.

…

Read More Read More

ONIE and Onyx (MLNX-OS) Install

ONIE and Onyx (MLNX-OS) Install

Switches that support ONIE (Open Network Install Environment) are amazing switches because you can just change which NOS (Network Operating System) you are running relatively easily.

ONIE was created by Cumulus Networks in 2012. In 2020 Nvidia bought Cumulus just after purchasing Mellanox the year before.

I will detail step-by-step how to install ONIE and how to install the Onyx (MLNX-OS) NOS on the SN2410 switch. The process for other NOS and other switches should be similar.

Prerequisites

  • Compiled ONIE recovery image for your switch.

I need the one for Mellanox/Nvidia that file will have a name similar to this onie-recovery-x86_64-mlnx_x86-r0.iso

  • The NOS install file.

I’ll be installing Onyx, the Onyx install file will have a name similar to this one X86_64-3.9.3202-installer.bin if you google around you should be able to find it.

  • Console connection to the switch
  • USB drive
  • Network cable plugged into mgmt0 on a network with DHCP.
  • BIOS password if one is applied. Here’s how to reset the BIOS password for Onyx (MLNX-OS) switches. If the SSD in the switch has nothing on it then you can get by without the BIOS password.

Installing ONIE

  • Download the most recent version of Rufus.
  • Write the ONIE recovery image to the USB drive.

The default settings should be fine. This is what I used.

Rufus Settings
…

Read More Read More

Onyx (MLNX-OS) Upgrade

Onyx (MLNX-OS) Upgrade

In this post I will show you how to upgrade your switch running Onyx (MLNX-OS). I will detail how to do it via command line and via the web interface.

The best way to make sure your upgrade is a success is to plan your upgrade path. I recommend following the upgrade paths as sometimes jumping from an old version to the newest version isn’t supported and could lead to issues.

When Nvidia purchased Mellanox some of the upgrade path planning resources became locked away behind a login/pay wall. One of the items behind the wall are the detailed release notes.

The release notes are the documents that will tell you exactly which versions are supported on which switches and what your upgrade path should be as sometimes direct upgrades skipping version can cause issues.

You get access to the release notes on your switch directly but only for the version you are currently running. I tried to see if I could extract the release notes from the upgrade image but they seem to be stored as blobs and I couldn’t figure out a way to open them.

Fortunately Onyx is used by other OEMs like Hewlett Packard Enterprise for example. You can find a public version of the upgrade path the HPE documentation here.

My SN2410 switch was originally running version 3.6.6102 my upgrade path was the following

  • 3.7.1134
  • 3.8.2204
  • 3.9.1020
  • 3.9.3302
  • 3.10.2102
  • 3.10.4006

With the upgrade path planning out of the way I will now show you step-by-step how to upgrade Onyx (MLNX-OS) via CLI or the Web UI.

The CLI Way

  • SSH into the switch.
  • Run the show version command to see what version you are currently running.
…

Read More Read More

Onyx (MLNX-OS) BIOS Password Reset

Onyx (MLNX-OS) BIOS Password Reset

Recently I’ve been playing with some Nvidia/Mellanox switches specifically the SN2410. An issue that I ran into was that I didn’t have the BIOS password.

All of the documentation says that the default BIOS password is admin. That password did not work on my switch.

You can absolutely use the switch even without the BIOS password, however it may limit some of your options in the future specifically recovery options if needed.

The documentation that I found says that if the default password doesn’t work you need to contact Mellanox/Nvidia support. The issue with that is I’m playing with this switch in my home lab so I don’t exactly have a support contract to contact support.

I took the switch apart and there doesn’t appear to be a CMOS battery to pull to clear the BIOS either. There might be a jumper for it but it wasn’t obvious which one it might be and I didn’t want to risk breaking it.

Inside the SN2410 switch

In the end I ended up figuring out a way to reset the BIOS password back to default.

Here’s how to do it.

…

Read More Read More