• Old role: VM box for local infrastructure analytics
• New role: BSD router w/ CNAs for RoCE-v2 RDMA fabric.

Installed for the purpose: QLogic QL41234HLCU, and…

• NIC #1, QLogic QL41164HLRJ, 4x 10GbE BASE-T,
• NIC #2, Supermicro AOC-S40G-I2Q, 2x 40GbE QSFP+, Intel XL710
• NIC for Nodes: Qlogic QL41234HLCU, 4x 10/25GbE SFP28 w/ RDMA

Great, nice upgrades. The router will have its 4x 10GbE QL41164 ports connected in 802.3ad to the switch-fabric, with two VM host nodes direct-connected using SFP28 ports on the QL41234 NIC. These systems run lightweight validation suites for Project Coherent Flash (the full cluster lives in the colo) and therefore require RDMA and RoCE networking.

In other connectivity areas, the XL710 based NIC is for non-RDMA traffic to the nodes via direct non-switched connection. It does not participate in the memory fabric. In this mode, each QSFP+ 40GbE port has a quad-lane breakout cable going to 4x 10GbE SFP+ connectors, and each of those (8 total) offers 2x 10GbE to each of the four nodes in an adjacent cluster.

Just remember, on this board there’s a jumper to inspect and possibly flip, for reasons better described via the following nerdery.

the X11SSM-F bridges the slot SMBus (SMCLK/SMDAT on PCIe pins B5/B6) to the board SMBus via jumpers JI2C1/2. Some PCIe cards expose an I²C device on those lines at the same addresses used by DIMM SPD EEPROMs (0x50–0x57). That creates address conflicts, which can block SPD reads, hide a DIMM, scramble IPMI sensors, or hang POST. Disabling JI2C1/2 isolates the slot SMBus and removes the conflict.

So, where to begin… let’s see… we start with the POWER9 at home on my desk, and me looking a wreck in the morning…

Where is this wonderful place that brings me so much joy? Hurricane Electric of course.

For Silicon Valley datacenter nerds, Hurricane Electric’s Fremont and San Jose facilities are the spiritual home of budget-friendly colo: cabinet and partial-cabinet plans with same-day turn-up, straightforward BGP, and native v6 on a backbone that’s been aggressively peering since the early 2000s, making it a historically significant and still notably affordable place to park gear in the Bay Area.

And as such… it’s where most of my gear resides. What was once a HomeLab at home has become a situation of _“let’s move everything into the colo and tell PG$E to suck it on their $300-600/month apartment electric bills (which is physically impossible, they’re a proven CA Governor sponsored monopoly).

Tools and So Forth

How about a photo of my lovely Wera designed Datacenter tool bag? Yes! It’s rugged and strong and doesn’t distort with heavy items. Love it. At the top of the photo is whichever purse happened to be brought along that day; Kors or Burbs or Coach or whichever fit size requirements at the time.

Not shown: a squared-rasp-file occasionally required for “standardizing” the rail mounting holes in those old-ass blue racks. My right hand still has scars from years ago - before owning that rasp to fix the burrs.

Always Getting Injured, but that’s OK.

Now, about those old scars - there are surely new ones periodically. While getting ready for the colo, just standing at the sink with some sheers, cutting… cutting… and ohffs that’s my finger. Yep, it’s definitely deep! Apply pressure! Wrap it tight! Not too tight though… and what was at my disposal? Paper medical tape and clear plastic industrial packing tape.

FINE. It’s FINE. The bleeding mostly stopped, and off we go to the datacenter!

done cut my self real bad

Networking, Servers, Drives, DPUs, and all the other fun!

Ok, enough rambling. Here’s a photo gallery with annotated pics. Each rack has some major revisions to undergo, which requires planning and documentation updates and all the usual production readiness routines.

More Hardware - More ARM64!

Now you may wonder, where is all of my ARM64 equipment and the other clusters? Those are still in the 2x 25U racks at home – they’ll be migrated to the colo as well, once I convert them to use industrial Compact-PCI chassis blade mounts with integrated/redundant power and backplane. I bring you… the ADLINK cPCIS-2632R 3U Compact PCI chassis.

Industrial ADLINK cPCIS-2632R CompactPCI

SATA3… Really?

And we wrap up the day’s fun with… decisions on what to do with these SATA3 SSDs recovered from some old R420 systems… perhaps the LC922 boxes can use those + some 240GB Intel or Samsung boot drives. These days I never bother with SATA, only SAS or NVMe. HoHumBoringSataSoSlow.

A Tale of Two Labs - One is Real, The Other…

I love having my own hardware for my own labs. I love not having to rely on a corporation over-promising on lab resources for the architecture design process – real physical hardware which is absolutely required for “Proof of Concept Engineering”.

As a wise-one once related, not everything can be a Power Point Presentation. Engineers need physical hardware for validation workflows, otherwise it’s all just vaporware.

🙉 🙈 🙊

During standard weekends, when not afflicted with the covids, all of the following tasks could be completed in a relaxing and reasonable pace. Perhaps two or three.. time will tell. 😌

1. Remove all ephemeral hardware from 25U rack #2
2. Swap out 4U APC 3kVA UPS for 2U APC 1.5kVA, in 25U rack #2
3. Move ARM64 cluster from 12U rack to 25U rack #2
4. Swap 1x APC PDU (metered only) for APC PDU (switched), in 25U rack #1
5. Validate LACP map on EX3300-48T, rack #2
6. Validate 40mm fan situation on EX3300-48T, rack #2
7. Recable 3x 24 port patch panels, rack #2
8. Deploy 1x 24 port patch panels, rack #1
9. Replace all fans in 2U net-infra SmartOS node
10. Finish firmware updates on all APC devices
11. Upgrade firmware on all managed switches
12. Validate 5GbE SFP+/BASET optic for FTTH on X710 NIC
13. Install 3x SM863 SSD boot drives for Talos II Power9
14. Install 3x 883DCT SSD storage drives for Talos II Power9
15. Swap 2x10G Intel X550 for 4x25G QL41234 in Talos II
16. Swap 2x 120mm & add 1x 92mm fans in Talos II
17. Ident 1000W PSU for EPYC box, needs PMbus
18. ooooh there’s so much more…

rack consolidation strongly needed

Ehrlich gesagt, just look at this mess, and it’s on carpet? Even with solid 14 gauge copper grounding straps on every UPS, PDU, ATS, along with GFCI + AFCI isolation breakers, I cannot in good conscience recommend such blasphemy of electrical code compliance concerns.

So, add this to the list: find some manner of solid platform with a grounding-strap for the two 25U racks (12U needs to be elsewhere, byeeeee).

• 1x Nvidia Jetson Xavier NX 16GB
• 3x Nvidia Jetson Orin NX 16GB
• 3x GoWin R86S, N6005 16GB
• 2x Pi-KVM v4-mini for remote/headless access
• 2x NanoPi nodes, a RISC-V LeechiPi, and LattePanda Mu
• Associated networking for 10/5/2.5/1G ports

Setup in a Pelican Air case, ready for the flight!