92d+ Uptime on Custom Gentoo Stage4 LiveISO

And that was a decent go on this build; one simply avoids ever using GCC and SystemD – use LLVM/Clang + OpenRC, and.. wow.. it’s like 2006 when things worked, Cobalt RaQ was still in the colo by the grids, an E10K or a few hundreds were still in service and would continue to stay online for years to come.

Oh, and admins and engineers didn’t avoid UNIX tools or determine ifconfig or netstat had to be "deprecated" (please, fuck off). So, get rid of the trash and then the systems become substantially more stable on the daily. Personally I trust my own ability to run and automate fleets of systems, not a tangled mess of low-hanging-fruit nod-to-redmond that most linux distros are suffered to use (CVE town is so rad, ja?)

92 days of uptime is generally close to mean value, and unfortunately it was ruined by having to upgrade the BMC and BIOS firmware versions due to “reasons”. Everything is being upgraded just in case because of the Github CVE in May. It’s wrecked so many upstream tools and sites and services and so many people are scrambling as if Log4J had just shit the bed and fell back asleep. But don’t worry.. “no customers were impacted” 😂

Anyway, Resuming to Workstations and Stage4

While there have been improvement on recompiles when optimizing test nodes’ kernels and enabling accelerators, the CI system for the building stages was upgraded to enable portage to use cross-datacenter distcc (distributed compiling) over a decent 5GbE symmetric fiber from a lab-rack to the colo cross-connect. Then the jobs are spooled off to SLURM nodes, rebuilding thin-LTO, PaX, PAE, ASLR. Very similar to a distributed Poudtierer, but without the memories of Poutine (and I love a good Poutine, especially in a light snowstorm in Banff, at least as much as the Chinese restaurant on the first left into town).

How many times do I need to follow this type of flowchart (made it, wow, so amazing) every time Supermicro releases a new BMC or BIOS upgrade? Dell, you’re not much better but at least the release notes are decent and there’s the Yum/Deb repo (one of these days I’ll put my Gentoo + Dell Driver overlay into a public git repo (not on git/msft-hub.com) for other Gentoo PowerEdge users to more easily remain up-to-date on firmware packages (fwupdmguur does’t cover everything, and the last time I trust it not to shit the bed it bricked my Thinkpad X1 Nano - thanks brosefs (am I saying that right? ehh).

fuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuu

The result of thoughtless planning is not freedom from consequences; it is becoming trapped by consequences.

As koans go this is one of my favorites, which perhaps stems from having trained mercelessly year over year in order to not simply do well in my formerly sponsored racing career, but to be capable of placing-podium at nationals for “24HR NORBA Cross-country Championship Series”. Before we get to that, some interjections are due.

Self-Defense in the Era of Endlessness

Twenty-ish years ago life was very different; I had not yet needed protection from a stalker, not needed multiple legal name changes, not needed to move every 1-2 years, not needed to take an FBI instructor course in “live-fire self-defense”, or needed to maintain corporate attire compatible with concealed-carry. There are many reasons for my not wanting to move back to CA, but daily life with hypervigilance in this state is simply a fact that’s not able to be debated. SF was my home, or at least where I felt most at home since the early 90s; and they let it fall into ruin - but politics are not the point here.

I “Respect The Chemistry” just as much as I “Respect The Bullet”; this implies that the weapon is always safety-locked and loaded, with a live round in the chamber. If you need to pull your side-arm it must be ready to fire and you must be ready to pull the trigger and accept the consequences. End of Story. When one fails to respect the bullet or the science then bad things happen: people make mistakes, reactors melt down, and people die.

Excuse my language, but reality exists and so do evil people who harm others on purpose. I never wanted to flag down a taxi full of post-clubbing-party people in the depths of The Mission at 3am, politely demanding through a partially opened passenger window for “Anyone! Someone! Call 911!”. I was also not a 2A advocate until I was held at gunpoint at 3am, but that is life.

PTSD Wrecks; Maybe More PTSD Helps? Nope.

Stripped of everything, robbed, wandering down 16th in complete system shock, vision and logic scattering from adrenaline, remembering the cold steel barrel he pressed against my back, then the middle of my sternum - then placed hard to the back of my skull while being told to “walk away and don’t look back or we will find you again”. They had my ID, cell, keys, and everything else*.* One of the two men held my arms locked behind my back, while the other held the gun with one hand. When finally, it was over… but it’s never been over, and it will never be over, the memory never goes away. Trauma does not go away.

Nearly half (45%) of women in the United States have experienced sexual violence in their lifetimes(*). This includes 21% who reported rape, 20.3% who reported sexual coercion, and 39% who reported unwanted sexual contact. [*] CDC’s National Intimate Partner and Sexual Violence Survey.

The police never found the two men, but I was told that on this particular night I was not their only victim. “Fuck San Francisco” became the refrain, and some months later I moved to Colorado and worked remotely. My first purchase in Colorado was a 12g pump shotgun, then a Ruger SP101 snubnose .357 revolver, then a beautiful hunting rifle in .300 WSM, and then began amassing a collection of ammunition, and of course the required licenses for training and reciprocity-based interstate carry. Any time I left the house I considered one thing above all else..

“Was today a .357 snub-nose type of day, or a Glock 27 sub-compact .40cal type of day? hm, maybe the HK, or the Colt Mustang II.. “

This routine for leaving the house was impacted when I was forced to short-sell after the 2008 financial crisis; the house where I raised twelve huskies, the house where I kept one pistol-grip pump shotgun under my side of the bed (stagger-loaded, alternating 00-buck with rifled slugs). If someone hears a 12 guage pump being racked in otherwise turgid silence of a potential B&E, that sound says one thing “WRONG HOUSE. YOU SHOULD LEAVE NOW”. If the reader ever wants to know if a 12g gauge slug can blast a hole through two walls and still kill someone, then go look at ballistics-gel penetration testing videos on youtube.

Back to that Koan, Back to the Races

Some of my 24hr race events were arranged for 2-4 rider team-relay, some were solo, and then sometimes co-ed teams (rarely). Planning, purpose, goals, dedication, and endurance. Harder, Faster, Never Stop.

My cross-country 10-20mile lap times were in the Top-3 consistently, though there were so very few women racing at my level in the 90s and early 2000s, with some 1st-finisher grabs on my favorite solo courses.

This was, back in my day, kind of a big fucking deal; it was and it still is for those who engage in 24hr mountain races. It is not an endeavor to enter into lightly. One tends to have 2-3 race bikes, at least one ultra-light road bike for hill-climb training days (each ~$5-10K USD circa the millenium $/value), own multiple wheelsets, and be able to resolve technical issues while on-trail after the inevitable crash occurs (especially after going head-over-bars into the darkness during nighttime training rides and maybe breaking components or twisting/snapping titanium spokes).

Then there are the years of experience leading up to the point where one is capable of enduring the required amount of caloric burn on a regular basis; and then more so for twenty-four hours at race-pace. For my metabolic system in the early-20s through mid-30s, this was 1,250 Kcal/hr equivalent to 178-188 BPM peaks and 168-174 average BPM heart-rate.

All of these expensive aspects had to be obtained via ‘pro-deal’ discount by working in the industry - and so I did that, and worked in a local bike shop during most of my college years, and raced, and trained, and raced, and also lots of skiing, and laying out on one or another of the beautiful beaches around the East Shore of Lake Tahoe… tales for another day.

I was not racing anyone but myself, torturously wanting to outlast and surpass all of those memories, the events, the people, everything. There is a type of focus in ultra-endurance which makes that possible - until you stop, and then the reality of living with PTSD almost immediately returns.

When I was not training or racing I was attempting to rest, recouperate, and that modality never stopped - except for when I temporarily died a decade later.

Hypertrophy & Catabolism for Fun and No-Profit

This kind of fitness is hard anaerobic territory which induces catabolic muscle wasting. When the body cannot access sufficient oxygen and sufficient reserve fuel resources it begins to burn muscle protein as fuel.

Least of one’s worries being body fat (“what body fat?”, back then I was cranking hills while leaned-out at 12-10% BF during training season, and leaning out to 8-6% at the end of a race). The body does not bother with reproductive functions at that stage, which was mostly irrelevant due to my genetics - though I did not yet know the full story on that front at the time. Too many chromosomes? Too few? Some manner of extremely rare condition? Who cares.. just go faster. Never stop going faster.

Dipping below 8% generally induces a state where organ tissue starts to be negatively impacted - the liver is hit especially hard, and is metabolically consumed. This survival mechanism has a hard shutdown limit. Going below 5% often leads to rapid cardiac failure, heat-induced seizure from elecrolytic deficiency, extreme dehydration, and eventually cellular aptosis causing brain death.

Fun! Not really, but it’s fascinating? Yes, the things we do and learn when we’re young and don’t care about risks, or older and don’t care about risks, or are simply too reliant on fitness to prioritize off-season rest (that’s what Cyclo-Cross and Telemark are for, right? I won some of those races as well.. and my knees and neck hate me in my 40s). Besides, how can there be “off-season rest” if there is never an off-season? Indeed.

Harder, Faster, Longer, and Never Stop

This mantra has stayed with me for as long as I can remember. One of my earlier memories of my father involves a mostly one-way conversation focused on me from his eyes to mine, with words sourced from deep within our shared-well of frustration and shared love for existence..

You don’t know when to stop, do you? When will it be enough?

I couldn’t have been older than ten years, so whatever it was which spurred such authority postured statement-questioning, there was only ever going to be a single answer.

Whenever I want it to be enough.

I have never lived half-assed, never taken the easy way out, and never avoided pain out of a sense of fear. Occasionally I sought it out, the things which caused physical pain - not for reasons of self-harm, but out of a mixture of first-person experiential curiosity mixed with the desperation to feel something, anything, to remind me that I was alive.

The events of my childhood were not always pleasant, and I did not always tell those of whom who should have known - what all was happening, or being done to me. At those ages the mind is still developing concepts of personal boundaries, emotional limits, self-awareness, social concepts of ‘right and wrong’, and cognitive capacity for the internal dialog and sense-of-self.

The mind is maleable and easily led towards fraught conclusions of okay-ness. I was not ok, but I stayed silent, because no one ever truly listens - that was the lesson.

What is This About Ten Percent Body Fat?

What does 10% look like, you may wonder? Here’s what 10% looks like from my mid-30s. Some pics are post-gym session. some are post-3hr rowing, some are after a nice 3-4hr ride in the mountains.

After I failed to have a child in my first marriage, and after DV ended it (no one hits me a second time or the ring is removed forever).. I left the mountains again and moved back to San Francisco to focus on my career. Arriving under-weight, abused, miserable, and lonely, all I did was ride and work. Raining? Ride. Fog in the Marin Headlands? Ride. Insomnia at 2am during wind-sheering on the Golden Gate bridge… RIDE. Can’t ride? Gym time, smash weights.

None of that mattered really. Nothing matters when your twin dies in front of you, and before the year was over I was in a coma, hooked up to life-support machines.

Then I re-learned how to speak, to walk, to feel emotions, and how to love all over again… but it took a long time.

4hrs of riding or 4hrs of rowing - oppositional muscle groups that don’t make you faster at either fitness obsession

That’s Exceedingly Sad, Why Discuss This?

Sadness is an emotion, a reaction, a biochemical response to sensory perception, nothing more. Memory of traumatic events is often inherently unreliable due to the state of shock, and the mechanics of neural processing of sensory inputs .. well.. if you care to read more about that, the associated document is very detailed.

Memory consolidation processes in the striatum go through a sensory filtering process with multiple layers of modulation of input.

A Knee Injury Induced Course-Correction

When I was thirty seven, no longer racing mountain or road, I switched to solo-crew (despite not liking boats). I had to keep doing.. something physically difficult, something challenging, something to keep the blood pumping.

When training I loved seated rows, mid-rows, reverse-tricep pulls, trapezius everything – vertical and 45-degree pull-downs, and everything else which inevitably results in striated delts and eight-pack abs. Give me a cable machine with ball-ended-ropes, a sit-down water-rower, and a set of 5-25lb dumbbells in 2.5lb increments and I was the happiest I’ve ever been in a gym. I never bothered with “max weight” or “3x3 sets”, that’s better for bulking. Instead, it was superset-burndowns for muscle group rotations, with 24-50 reps per set, with core crunches in the hundreds at a time.

No one cares though. Not really, unless it’s to tell women that muscles are for men and “why do you want to do that anyway?”. Because it feels good, and when it’s done correctly it’s healthy.

I can never claim to be “natty”, not because I love steroids but because my endocrine system is missing parts entirely on a physical level. Everything has to be prescribed and supplemented - everything. One month without all of it and I am not alive; and not by choice, but by the simple medical reality that I should never have made it to adulthood. I have been broken since birth and I am fully aware of the implications with which I’ve had to govern my medical life, my tangible existence, and the extreme cost of being required to do so.

Why? Because “women who can’t conceive are irrelevant”, and because “I am not a profitable healthcare patient in America”. Those two operating principles are primary reasons for constantly leaving the country and never wanting to return. Every time I’m back here it’s a matter of 2-5 years before the medical industry fails to sustain my “unique neuro-endocrine needs”. Crossing the border just to get the medication that my body should be able to make on its own; the resources that so very many people don’t even know the names of - and take it for granted that their autonomic systems keep the’ol heart beating and lungs breathing. Please don’t take those systems for granted.

None of it matters because I don’t have a choice. I can make the most of it and whenever I can, I go as fast and as hard as possible until nothing is left. That is my life well-lived. It’s taken me to the summits of countless peaks, down nearly every double-black run in the Sierra and Rockies on telemark skis, and into the wilderness depths of peaks and valleys in winter and summer to simply leave humanity to its unlimited amount of problems which I am not able to solve. That last part is challenging, because I love solving complex problems (ref: career, neurology, physiology, genetics); another aspect I’ve had to adapt to.

What matters is that I keep breathing and keep going. So that’s what I do, and I guess most of the time I’m pretty decent at it, or so it goes.

On Self-Awareness of Limits

Training for 24hr mountain bike races involved 4-6 days/wk with sunrise-morning rides on the mountain passes involved with each summer’s 120+mi thigh & lung burning electrolye depleter known as ”The Death Ride - California Alps”, interspersed with long hours in the saddle at night, riding sections of the "Tahoe Rim Trail", almost always solo, and occasionally "The Downieville Cross-Country Classic" for good measure.

You cannot simply ride a lot and win. Understanding the mechanics involved with one’s metabolism, endocrine modulation, internal cycles required for mental and physical rest & recovery, a combination of immediate + long-term + short-term project planning, and the ability to be intimately familar with ones maximal-limits and failure-modes: all of which requires metrics collection and data-analysis for VO2 capacity, hold-limits for heart-rate, patterened respiration to optimize training for aerobic and anaerobic ability, tracking blood glucose levels, wearing a chest-strap for 24/7 ECG, the list continues.

Success in the ultra-endurance world is a mixture of genetics, mental and physical capacity planning, self-actualization, self-awareness, dopaminergic drive + desire + reward system modulation, and clear-headed awareness which informs all forward movement and progress toward a sustained and unrelenting goal. Usually these qualities are found at the Olympics, but my genetics would have banned me from global competition even back in the 90s when I was actively competing, for my blood carries two sets of 46 (the curious may search for this), and for this reason I must supplement everything, and this medical reality is a blocker not to regional, or state, or local - but WADA is another story.

So, inescapable maladies and compensatory treatments aside, my genetics were perfect for that kind of racing, and I happily made the most of those years. I missed that time of my life severely after moving from Tahoe to San Francisco to start my career in tech. I kept riding, but not racing.

My race against time would come sooner than I knew, sooner than anyone could know at the time. It was an inevitability of consequence.

An Evolved Definition of Leadership

Over the years I stopped racing. I stopped riding solo nearly all of the time in favor of riding double, accepting more invitations to group rides (working at FitBit was interesting, but I expected more fitness obsessed people like myself).. and I remembered that leadership is not limited to “Those Operating Above”, nor is leadership well-defined by situations involving “Topping from The Bottom”.

Leadership is about shared awareness and shared problem solving. Leadership does not occur in a vacuum and it certainly does not occur out of happenstance. Leadership is a choice, insofar as Free Will is anything more than a convenient explanation for guilt and sadness: blame must go somewhere, yes? Not always. Sometimes shit happens and there was nothing that anyone could have done differently.

Successful Leaders Define, Design, and Drive Shared Goals

The future is not ours to inherit, but to build together. Life advances not through ease, but from asking better questions;
Ones which embrace complexity, challenge assumptions, and strive for success. Visionary leaders build the future; Our Future.

The Inevitable Return to Consequences

So, back to the Koan. Hyakujō’s Fox inevitably realizes the lesson.

A former head monk says that an enlightened person does not fall under causation. Because of that wrong answer, he says he has endured five hundred rebirths as a fox. Hyakujō’s corrective answer is not that enlightenment escapes causality, but that one “does not ignore causation.”

For leadership, the reading is direct: a leader who believes they are above cause and effect—above feedback, incentives, dependencies, timing, or consequences—has already become the fox.

The result of thoughtless planning is not freedom from consequences; it is becoming trapped by consequences.

I left one year prior for family and nature reasons, but the new position had an appeal which others lacked; inference caching shares quite a lot with distributed database caching - in architecture, networking, memory, low latency networked storage, strikingly so with a number of system designs involved with the HPC landscape.

Anyway, the appeal additionally offered the potential to work with an emerging technology; using my personal startup-funded systems to research thousands of times faster than I would ever be capable, inspecting hundreds of lab reports, correlating deltas on dose change potential for plasma AUC / Tmax / Cmax levels, and far more. These systems had already helped solve a critical medical concern which no neurologist or endocrinologist could answer over many years.

So, we moved. The move could have gone better, and we floated for the first two weeks. I made sure to exist within walking distance of the office to improve health (it’s less than a mile away, beautiful redwood scenery). Staying in the area was helpful.

Labs, Everywhere Labs - Except Here

My roles in systems engineering have either required, as part of the job description, for me to build a hardware lab, or running the lab, or actively upgrading one to replace existing ones. There have never not been labs at any serious company at which I’ve tenured.

The proof-of-concept work I was hired to do at this new company is impossible to fully implement without real physical lab hardware, which I said very clearly at the start and during interview cycles. I asked about the operating system, the laptops, VDI systems, dev boxes and developer resources, the usual. I asked about ADA compliance accommodations, because I am disabled; simply checking all of the usual boxes before signing a contract and relocating my entire life to go slog away for an IPO, right? Right.

I asked, "~why do the dates for the lab deployment keep slipping?" Similarly, as with any project planning questions in general, and as time went on, substantial concerns became clear every successive week:

1. Where is the company going?
2. Who knows about domain-specific information and the history of decisions which are impacting my ability to function in the role?
3. Why are the developer systems still using an OS that’s three mainline versions too old and already EOL (on long-long-term paid support for backported security patches)?
4. <Insert all of the other questions to which I’ll not bore the reader>

About that Cross-Functional Leadership Team

Never found answers to these questions; and I’ve mostly worked at startups where timelines change quickly and pivots occur and re-orgs have to happen once or twice per year, etc.

• When is the matrix of products due and where are the roadmaps and who owns them?
• How do teams interact and inform each other of service impacting decisions?
• Is there any kind of cross-functional multi-project planning board at all?
• What does the data analytics pipeline look like? (I can build it from zero if needed)

Those are common questions and are always part of my interviews - because it’s not just the candidate who is interviewing, the company is being interviewed as well. The answers before moving were just specific enough to trust, but I should never have trusted, not again.

Signs to Depart - Signalled Months in Advance

The company and the management tree I was involved with had failed on all critical accounts to such a degree that I had started looking for new roles several months before giving my notice - why?

Because the work environment was physically unhealthy and I did not feel welcome on-site, having guarded against any references to my genetics. There was a war going on and I was from the side that too many people at the company were viscerally and vehemently vicious towards.

Well then, What to do? What to do?!

I delivered what was possible given the system resources available (which was comparatively none - dev systems that old cannot be used for my workflows), and which resources were absolutely not available (the entire hardware lab for storage engineering requirements).

No NDA issues here, no personal attacks, etc. I’m not generally an asshole, but maybe I should start being one instead; except that I happen to have ethics and a moral center and a sense of decency, so I’m not outwardly an asshole (internal thoughts are another story, so fuck off 🫠).

Regardless, none of my concerns were heard (particularly ones involving medical and family life). Time went on. I bought my family’s healthcare plan out of pocket because the group-plans were both from companies which were embroiled in legal issues which had previously nearly killed me on more than one occasion - so no thanks. That substantially cuts into base-salary comp, and when a planned and promised IPO is cancelled - there goes any hope of accruing RSUs that were supposed to be a component of Total Comp after the first year cliff. Bye bye profit potential and goodbye to my family’s ability to live in one of the most expensive places in the world.

Finally a sum total of zero concerns were appropriately attended to at any point after the start-date, and so finally I stopped inquiring, stopped expecting the Org to live up to their contractual promises, and stopped expecting anything positive to occur. Nothing positive ever occurred until I left - and it was simply not being involved with the daily environment and experience of being a Jewess of Germanic lineage living with ADA disabilities.

Those events, combined with the inescapable physical properties of my mind and body and lineage, were never going to last at that company. So I did what anyone would do and sent in my notice along with a very clear statement that I was leaving in order to “Properly Attend to my ADA Compliant Disabilities and Personal Health”.

We are Very Limited in Time

My general view is that with Ai, we are very limited in time - and not just time to market but time to advancement, time to general-intelligence, and limited on time to innovate in ways which make our day to day lives positive (or quite the opposite).

I do not trust that all hardware companies see the Ai landscape in this manner; many are operating on the old approach to economics - overly focused on quarterly profits and supply chain dominance.

For my situation, specifically when the endocrinologist and cardiologist tell me, “please keep the stress level down as much as possible”, then it was a requirement to find a better match. Paying $2K out of pocket for healthcare is just not a long term option, and I can get 100% of that covered almost anywhere in the Valley ecosystem.

People in the industry are playing hard and fast, and whether those qualities may be considered a good thing is up for endless debate, but not right now. Right now I’m going to show some End of Year snapshots which are relevant to the blog’s purpose: wtf am I doing here anyway?

2025 was mostly extremely challenging, and mostly in helpful ways, but also featured a repetition of lessons which it seems I need to reflect on for additional forward momentum.

Let’s take a look at some of my core qualities as told by the oft-loved and oft-maligned OpenAi’s Chat-GPT 💟.

Your Year, Painted in Pixels

Very close, nice work GPT. While I do not have a 400GbE switch on my desk, I do have a few 100GbE 32-port switches in the racks, and occasionally 200GbE NICs and optics will float around the test bench.

The Architect Archetype ~ 3.9% of ChatGPT Users

It’s true! Over the 25 years of this career I’ve held architecture specific roles across the domains of Hardware Systems, Infrastructure Automation, Databases, and HPC Storage realms. It’s what I’ve always wanted to do, going all the way back to 1985 when introduced to the 4.77MHz IBM PC, Model 5150.

Personality: Specialist, Planner, Conceptual

That’s fairly accurate, more so if we add “spatial reasoning” to a list of cognitive methods.

“Thinks in structures and systems. Uses ChatGPT to design elegant frameworks and long-term strategies within a domain.”

A Datacenter in the Living Room?

Close in accuracy, however the “living room datacenter” is more commonly located in a home-office room (when living in an apartment), or in a dedicated area of the garage with HVAC filtration + dedicated 208/220/240V power, and the requisite automated cooling systems (usually Tripp-Lite units with duct-blowers). For systems at home the machines are often bought as separate server components along with a 4-5U chassis + rails + multiple sub-50 decibel large-format 120-180mm fans. Systems with high RPM 40-80mm axial screamers belong in a proper colocation facility.

How about that Homelab - is it in a constant state of flux?

Yes, mostly. It’s often helpful for proof of concept tests, hypothesis validation, and running benchmarks without having to drive to the DC or go back and forth with Remote Hands requests. Plus, lots of fun going on with hardware at home, but sometimes results in broken nails 😔.

Messy cables? Sometimes. Labels? Sometimes.

The homelab is often rebuilt: storage devices, fiber, dac, cables, and test clusters revised.

Homelab, sure… but how about those colocation photos instead?

As we’ve seen, some of the server hardware lives at home in my 25U shorty-racks, some is across the bay at a datacenter where big-loud machines require CRAC style cooling, and some of it is spread across the globe at other colocation facilities where The Providers have megawatt and gigawatt infra capacity. My personal infrastructure is a hybrid design, utilizing everything from embedded systems to multi-GPU hosts to public-cloud virtual machines… so, let’s see a bit more…

• Recent datacenter photos: https://prose.winterschon.com/a-recent-datacenter-colo-visit/

Your Chat Style

The analysis says… well, it’s fairly accurate for a description of being in-person as well as at the keyboard during remote meetings or daily chats.

Technical or otherwise, usually this is consistent. I’m often chatty, verbose, attentive, talkative, opinionated, even positively bubbly at times. Unless you catch me on a day where those qualities are supplanted by the clouds of quietly recalcitrant moods - the type which are grounded on a dualistic view of existence best described as Stoicistic-Nihilism, having been sourced from far too many post-traumatic stress events endured.

Your Chat Style: Direct, technically precise, often layered with humor and a critical edge, mixing structured reasoning with conversational spontaneity and sharp observational wit.

Your 2025 Chat Stats

Hold on now. I’ve used em dashes since 6th grade English class—so if seeing those in my writings is giving an impression that I use LLMs for all of my authoring is quite incorrect. For all of those people who don’t like them I have a solution: go to the settings section and set your rules and move on with your life. 😏

• Total Chats: 720
• Messages sent: 4,861 (not tokens, messages)
• Chattiest day: September 5th (why? we’ll need to see…)
• Images generated: 3 (Sora kinda sucks)
• Em-Dashes “Exchanged”: 1,882
• Top Messages Sent: 1% (percentile)

Regardless of those stats, even with my use of ChatGPT’s advancements with their “Pro Account”, most of my Ai/ML occurs on self-made infrastructure. I don’t rely on closed source, I don’t like closed source, and I don’t like non-democratized intelligence services that are just in it for profit. My career was built on Open Source and this current infra uses tooling like Ollama, Open-WebUI, vLLM, llama.cpp, CUDA, PyTorch, ROCm, and many other examples of collaborative ingenuity - free to use, free to improve, so please contribute if you can.

Three Big Themes of 2025

1. Building Ultra-Low-Latency Systems (true)
2. Experimenting, Measuring, Refining (true)
3. Engineering Meets Everyday Life (true)

Thoughts on 2026, Ai Industry Trends != Rules

Working in the Ai/HPC/ML industry, an observable fact is that the majority of consumer-facing Ai companies are still trying to figure out this new landscape, how it works, why anyone should care, and how to monetize whatever it is that they’re focused on.

Most will repeat the errors of the past, most will fail, most will malign the tech while misunderstanding nearly everything about the topic; and that’s fine. That’s the routine in Silicon Valley, but it doesn’t have to be.

1. We cannot solve today’s problems with yesterday’s methods, and that is a simple fact. We need the core technologies involved with Ai in order to resolve the massive amount of tech debt from earlier generations, but we need to stop expecting the same success from the same methods from the past; this is not the same world as 10, 20, or even 5 years ago.
2. The current anti-Ai mentality in mass-media is occurring despite companies like IBM, Cray, Fujitsu, BullSequana, and a handful of others with Top-500 dominant postures being focused on Ai-enabled Supercomputing, HPC, and acceleration hardware domains for well over two decades. These clusters have been active at research facilities with names like CERN, Technion, and of course the DoD and Military labs. These clusters keep us safe, they advance human understanding, assist with solving otherwise unsolvable problems, and are used to maintain the systems and functions our current world relies on for security and the advancement of human potential.
3. Ai is not just chatbots, it’s not limited to multi-modal agentic buzz-words, but that’s a tangent for another day. Datacenters are not ruining the world like mass-destruction of rainforests just so people can eat more cows. Conversely, we have an entire section of the Supercomputing industry focused on green energy and resource efficiency. Medical advancements, brain research, climate science and modeling, cancer research, etc… all of this is and has been advanced by Ai - but it’s not from the consumer side and it’s not how the media presents the information. Gosh, I wonder what incentive the media has to engage in such maligning tactics?

How Do We Move Forward?

In spite of or despite anti-Ai sentiments, while the public-facing corporate marketing landscape fights for “Consumer Economy Dominance”, engineers and architects like myself are busy in the background innovating with (sometimes our own) Ai/ML systems and (sometimes our own RSU/ISO/BST* funded) heavy-compute big-data storage infrastructures.

People like myself do this because we love it (mostly, nothing is perfect), and because we want to improve the lives of those around us, not to simply make money and acquire material goods. These skills require compromise, dedication, attention to detail, and a deeply held desire to improve the human condition through personal investment of time, mentoring younger engineers, and the occasional runway.

TOGETHER WITH THE AIs WE WILL BUILD A BETTER ULTRON

Kidding, mostly. Ultron had some good philosophical points, but we’re not in Strawberry Fields or the BS about paperclips from unimaginative and uninformed bloggers; this is real life and The Terminator is not around the corner. Pass the espresso and let’s check the local-node TTFT and CHRs to see how SkyNet is doing. ❤️

This post was written over a series of months, initially as lab notes, with some bits and bytes added each time the tools come out or the patch panels are rewired. While perhaps not a common task, hopefully the knowledge finds someone else looking for these solutions.

Switch-Port System Roles

This post discusses several types of connections and signals conversion in a field where the list of potential operating modes is prone to complexity. Brands involve APC, Avocent, Blackbox, Cisco, Cyclades, Moxa, Netshelter, OpenGear, Vertiv, and more - but here in my HomeLab and in the enterprises in which I’ve worked, there’s a common requirement for critical equipment to talk to a System Infrastructure Manager system or OOB device.

System Infrastructure Manager

Featured in the post’s header image, I’m using Blackbox for OOB services, but these specs are matched by various brands in similar price points.

Device: BlackBox model LES1248a out‑of‑band (OOB) connection manager
Specs: 48‑port, 1U, dual-PSU, dual-LAN, cellular 3G and POTS optional

Provided Features

1. Remote access services using web-based serial console interface for infrastructure devices (switches, routers, PDUs, UPS, etc).
2. Centralizes remote‑management via multi-protocol connections and bridging between serial devices and user-facing network services.
3. Access‑control lists (ACLs) and role‑based access control (RBAC)
4. Monitoring, logging, and alerting

Reference: Generic Pin Mapping for 8P8C RJ45

This represents the usual 8P8C RJ45 connector layout, with pin usage according to different serial standards. While we are focused on RS232, it can be helpful for visualization purposes to see how other standards are implemented.

Working with Keystone Connectors

Those 48 ports aren’t going to fill themselves up without some occasional head scratching along the lines of..

Q: “How can I connect this RJ12 RS232 based APC PDU to a RJ45 RS232 infrastructure manager?”

A: You’re going to have to convert the pin-out for an RJ12 cable to the expected pin-map of an RJ45 RS232 rollover cable.

Great. I have those requirements!

Let’s see the various pin and cable maps involved with some common use cases. We’ll get to the RJ12 content, but let’s see standard mappings first.

1) Standard Ethernet, 8P8C Straight-Straight

These are simple and have no pin conversion to map. This is a basic reference only, there’s nothing to do.

2) Cisco Rollover, 8P8C Switch-Crossover

Standard rollover cable pin-map. In this case it’s what we want to use when connecting a Cisco style switch to our Blackbox (both with RJ45 RS232 ports).

Keystone Pins, 8P8C Switch-Crossover

My patch panels use keystone jacks to simplify things, and they ship with straight-through RJ45-RJ45 connectors which do not change the wiring. That’s fine in some cases, but let’s see what we need to do to build a custom crossover keystone jack.

3) APC PDU Conversion, 6P6C RJ12 → 8P8C RJ45

• APC PDU side: 6P6C RJ12 connector with grey ethernet cable
• Blackbox side: 8P8C RJ45 connector with grey ethernet cable

RJ12 6P6C Connector for APC (940-0144A)

While APC has more than one RJ12 connector on their products, this one is the most common from my SKU usage in the enterprise.

Keystone Pins, 6P6C RJ12 for APC PDU

This is where things get fun? We have an 8-pin cable on both sides of the jack, but only 4 pins are needed on the APC RJ12 side. Look closely and plan ahead when you’re chopping wires and punching connectors.

Rj45 Blackbox View

Rj12 APC View

USOC (RJ‑12) Pinout

RJ‑45 (T568B) Pinout

Both USOC/T568B tables assume the connector tab is facing down and the pins are read left‑to‑right.

Regarding Flow Control - DSR/DTR Assertion

Fun times with RS232 sometimes involve flow-control, though often this is set to “None”, there are more options than None where None is concerned. Let’s see why Blackbox offers the following options, given the following debugging scenario:

Defining a Common Flow Control Issue

Problem: Blackbox shows incomplete port status*, with no terminal output when connecting via Cisco rollover cable.

[*] (“RTS | DTR” instead of “RTS | DSR | DTR | DCD”)

Signal Debugging Workflow

Inspecting Flow Control Modes

• Q: What, surely the two None types are equivalent?
• A: They are not equivalent.

The port is usable in either “None-Type” modality only if TxD/RxD signals are present. The console server will report “No signal data available” when the connected Blackbox port is operating in the following states:

1. not powering when it needs power
2. actively sampling modem-control lines
3. when the remote device side never asserts them

Confusions with State-Mode Usability

Using “None - Flow control signals unpowered” disables the port’s modem-line sensing, which prevents the appliance from reporting DSR/DCD/CTS.

• Whether or not you can connect over the line depends whether the device has active TxD/RxD connection signals, irrespective of whether Blackbox knows about DSR/DCD/CTS state.
• Alternatively, the remote device may never assert DSR/DCD, as many network-console ports don’t provide carrier signals, so the server has nothing to report.

Ok… What else?

Despite many efforts and assumptions, it’s a fact that RS232 is not going anywhere any time soon and it remains not simply useful but necessary.

The global view of serial connection usage includes datacenters, industrial automation systems, expansive cellular / broadband / telco networking hardware, mainframe connected terminals, all the way to a near-endless number of embedded systems. Good times, plenty of TxD/RxD signals to go around. 😄