92d+ Uptime on Custom Gentoo Stage4 LiveISO

And that was a decent go on this build; one simply avoids ever using GCC and SystemD – use LLVM/Clang + OpenRC, and.. wow.. it’s like 2006 when things worked, Cobalt RaQ was still in the colo by the grids, an E10K or a few hundreds were still in service and would continue to stay online for years to come.

Oh, and admins and engineers didn’t avoid UNIX tools or determine ifconfig or netstat had to be "deprecated" (please, fuck off). So, get rid of the trash and then the systems become substantially more stable on the daily. Personally I trust my own ability to run and automate fleets of systems, not a tangled mess of low-hanging-fruit nod-to-redmond that most linux distros are suffered to use (CVE town is so rad, ja?)

92 days of uptime is generally close to mean value, and unfortunately it was ruined by having to upgrade the BMC and BIOS firmware versions due to “reasons”. Everything is being upgraded just in case because of the Github CVE in May. It’s wrecked so many upstream tools and sites and services and so many people are scrambling as if Log4J had just shit the bed and fell back asleep. But don’t worry.. “no customers were impacted” 😂

Anyway, Resuming to Workstations and Stage4

While there have been improvement on recompiles when optimizing test nodes’ kernels and enabling accelerators, the CI system for the building stages was upgraded to enable portage to use cross-datacenter distcc (distributed compiling) over a decent 5GbE symmetric fiber from a lab-rack to the colo cross-connect. Then the jobs are spooled off to SLURM nodes, rebuilding thin-LTO, PaX, PAE, ASLR. Very similar to a distributed Poudtierer, but without the memories of Poutine (and I love a good Poutine, especially in a light snowstorm in Banff, at least as much as the Chinese restaurant on the first left into town).

How many times do I need to follow this type of flowchart (made it, wow, so amazing) every time Supermicro releases a new BMC or BIOS upgrade? Dell, you’re not much better but at least the release notes are decent and there’s the Yum/Deb repo (one of these days I’ll put my Gentoo + Dell Driver overlay into a public git repo (not on git/msft-hub.com) for other Gentoo PowerEdge users to more easily remain up-to-date on firmware packages (fwupdmguur does’t cover everything, and the last time I trust it not to shit the bed it bricked my Thinkpad X1 Nano - thanks brosefs (am I saying that right? ehh).

fuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuu

For those not familiar, the SoW is a consulting agreement between contractor and employer, which defines the scope and timeline and requirements etc to “do the needful” in a broader sense of everyone’s favorite phrase from the ‘ol “Follow the Sun” SLA support model from the early pre-Cloud era (sorry cloudbabies it’s from when you were single digits old or not yet born).

This specific SoW has been used in prior endeavors which eventually follow the typical enterprise hardware lifecycle in no specific order [1]: forklifts, retros, rebuilds, expansion, and… words.. so many words.

Labs last a long time, so it’s important to build things correctly the first time - not waste time hacking together whatever compromised decision trees were necessary due to last-minute planning (or no planning at all) and little to no experience with building labs. Remember, if you are a specialized engineer who excels in one or two areas, that does not imply domain specific knowledge from adjacent or entirely different sides of engineering worlds. I’m a systems architect .. I don’t build jet engines (though it’s probably fun), so I do what I’m good at. Otherwise to say, “stay in your lane or learn to love losing - but protect that ego at all costs and never admit fault ever!” (yes, I’m bitter and for exceptionally valid reasons).

So, until the pre-IPO runway runs out, or the post-IPO environment bores people to tears and they jump ship with their ISOs and RSUs and START ALL OVER AGAIN! (I have definitely never not done this exact thing several times never for sure for sure? sure.)

Successful Labs Require People and Hardware

Vaporware labs don’t have hardware. Real labs do not revolve around PPTX slide decks, or Slack Canvas, or Sharepoint whatever, or ephemeral Confluence pages that change ownership by a click of a button. Real labs have real hardware and real engineers and architects involved at every critical stage to ensure that standards are being used, implemented, tested, and validated.

If you think I’m wrong, go ahead, do it all by slide deck, by half-baked simulators that don’t even use SST and parallelization correctly, with lots of Big Discussions for the Big People you want to impress to get ahead instead of making quality hardware.

See how far the Peter Principle gets you - chances are, if you’re surrounded by “Yes Men” and “C-Suite but never Technical” boardrooms, you will get far enough to be proven the fraud that you are.

The Peter principle is a concept in management .. which observes that people in a hierarchy tend to rise to “a level of respective incompetence”: employees are promoted based on their success in previous jobs until they reach a level at which they are no longer competent, as skills in one job do not necessarily translate to another.

Having risen rapidly, since you’re up high now - you can destroy entire companies by making extremely stupid choices “to save money” by not spec’ing hot-swap trays for NVMe drives on 1U systems which were never designed for the wattage you’re running - and were not spec’d for the BTU and LFM required to keep the systems from burning out 100G optics..

Just when you think it’s all good, um why are there only seven drives in this chassis that can run with eight?

• Oh it’s because the 8th drive would draw too much power and overload the non-redundant PSUs. What happens when a system which should have N+1 has only N?
• The PSUs could be in redundant mode but Billy didn’t want to authorize the enablement of active/active mode, and instead has demanded that his lacky make the systems run the PSUs in “combined mode” to try to get more power to that 8th drive… but no… not possible.
• So all of the fancy new systems can’t have eight drives like the original spec REQUIRED in order to fulfill global fleet machine generation scaling models. As in, you lose 1/8th of your drives on every new system for so many thousands of systems. 1/8th, for those not inclined to fractional conversions, is
• So now all of those systems will be missing ~2-16-64TB of NVMe drives (multiply by number of nodes per rack, per pod, per cage, per site, per region, per global fleet). That’s a lot of expected high-speed storage that is now not being deployed.
• Hopefully no contracts were made to customers based on “Total NVMe Raw Storage per <unit/region/etc>” variables, which now renders the deliverable impossible.
• What can be done? Those contracts have to either be revised and compensated (yeah good f’in luck), or a massive workaround must be implemented to compensate - at additional cost to the company.

Why does that matter to building labs?

Because any lab worth their salt would have raised immediate red flags and halted the entire ill-devised scope. Then they would use standardized and periodically revised for betterment:

• provide technical backing evidence
• scaling costs, IOP/s cost, power assessments, baseline to STDDEV ratio, then appropriation wear-cycle equations
• All that industry jargon that managerializes into the ether for all to repeat.
• “keep me honest” you may say, so then.. are you often lying? Ill-informed, unprepared, completely clueless?
• Maybe not, hopefully not, but analytical people will see you that way and eventually the Peter Principle catches up and BAM! You’re hit by a and your entire team is fired, and so you move on with your RSUs, talk yourself up, and try to ruin those who smote you from afar.

Congrats, Billy, you’ll never change - and that is why we test in labs - because things change.

The Contract Statement of Work

Always get the scope in writing with sign-off. Scope changes mid-stream, not awesome.

Statement of Work

Consultant Services Engagement

This Statement of Work ("SOW") is entered into between [Client Legal
Name] ("Client") and [Consultant Legal Name] ("Consultant"), effective
as of [Effective Date], and is governed by the [Master Consulting
Agreement / Professional Services Agreement] dated [Agreement Date] (the
"Agreement"). In the event of any conflict between this SOW and the
Agreement, the Agreement will control.

  -----------------------------------------------------------------------
  Client                              [Client Legal Name]
  ----------------------------------- -----------------------------------
  Consultant                          [Consultant Legal Name]

  SOW Effective Date                  [Effective Date]

  Initial Term                        Six (6) months from the SOW
                                      Effective Date

  Target Level of Effort              Forty (40) hours per week,
                                      approximately 1,040 hours over the
                                      initial term

  Primary Stakeholders                VP, OCTO, Eng-Org, App-Team, and
                                      Sales-Org

  Commercial Terms                    Billing rate, invoicing, payment
                                      terms, and approved expenses as set
                                      forth in the Agreement and/or
                                      applicable order form
  -----------------------------------------------------------------------

1. Purpose and Objectives

The Consultant will provide technical leadership, architecture,
implementation, documentation, mentoring, and subject matter expertise
to establish standardized HPC benchmarking, workload automation, and
telemetry/analytics capabilities, while also improving cross-functional
execution and the overall developer experience for CORP products and
associated collateral.

- Define, document, and deploy a modular "Industry Expectations
  Benchmark Suite" leveraging HPC, SPEC, Top500, AI/ML GPU leaderboards,
  and SNIA methodologies to support rigorous adherence and consistent
  results using standard performance testing tools.

- Define and deploy a standardized HPC "Workload Automation Pipeline"
  utilizing SLURM, Jenkins, Celery, RabbitMQ ("RMQ"), ClickHouse,
  Ansible, and related tooling.

- Define and deploy a standardized HPC "Telemetry + Analytics
  Aggregation + Data Visualization Processing Pipeline" utilizing
  Check_MK, SNMP, Elasticsearch + Kibana, Prometheus, InfluxDB, Grafana,
  and related tooling.

- Assess current organizational working patterns across Eng-Org,
  App-Team, and Sales-Org and provide a quarterly review to the VP and
  OCTO.

- Assess the current CORP collateral surface area, including SDKs,
  documentation, libraries, proofs of concept, and reference designs, as
  an ongoing initiative to improve the developer experience.

- Author white papers, research documents, and related technical
  materials, with a feedback loop to validate and improve internal and
  external documentation.

- Provide mentoring for junior through senior application and systems
  engineers, plus reasonable availability as an SME across storage,
  systems, database, and AI/ML domains.

- Provide ad hoc consultation to support Sales-Org customer scoping,
  implementation, and integration needs.

2. Term and Working Model

The engagement will begin on the SOW Effective Date and continue for an
initial six (6) month term. The Consultant will generally allocate forty
(40) hours per week to the engagement, subject to mutually agreed
priorities, holiday schedules, and reasonable scheduling adjustments.

The parties acknowledge that some workstreams in this SOW are ongoing or
iterative in nature. Accordingly, the Client and Consultant will jointly
prioritize and sequence the work so that the highest-value deliverables
are completed within the available time and level of effort.

Services may be provided remotely or through other mutually agreed
working arrangements. Any onsite work, travel, or work materially
outside the target level of effort will require prior written approval.

3. Scope of Services

3.1 Modular "Industry Expectations Benchmark Suite"

The Consultant will define, document, and deploy a modular benchmark
suite intended to measure representative performance, reproducibility,
and adherence to industry-standard methodologies across compute,
storage, and AI/ML use cases.

Illustrative deliverables:

- Benchmark taxonomy and workload catalog covering applicable HPC, SPEC,
  Top500-style, AI/ML GPU leaderboard-relevant, and SNIA-aligned
  methodologies.

- Benchmark execution standards, parameterization guidance, and runbooks
  designed to improve repeatability and result consistency.

- Result templates and reporting conventions for capturing baseline
  measurements and comparative findings.

- Initial implementation and/or deployment in designated Client
  environment(s), subject to infrastructure readiness and access.

- Validation summary describing observed results, known limitations, and
  next-step recommendations.

3.2 Standardized HPC "Workload Automation Pipeline"

The Consultant will define and deploy a standardized HPC workload
automation architecture using SLURM, Jenkins, Celery, RMQ/RabbitMQ,
ClickHouse, Ansible, and related tools as appropriate to the Client
environment.

Illustrative deliverables:

- Reference architecture and component interaction design for job
  submission, orchestration, execution, retry handling, and result
  capture.

- Deployment-ready configurations, automation artifacts, or initial
  implementation in designated environment(s).

- Operational workflows for benchmark or workload scheduling, execution,
  data collection, and reporting.

- Runbooks and handoff documentation sufficient to support repeatable
  operation and internal adoption.

3.3 Standardized HPC "Telemetry + Analytics Aggregation + Data
Visualization Processing Pipeline"

The Consultant will define and deploy a standardized observability and
analytics processing pipeline spanning infrastructure telemetry,
log/event aggregation, time-series monitoring, analytics storage, and
visualization.

Illustrative deliverables:

- Reference architecture covering Check_MK, SNMP, Elasticsearch +
  Kibana, Prometheus, InfluxDB, Grafana, and related integrations as
  applicable.

- Recommended data model, retention approach, dashboard structure, and
  operating practices for standardized reporting and analysis.

- Initial dashboards, visualizations, and/or deployment artifacts for
  designated environment(s).

- Documentation for data sources, ingestion paths, dashboards, and
  operational ownership considerations.

3.4 Cross-Functional Collaboration Assessment

The Consultant will assess current organizational working patterns
across Eng-Org, App-Team, and Sales-Org in order to identify friction
points, handoff gaps, role ambiguity, communication issues, and
opportunities to improve execution.

Illustrative deliverables:

- Current-state assessment and concise findings memo.

- Recommendations for cadence, decision ownership, communication flow,
  and escalation patterns.

- Quarterly review readout(s) presented to the VP and OCTO during the
  engagement term.

3.5 CORP Collateral Surface Area and Developer Experience

The Consultant will assess the current ‘surface area’ of CORP collateral,
including SDKs, documentation, libraries, proofs of concept, and
reference designs, as an iterative stretch initiative with the overall
goal of improving the developer experience.

Illustrative deliverables:

- Inventory and gap analysis of current CORP technical collateral.

- Prioritized recommendations and/or backlog themes to improve
  consistency, usability, completeness, and customer/integrator
  experience.

- Feedback loop for validating and improving internal and external
  documentation and supporting artifacts.

- Periodic progress updates and next-step roadmap recommendations within
  the engagement window.

3.6 White Papers, Research Documents, and Documentation Improvement

The Consultant will author white papers, research documents, technical
notes, and similar written materials as prioritized by the Client, and
will help create a feedback loop to validate and improve internal and
external documentation quality.

Illustrative deliverables:

- White papers, technical briefs, research documents, architecture
  notes, or similar written outputs as mutually prioritized.

- Document review comments, revision recommendations, and technical
  fact-checking input.

- Structured feedback loop to capture lessons learned and improve future
  documentation quality.

3.7 Mentoring and Knowledge Transfer

The Consultant will provide mentoring support for junior through senior
application and systems engineers, with an emphasis on practical
technical development and hands-on knowledge transfer.

Illustrative deliverables:

- Office hours, design reviews, pair-working sessions, workshops, or
  other mentoring interactions as appropriate.

- Guidance on benchmarking, automation, observability, storage, systems,
  database, and AI/ML topics relevant to the engagement.

- Knowledge transfer materials or summary notes where appropriate to
  support broader team adoption.

3.8 SME Availability and Sales-Org Consultation

The Consultant will maintain reasonable availability to organizational
staff in applicable subject-matter areas of focus and will provide ad
hoc consultation for Sales-Org customer scoping, implementation, and
integration needs, as capacity permits within the contracted level of
effort.

Illustrative deliverables:

- Advisory support for storage, systems, database, and AI/ML questions
  arising during the engagement.

- Participation in internal and customer-facing scoping discussions,
  solution reviews, or implementation planning sessions, when requested
  by the Client.

- Technical input intended to improve solution fit, implementation
  readiness, and integration clarity; provided as advisory support
  rather than as a standalone managed service or customer support
  commitment.

4. Indicative Delivery Plan

The following plan is illustrative and may be reprioritized by mutual
agreement based on Client needs, environment readiness, and emerging
business priorities.

  -----------------------------------------------------------------------
          Period          Primary Focus           Illustrative Outputs
  ----------------------- ----------------------- -----------------------
          Month 1         Discovery and Baseline  Stakeholder alignment,
                                                  current-state review,
                                                  environment access,
                                                  priority mapping,
                                                  benchmark/workload
                                                  definition, and
                                                  architecture framing.

        Months 2-3        Design and Initial      Benchmark suite v1,
                          Deployment              automation pipeline
                                                  design and initial
                                                  deployment artifacts,
                                                  telemetry pipeline
                                                  design, and initial
                                                  dashboards or data
                                                  paths.

        Months 4-5        Validation and          Validation runs,
                          Expansion               documentation
                                                  expansion,
                                                  collaboration
                                                  assessment findings,
                                                  collateral review,
                                                  white papers/research
                                                  outputs, and mentoring
                                                  support.

          Month 6         Handoff and Executive   Runbooks,
                          Review                  recommendations,
                                                  backlog or roadmap
                                                  items, knowledge
                                                  transfer, and
                                                  quarterly/executive
                                                  review presentation(s)
                                                  to the VP and OCTO.
  -----------------------------------------------------------------------

5. Governance and Reporting

- The Client will designate a primary engagement owner or point of
  contact responsible for prioritization, coordination, and consolidated
  feedback.

- The parties will maintain a regular working cadence, which may include
  weekly or biweekly syncs, technical reviews, design reviews, and
  status reporting as appropriate.

- The Consultant will provide concise updates on progress, risks,
  blockers, and recommended next actions.

- A quarterly review readout will be provided to the VP and OCTO during
  the engagement term.

6. Assumptions, Dependencies, and Constraints

- The Client will provide timely access to relevant systems,
  environments, credentials, documentation, repositories, data, and
  subject-matter stakeholders reasonably required for the services.

- The Client is responsible for providing or approving the hardware,
  software, licenses, network access, security approvals, and
  change-management windows required to implement or deploy
  deliverables.

- Any production deployment, change control, or release activity remains
  subject to Client approval processes and operational safeguards.

- Where this SOW references industry-standard benchmarks or
  leaderboard-adjacent methodologies, any formal external publication or
  third-party submission will remain subject to separate Client approval
  and third-party rules.

- The Consultant’s work product may include recommendations, reference
  architectures, configurations, code or scripts, runbooks, dashboards,
  presentations, reports, and live working sessions; not every
  workstream will result in a standalone written deliverable.

- Because certain activities are advisory and ongoing in nature, the
  parties will periodically rebalance priorities to remain within the
  contracted time and level of effort.

- Travel, onsite presence, after-hours support, and work materially
  beyond the initial term or weekly level of effort are excluded unless
  separately approved in writing.

7. Out of Scope

- Managed services, long-term production operations ownership, or 24x7
  operational support.

- Hardware procurement, datacenter installation, or unrelated
  infrastructure build-out not directly tied to the services described
  in this SOW.

- Binding commitments to end customers, contractual promises to third
  parties, or customer project delivery obligations not expressly
  authorized by the Client.

- Ongoing maintenance, enhancement, or support of deliverables after the
  end of the engagement term, except as separately agreed in writing.

8. Acceptance and Change Control

Commercial terms, invoicing, payment terms, intellectual property,
confidentiality, warranties, limitation of liability, and any formal
acceptance procedures will be governed by the Agreement.

Where the Agreement does not specify a separate acceptance procedure,
the Client will review deliverables promptly and provide consolidated
feedback within five (5) business days after delivery. Deliverables will
be deemed accepted when they materially conform to the scope described
in this SOW, subject to any documented punch-list items or agreed
revisions.

Any material change to scope, deliverables, timeline, or level of effort
will require written approval by both parties.

9. Signatures

The parties acknowledge and agree to this SOW as of the dates written
below.

By: ________________________________ 
Name: ______________________________ 


[Consultant Legal Name]
Title: _______________________________
Date: _______________________________
By: ________________________________ 

[Client Legal Name]
Name: ______________________________
Title: _______________________________
Date: _______________________________

People in the industry are playing hard and fast, and whether those qualities may be considered a good thing is up for endless debate, but not right now. Right now I’m going to show some End of Year snapshots which are relevant to the blog’s purpose: wtf am I doing here anyway?

2025 was mostly extremely challenging, and mostly in helpful ways, but also featured a repetition of lessons which it seems I need to reflect on for additional forward momentum.

Let’s take a look at some of my core qualities as told by the oft-loved and oft-maligned OpenAi’s Chat-GPT 💟.

Your Year, Painted in Pixels

Very close, nice work GPT. While I do not have a 400GbE switch on my desk, I do have a few 100GbE 32-port switches in the racks, and occasionally 200GbE NICs and optics will float around the test bench.

The Architect Archetype ~ 3.9% of ChatGPT Users

It’s true! Over the 25 years of this career I’ve held architecture specific roles across the domains of Hardware Systems, Infrastructure Automation, Databases, and HPC Storage realms. It’s what I’ve always wanted to do, going all the way back to 1985 when introduced to the 4.77MHz IBM PC, Model 5150.

Personality: Specialist, Planner, Conceptual

That’s fairly accurate, more so if we add “spatial reasoning” to a list of cognitive methods.

“Thinks in structures and systems. Uses ChatGPT to design elegant frameworks and long-term strategies within a domain.”

A Datacenter in the Living Room?

Close in accuracy, however the “living room datacenter” is more commonly located in a home-office room (when living in an apartment), or in a dedicated area of the garage with HVAC filtration + dedicated 208/220/240V power, and the requisite automated cooling systems (usually Tripp-Lite units with duct-blowers). For systems at home the machines are often bought as separate server components along with a 4-5U chassis + rails + multiple sub-50 decibel large-format 120-180mm fans. Systems with high RPM 40-80mm axial screamers belong in a proper colocation facility.

How about that Homelab - is it in a constant state of flux?

Yes, mostly. It’s often helpful for proof of concept tests, hypothesis validation, and running benchmarks without having to drive to the DC or go back and forth with Remote Hands requests. Plus, lots of fun going on with hardware at home, but sometimes results in broken nails 😔.

Messy cables? Sometimes. Labels? Sometimes.

The homelab is often rebuilt: storage devices, fiber, dac, cables, and test clusters revised.

Homelab, sure… but how about those colocation photos instead?

As we’ve seen, some of the server hardware lives at home in my 25U shorty-racks, some is across the bay at a datacenter where big-loud machines require CRAC style cooling, and some of it is spread across the globe at other colocation facilities where The Providers have megawatt and gigawatt infra capacity. My personal infrastructure is a hybrid design, utilizing everything from embedded systems to multi-GPU hosts to public-cloud virtual machines… so, let’s see a bit more…

• Recent datacenter photos: https://prose.winterschon.com/a-recent-datacenter-colo-visit/

Your Chat Style

The analysis says… well, it’s fairly accurate for a description of being in-person as well as at the keyboard during remote meetings or daily chats.

Technical or otherwise, usually this is consistent. I’m often chatty, verbose, attentive, talkative, opinionated, even positively bubbly at times. Unless you catch me on a day where those qualities are supplanted by the clouds of quietly recalcitrant moods - the type which are grounded on a dualistic view of existence best described as Stoicistic-Nihilism, having been sourced from far too many post-traumatic stress events endured.

Your Chat Style: Direct, technically precise, often layered with humor and a critical edge, mixing structured reasoning with conversational spontaneity and sharp observational wit.

Your 2025 Chat Stats

Hold on now. I’ve used em dashes since 6th grade English class—so if seeing those in my writings is giving an impression that I use LLMs for all of my authoring is quite incorrect. For all of those people who don’t like them I have a solution: go to the settings section and set your rules and move on with your life. 😏

• Total Chats: 720
• Messages sent: 4,861 (not tokens, messages)
• Chattiest day: September 5th (why? we’ll need to see…)
• Images generated: 3 (Sora kinda sucks)
• Em-Dashes “Exchanged”: 1,882
• Top Messages Sent: 1% (percentile)

Regardless of those stats, even with my use of ChatGPT’s advancements with their “Pro Account”, most of my Ai/ML occurs on self-made infrastructure. I don’t rely on closed source, I don’t like closed source, and I don’t like non-democratized intelligence services that are just in it for profit. My career was built on Open Source and this current infra uses tooling like Ollama, Open-WebUI, vLLM, llama.cpp, CUDA, PyTorch, ROCm, and many other examples of collaborative ingenuity - free to use, free to improve, so please contribute if you can.

Three Big Themes of 2025

1. Building Ultra-Low-Latency Systems (true)
2. Experimenting, Measuring, Refining (true)
3. Engineering Meets Everyday Life (true)

Thoughts on 2026, Ai Industry Trends != Rules

Working in the Ai/HPC/ML industry, an observable fact is that the majority of consumer-facing Ai companies are still trying to figure out this new landscape, how it works, why anyone should care, and how to monetize whatever it is that they’re focused on.

Most will repeat the errors of the past, most will fail, most will malign the tech while misunderstanding nearly everything about the topic; and that’s fine. That’s the routine in Silicon Valley, but it doesn’t have to be.

1. We cannot solve today’s problems with yesterday’s methods, and that is a simple fact. We need the core technologies involved with Ai in order to resolve the massive amount of tech debt from earlier generations, but we need to stop expecting the same success from the same methods from the past; this is not the same world as 10, 20, or even 5 years ago.
2. The current anti-Ai mentality in mass-media is occurring despite companies like IBM, Cray, Fujitsu, BullSequana, and a handful of others with Top-500 dominant postures being focused on Ai-enabled Supercomputing, HPC, and acceleration hardware domains for well over two decades. These clusters have been active at research facilities with names like CERN, Technion, and of course the DoD and Military labs. These clusters keep us safe, they advance human understanding, assist with solving otherwise unsolvable problems, and are used to maintain the systems and functions our current world relies on for security and the advancement of human potential.
3. Ai is not just chatbots, it’s not limited to multi-modal agentic buzz-words, but that’s a tangent for another day. Datacenters are not ruining the world like mass-destruction of rainforests just so people can eat more cows. Conversely, we have an entire section of the Supercomputing industry focused on green energy and resource efficiency. Medical advancements, brain research, climate science and modeling, cancer research, etc… all of this is and has been advanced by Ai - but it’s not from the consumer side and it’s not how the media presents the information. Gosh, I wonder what incentive the media has to engage in such maligning tactics?

How Do We Move Forward?

In spite of or despite anti-Ai sentiments, while the public-facing corporate marketing landscape fights for “Consumer Economy Dominance”, engineers and architects like myself are busy in the background innovating with (sometimes our own) Ai/ML systems and (sometimes our own RSU/ISO/BST* funded) heavy-compute big-data storage infrastructures.

People like myself do this because we love it (mostly, nothing is perfect), and because we want to improve the lives of those around us, not to simply make money and acquire material goods. These skills require compromise, dedication, attention to detail, and a deeply held desire to improve the human condition through personal investment of time, mentoring younger engineers, and the occasional runway.

TOGETHER WITH THE AIs WE WILL BUILD A BETTER ULTRON

Kidding, mostly. Ultron had some good philosophical points, but we’re not in Strawberry Fields or the BS about paperclips from unimaginative and uninformed bloggers; this is real life and The Terminator is not around the corner. Pass the espresso and let’s check the local-node TTFT and CHRs to see how SkyNet is doing. ❤️

A Tenuous Timeline

1) Some years prior to a decade ago, I joined LinkedIn, and a small handful of years ago I joined Twitter.

2) Along the way a serious incident occurred, resulting in PTSD from a bodily assault; the details of which require privacy. During associated therapy a HIPAA violation occured, eroding my trust in the medical establishment.

3) Periodically I would receive messages on LinkedIn from recruiters at the company where that man worked. I did not want to be reminded of the events, and would never work at the same company. These external intrusions into trauma memories were not healthy and they needed to stop.

4) Concerns evolved which necessitated telling that company to remove me from their recruiting database and never message me ever again.

5) The company removed my profile, and they asked if I wanted to be involved with an investigation, which I refused for the unfortunately common reasons:

• I did not want to relive events, especially after the HIPAA issue
• I did not want to discuss those memories with yet another stranger
• I did not want to become more of a target for speaking up

It’s a terrible catch-22, and yet it’s nothing new. This person needed to be prevented from ever contacting me or knowing anything about my life, requirements which the company could not ensure.

6) For safety reasons, I changed my legal name, then I moved, and started a new job. Then I enabled a service which monitors for identity theft issues and personal-info data leaks. Other precautions were made. The whole process is time consuming, emotionally disruptive, and fiscally burdensome.

7) Years pass. In the interim, LinkedIn’s “people you might know!” feature will routinely recommend that I connect with people who reminded me of that trauma.

• The feature cannot be disabled and profile settings do not help. Blocking people and profiles does not sufficiently help. Their algorithm always does this.

8) Years pass. Identity theft issues become a greater risk. Someone on Twitter steals my profile, impersonating me to sell web3 tokens or whatever. I report it, and Twitter demands my government ID - which I refuse to upload.

• Preferring a human interaction, I offer to bring my ID to the Twitter office and show it to someone in person, but no - they want the digital copy to keep on file, which will totally never be leaked in a data breach.
• I refuse to upload the document. They lock my account. I have not bothered since.

9) Years pass. A few weeks ago, I’m running a social media account audit and notice that LinkedIn has two unknown logins from another country - and I’ve never been there, nor connected to an ISP or VPN or service related to that location.

• I revoke the logins, yet they reappear a week later.
• My account has 2FA enabled, yet the auth logs do not show it being used.
• Who is behind these logins? What amount of personal info and connections and messages do they have access to?

10) After seeking help from LinkedIn’s support system, they now want my government ID, just like Twitter.

• They want a third party service (Persona) to scan it, and surely this will never be leaked (sure sure).
• They assure that this third party is very safe, that they only give LinkedIn redacted info (yet Persona keeps the full info?)
• Surely I should trust their entire system. I do not trust their system, and I will not upload my ID for it to be stolen or have any of the data used in some way to harm me.
• The result is similar to Twitter, my account is restricted with no access.

End Results and Oh Well…

So, goodbye LinkedIn, goodbye Twitter.

I cannot place my trust in these faceless social media corporations, the ones which never assume responsibility for data leaks, data theft, or the sale of demographics and usage meta-data. The common refrain of, “we’re telling you that we redacted your personal information”, means nothing substantial from these organizations.

These sites profit extensively from user data - we are their product for sale, and when that happens it can lead to extremely unsafe conditions, particularly if that data can be used for location tracking, enabling access which leads to physical violence, stalking, harassment, or other detriments by which women too-often experience the online world.

Sure, I could upload my ID and “just trust them”, but that would not make me safer; it would invalidate the entire purpose of protecting my body, my mind, my future. Handing them my trust, when they have routinely proven that they are not trustworthy, would render-pointless every hardship and effort towards safety and security which I’ve been required to endure since those events so many years ago.

By prioritizing my personal safety in both of these cases, the end result is that I am no longer able to engage and connect with people from my work life via the two most common business-centric social networks. The alternative is no alternative at all; choosing not to be safe is not a choice.

I know what it’s like to have my choice taken from me, and I never want that to happen again.

In the worst of circumstances, Fedora notified me of a firmware update, which was a hands-off type of “reboot to install” EFI blob which bricked my Thinkpad X1 Nano laptop just last week. That’s an expensive failure and one which has no simple resolution (many hours, many iso to img to efi/cap conversions, and maybe soon a WSOP-8 NAND reprogramming).

I’ve been all-in on the RedHat infra train since 1999, having used it at global backbone providers, satellite infra providers, hpc clusters, media conglomerates, gaming networks, streaming services, cloud providers and hyperscalers, and just about everything in-between… and while I have my qualms on certain technical aspects and corporate strategy over those years, I still consider it one of my favorite Linux distos (top five at least), and it’s my first choice for enterprise Linux. So, I’m not just hating on the ecosystem here, I’m a vested user who’s been disappointed too many times by the direction

Unfortunately, with Fedora, I stand behind my position that Redhat has been too hands-off in conducting standards requirement alignment, in requiring a base level of stability upon which serious engineering development can be done, without having to clean up other people’s messes every few weeks.

It’s just not a great dev platform distro anymore, and yet the org markets the name as if it were a valid OS for all manner of production uses. IoT use? Never, no thanks. Edge devices? Never again. Security? Forget about it.

It’s not a “bleeding-edge issue” - as that’s Rawhide’s territory where stability of any sort is not expected. The evolving issue with Fedora is that it is uncommonly unstable for a “dev env”, and one which ships far too much untested code and pushes the equivalent of “crowd source fail-tests” where they expect the users to “just tell us if it doesn’t work”. Their release engineering and load-testing seems to occur with zero multivariate matrices sequences for all manner of performance, validation, and functional tests. I suppose they consider unit testing to be “full coverage”, but it is not. Happy to be corrected here, though I will remain disappointed and avoid any further deployments or firmware updates from Fedora.

Oracle bought Sun to get the ZFS file system. They have been porting ZFS features into Btrfs. The head of the Ext4 file system project, an Oracle employee, said that Ext4 was just a temporary stepping stone on the way to Btrfs.

A primary motivation for Oracle’s acquisition of Sun was to obtain and control the IP for the MySQL database project, which had been obliterating a large amount of Oracle DB customer licensing and dominating the OSS database landscape.

A primary motivation for Oracle’s acquisition of Sun was to obtain their hardware platform for running Oracle DB, Oracle J2EE applications, and their vast middleware stacks. This was everything related to the UltraSparc architecture, Sun Fire line of Opteron based servers, all of the high performance storage technology, and all of Sun’s enterprise contracts with the US military, US gov, various nation states, and all manner of other high-end “serious business” service compute providers.

Sun created Java, nearly all of Oracle’s entire stack requires Java, and Larry wanted that as a foundational piece of IP. At the time of the acquisition, the world ran on Java and it was the most profitable and most widely deployed programming language as well as server runtime environment (JDK, JRE). This aspect is also related to various IP lawsuits around Java, see Oracle vs Google later on.

ReiserFS features were ported into Btrfs, not ZFS features. ZFS has always dominated enterprise storage, never btrfs, never ReiserFS. ZFS in an entirely different class of quality, performance, scalability, reliability, and global market penetration.

RedHat was tired of supporting Btrfs after too many customers reported critical data-loss incidents, which could only serve to harm the company’s reputation for stability and technical excellence.

ZFS continues to be amazing, Btrfs continues to suck, Oracle still has their own original ZFS source tree separate from the OpenZFS project, Oracle still sells UltraSparc architecture, Oracle still owns Java IP, Oracle still doesn’t prioritize Btrfs, nothing important runs on Btrfs.

The only recent news about btrfs and Redhat, which invalidates the original claims about Redhat vs Oracle conspiracy, is that Fedora Linux 33 has Btrfs as a default option along with XFS and Ext4, and we all know that Fedora is upstream development for RedHat Enterprise Linux.

Some timeline details:

• 2007, Btrfs concepts and originated at IBM
• 2008, rip-off of ReiserFS features started by developer at Oracle, who was a former dev of ReiserFS (before Hans Reiser murdered wife, tried to cover it up, landed in prison, and ReiserFS project basically died too)
• Btrfs continues to be experimental for years, cannot complete with ZFS on any important metrics
• 2008, Oracle Exadata v1 - a specialized hardware platform for running Oracle databases on HP-UX with Itanium procs
• 2009, Oracle Exadata v2 - now running on Sun hardware, using modified Sun Fire systems with AMD Opteron procs
• 2010, Oracle acquired Sun which had recently acquired MySQL
• 2011, Redhat offers Btrfs as experimental in RHEL 6 beta
• 2012, Btrfs main developer leaves Oracle for Fusion-IO
• 2012, Oracle and SUSE offer Btrfs in their distro for prod use
• 2013, Btrfs main developer leaves Fusion-IO for Facebook
• 2017, Redhat removes Btrfs from inclusion to RHEL 8, switching to XFS (originally from SGI)
• 2018, Redhat acquisition by IBM

Some enjoyable references:

• en.wikipedia.org/wiki/Btrf…
• en.wikipedia.org/wiki/Acqu…
• en.wikipedia.org/wiki/Hans…
• en.wikipedia.org/wiki/ZFS
• en.wikipedia.org/wiki/XFS
• en.wikipedia.org/wiki/Orac…
• www.oracle.com/webfolder…
• flashdba.com/history-o…