Also a lot of talk about Github being compromised and having 3,000+ internal repos exfiltrated. Surely GitHub has 100% Code Coverage Testing and End-to-End Test (Work)Flows, right? No, they do not.

This post is focused on E2ET (End-to-End-Testing) of application service roles and machine provisioning automation. So, let’s start by seeing just exactly WTF I’m referring to with this type of testing in the first place.

CI Systems as a “Failure Mode” Example

In the reporting system here we use a host type definition of K10 which is mostly arbitrary, but the root story is an irrelevant tangent, used in this infrastructure as an example of a standdalone system which requires specific attention to failure state definitions, stateful analysis of gated operations, and is used for iterative looped-based resolution testing.

The K10 host is assigned to a role which features CPU-native cpuid2cpuflags architecture optimized compiler flags. We never use generic kernels, or generic machines, because efficiency and optimization in the industry is basically a broken afterthought and it gets old after a while - dealing with non-optimized non-attentitive systems engineering approaches featuring inefficient broken theories based the shrug-method, “just make it auto-scaling and put more kubes or something?”

Network Booting the Baseline OS

The K10 baseline system boots over the network using iPXE/HTTPv4, while other devices require chainload PXE/TFTP -> iPXE/HTTPv4 boot images. This facilitate a system of host-based discovery and domain authentication via initrd embedded kernel modules + connection parameters for FreeIPA/SSSD for RBAC + AAA using MAC and PKE enrollment. We also do this for FreeBSD hosts, with some minor differences.

The K10 is also connected to local-rack PDUs with remote power cycling via either ACPI signal for “soft shutdown” or via its switched-outlet port. The PDUs are controlled by the Command-&-Control Server using SNMPv3 user + group authentication via its infrastrcture management LAN connection with an auth-hook to FreeIPA via RADIUS.

Let’s See Some Reports!

Having covered some blathering about provisioning systems, we can move on to agent-based querying for SITREP , which is executed on our in-house LLM structured query execution tooling. That whole chain of events required writing a new “Agentic Forge Control Plane”, which was enjoyable most of the time… and it has become a baseline distributed comms system for “Project Coherent Storage” which is currently managing a bit over 1PB of data on OpenZFS.

Earlier this year I paused for some PTO and rewrote all of my formerly Ansible driven and automated Infrastructure as Code workflows to support an upcoming revision to our dual-continent presence at six datacenter locations. It’s occasionally and very tentatively being referred to as "minimalist-maxxing" which is mostly pure nonsense where grammatical excellence is concerned, but ultimately accomplishes the following:

• Gentoo w/ Optimized Stage4, LLVM/Clang, OpenRC
• Minimalist service containers using our Stage4 baseline, with service roles layered on top. Try it out:

No, these were not Kubernetes, not Helm, not Cloudslop. Sometimes translates into Infrastructure as a Service, for enablement of event-driven architecture with a hybrid-local service enablement for LLM predictive action controls.

That’s a lot of words to say, “I make the code do the things with the signals processing and patterns and stuff.” Wait wait.. no, this is not going to cover anything about “N~less Computing”.

Stop Blathering About “Serverless Computing”

This term… it is bull-sheet top to bottom front to back.

• Serverless does not mean “no servers.” Cloud providers rely on physical infrastructure to run your code and your applications.
• The term is marketing shorthand, not architectural reality. Serverless means server management is abstracted away from developers, not eliminated, nor emancipated.
• It does not mean “no infrastructure.” Provisioning, patching, security, scaling, and runtime management are simply shifted to the provider, just like the ‘ol days where users had websites on shared hosting systems (and they still do) without having to handle, manage, or maintain the infrastructure.
• It is not always cheaper. Serverless can be cost-effective for sporadic or bursty workloads, but long-running or compute-heavy jobs may cost more than VMs or dedicated servers.
• It is not a universal replacement. Serverless differs from IaaS and PaaS by emphasizing event-driven execution, automatic scaling, and scale-to-zero behavior when idle; lies, all marketing lies. Follow the money.

[host K10]: currently has transient AAA validation, not E2ET PASS

• APC PDU AP7901 initiated hard-reboot proved the current netboot rootfs does not persist SSSD/IPA state.

Under this definition, K10 must rebuild rootfs or disk-install with aaa-domain-client, reboot, then pass the full post-boot E2ET suite before it can unblock xen-sun99-x12spl-099108.rfc1918.host reimage confidence.

Standard Example ‘K10 Automation Report’

# E2ET Definition
End to End Testing / E2ET should mean: a reproducible host acceptance pipeline that proves a machine can move from inventory intent to installed, rebooted, validated, scored, documented, and release-eligible state.

For RFC99, a host is not E2ET-passed just because we fixed it live. It passes only after the fix is represented in source-of-truth and survives the full lifecycle.

## Policy Definition
For validation hosts like `K10`:

1. Fix live only to recover evidence or confirm root cause.
2. Backport the fix into repo-owned profile, role, manifest, package list, NetBox/DNS/IPAM metadata, and docs.
3. Reinstall or rebuild/reboot through the intended provisioning path.
4. Run post-boot validation.
5. Generate a conformance report.
6. Only then mark the host RC/GA-capable.

> A one-off live fix can be marked transient validated, but never E2ET passed.

## E2ET Stages

### Host Pipeline

1. **Inventory Gate**: NetBox/IPAM/DNS/Ansible host metadata matches expected MACs, interfaces, service IPs, PDU mapping, console path, boot protocol.
2.** Provisioning Gate**: iPXE or PXE-chainload path works, installer runs, ZFSBootMenu/bootfs/rootfs pools validate, OS install completes.
3. **First Boot Gate**: kernel cmdline, serial console, hostname/FQDN, SSH, time sync, logging, package profile, OpenRC services.
4. **Platform Gate**: CPU model, RAM, disks, NVDIMM/Optane if present, sysfs, kernel modules, firmware, CVE mitigation status.
5. **Network Gate**: management interface, service interfaces, VLANs, routes, DNS forward/reverse, NetBox consistency, nmap readiness.
6. **Storage Gate**: ZFS pools, NFSv3/v4, NFS-RDMA, Ceph, iSER, iSCSI, sshfs as applicable.
7. **Identity Gate**: FreeIPA/SSSD/PAM/SSH keys, UID/GID consistency, sudo policy, offline cache, break-glass account.
8. **Service Gate**: required services running, no unexpected failed services, role-specific smoke/functional checks.
9. **Performance Gate**: CPU, memory, disk, network, build/distributed compile benchmarks against baseline.
10. **Conformance Report**: JSON + Markdown + optional JUnit output with hard-gate pass/fail and weighted score.

### Scoring Model
Use hard gates plus weighted scoring.

### Hard Fails
  - Cannot boot.
  - Cannot SSH via management path.
  - IPAM/DNS mismatch for primary identity.
  - Root pool invalid.
  - SSSD missing on an AAA-required profile.
  - Required service failed.

### Score Zones
  - p60: minimally usable, not release candidate.
  - p80: acceptable lab host.
  - p90: RC candidate.
  - p95: GA for normal infra.
  - p99: production-critical or rebuild-template quality.

## Policyinctions
While there is overlap with "conformance tiers", this functionally is separate from "statistical confidence" until we have sufficient historical run-data to compute "Real-Number Percentiles" and data-driven pattern-based repeatably-provable "Statistically Significant" probability assessments.

### K10 Applied Meaning
The host definition for `K10` currently has transient AAA (not AAA DNS: `"ACK & AGREE & APPROVE"`) validation, not E2ET pass.

#### **Actioned Event**
Its connected PDU port *(SKU: APC AP7901)* reboot proved the current `netboot rootfs` does not persist `SSSD/IPA` state. 

#### **Action Rephase**
Host definition `K10` requires a rebuild of rootfs or disk-install with `aaa-domain-client`, another reboot, then pass the full `Post-Boot E2ET` test suite before it can unblock dependency-trees. 

Otherwise K10 risks being labeled a permanent `reimage-processing confidence blocker`, which leads to hardware decomission.

## K10 Dead-Reckoning
Previously we defined the `RFC99` & `SUN99` workflow as `K10 Host E2ET Acceptance Pipeline v0.1`, for this we'll re-implement the process using block-notation in Ansible + Python with the new report tooling, and enable `cicd-rfc99-jenkins-099199` for orchestrating it once the checks are stable.

Choose your failure mode and you will find it quickly.

Please Skip the Philosophy

A valid request. Perhaps you don’t care about this verbosity of thought, or considerations therein, or maybe you’re aware of the reprisals and follies and want to move on to the engineering discussion? If so, simply scroll down to the section titled, “How Does This Relate to Memory”. Note that in doing so you will miss a blurb from Office Space.

Endeavor to Endure the Philosophy

A fair degree of the human experience involves a process by which those of “early-phase cognition” attempt to self-obfuscate their “desire for desire”, attempting to live a life of nebulous and fleeting happiness, non-blissfully and unaware that it is this desire to feel and experience desire itself which has cornered their experience into a lifetime of endless searching, yearning, wanting, never satiated, eventually growing old and stale and losing their imaginative or creative or physical edge.

This cycle of The Great Chase is one basis of Sisiphean frustration; one which - until it is surpassed - which requires being acknowledged, accepted, and integrated into their forward movement

Some understand that to exist with desire, that all-controlling force, seemingly inescapable force which presses down upon the rock, forever trapping your perspective into the wheel of the unknown yet forever traveled, repeating mistakes which others warned against, refusing awareness of one’s own misfortunate mentality and guiding-line towards an inevitable failure; experiences from which all lessons are never to be learned.

Go ahead. Do it. Don’t learn from others. Do everything yourself, do it with the veracity by which you have become convinced is correct - do everything as though this is the first time anyone has ever tried to achieve the new shiny thing. Surely, this must be the path towards truth, self-realization, self-actualization, self-assurance, and that desperation of desire for freedom from desire to desire everything… and then… surely success is ensured?

No, that modality of mentality would achieve its opposite effect, and it would be a waste of everyone’s time; because despite the ego-driven main-character syndrome which has become so performatively boring in the past two decades of social media, it’s not all about You, or Me, it’s about Everyone alive and even those who we have loved who died.

Mr. Gibbons

Peter Gibbons:
It’s not just about me and my dream of doing nothing. It’s about all of us… Michael, we don’t have a lot of time on this earth!

We weren’t meant to spend it this way. Human beings were not meant to sit in little cubicles staring at computer screens all day, filling out useless forms and listening to eight different bosses drone on about about mission statements.

Michael Bolton:
I told those fudge-packers I liked Michael Bolton’s music.

Peter Gibbons:
Oh, That is not right, Michael.

Purposeless self-serving inefficiency which is predicated on assured failure wastes time that humanity does not have - we do not have time for oppositional defiance induced mediocrity. Stop the gamification.

The Mindset of Being Simply Wrong

Instead of repeating the pattern described above, grow the fuck up and learn how to effectively learn from others, to learn from the past, to learn using a reinforcement framework of concept-action-processing. “Thems may be big words”, but this is not a complicated concept. One does not have to choose the path of stupidity; ignorance - we’re all ignorant in all manner of ways and that’s inescapable; however when one ignores the data, ignores the lesson, and chooses the path before them via oppositional defiance, that’s called “being stupid”. It’s stupid not to learn from others.

How Does This Relate to Memory

Memory is everything. The conscious experience, the neural network running inside our brains, the execution of “The Default Mode Network”, requires effective memory - short term a well as consolidation to long-term memory.

So, when the industry was informed of the untimely discontinuation of Intel’s Optane NVDIMMs… we were all screwed. Until Q2 2026. Now it’s Q2 and finally CXL non-volatile memory is available in sufficiently equivalent density, performance, and lifespan to exceed Optane’s role, and it is available in PCIe Gen-4 and Gen-5 and soon… Gen-6.

An Introduction to CXL

• It’s not HBM, it’s not 3D-Xpoint, it’s not SRAM, nor vRAM
• CXL is packaged in PCIe, U.3, and EDSFF formats
• CXL offers dense capacity, low-latency, and exceptional I/O
• CXL is made for Ai/HPC/OLTP memory acceleration
• CXL is the new hotness, it is the new standard, it is performance

HPC Cluster Storage Acceleration

CXL is being built into (not next-gen, now-gen) storage appliances and network protocol accelerator systems used in High Performance Computing, Ai/ML which leverages HPC tech, and Supercomputers.

CXL in Telco DSP as Protocol Caches

CXL is perfectly suited for digital signals processing on a range of network protocols and their intra-chassis and inter-chassis RDMA driven networks. There’s one person who I know that hates it when I bring up RDMA - truly an example of being willfully ignorant. Moving on…

CXL Memory in Telco ATAC Chassis

• 14 slots in 19-inch rack ≈→ ∆ {$N total trays per 42U ≈ μ19}
• 16 slots in 23-inch ETSI rack ≈→ ∆ {$N total trays per 42U ≈ μ23}
• Chassis heights from 4U to 15U ≈↑ ∆ [[ μ4, μ19, μ23 ]]
• Telco grade Power & Cooling Efficiency & Redundancy ++
• Mil-DoD Compliant Physical & Operational Security ++
• Industrial NEBS Environmental Operating Ranges ++
• Zero-Downtime Serviceability & Upgrade/Maint ++
• Intra+Inter-Chassis Redundancy (Modular Scale) ++
• Geo-Deploy Designed for regional XOR continental mesh ++

So, Why Should We Listen to Her?

Becausee I’ve been building and iterating these types of storage architectures for nearly twenty years of my career. If you want to do it all from scratch or buy this type of storage performance for 10x the cost, go ahead; the market exists and you can be the big winner.

However, if you don’t want to pay a premium for non-OpenSource storage clusters which are capable of serving and scaling from 10s to 100s of Petabytes, then this is the direction by which engineering solutions arrive at the datacenter. My storage designs range from Oracle’s Exadata & Exascale clusters which have been adapted to run non-standard protocols, to fully purpose-built data-acceleration clusters which use COTS building blocks to provide similar performance and scale at a fraction of the cost (not including consulting, engineering scope, and top-to-bottom fully-automated storage architecture deployments).

As networking has improved, NAND designs have advanced, and costs have reduced on $/TB and $/Watt… yet always… always the storage sizes grow in density and capabilities. This is natural and expected.

Project Coherent Storage, an Evolution

Six primary iterations, tracking the advancement of NAND acceleration, Network Acceleration, Protocol Offloading, Non-Volarile RAM, and now we see CXL capable of being deploying into reference architectures by which all others may be measured.

• Project Coherent Storage v1, (2010-2012)
• Project Coherent Storage v2, (2011-2015)
• Project Coherent NAND, (CMPLX.io) (2015-2017)
• Project Coherent IO-Max v1, w/ Optane PMem (2020-2024)
• Project Coherent IO-Max v2, w/ Optane PMem (2024-2025)
• Project Coherent Storage v3, w/ CXL AIC/U3 (2026)

Development Details - Concise Timeline

A little bit of engineering nerdery for those who care… and btw, the community myth that ZFS cannot fulfill a role as a multi-node clustered filesystem are wrong.

What About the “ZFS Can’t Cluster” Myth?

The “Clustered ZFS” operating modality for multi-controller interactions (not limited to A/B active/passive or even active/active with modified pool-async hooks) requires advanced network hardware and engineering experience which most are not aware of, and so the myth perpetuates; but it is wrong and it has been wrong for as long as SAS drives have had dual-heads and shelves have had 4+ SFF connectors ready for multi-multipath-chaining - and enterprise NVMe drives are also available in this arrangement for the same purpose.

Project Coherent Storage v1 (2010-2012)

1. Coherence Grid (DDR3 (Tier1)) + Custom Relational-Vector DBMS
2. Sunfire ZFS controllers w. Multi-Protocol-Net Interlinks (IB, IPoIB, mNFS+, FC, FCoE)
3. Geo-Dist-Nodes w. DDR3-PCIe SuperCap “Hot Cache Layer” (Tier2), w. Sunfire X4600/M2 @ 64 DIMMs Hotswap
4. Geo-Dist-Nodes w. SAS SSDs (Tier3) + SAS Spinner (Tier4)
5. Controller A/B+ nodes w. SAS-HBA to LTO-N (Tier5)

Project Coherent Storage v2 (2011-2015)

1. Coherence Grid (DDR3 (Tier1)) + Relational-Vector DBMS
2. Sunfire ZFS controllers w. Multi-Protocol-Net Interlinks (IB, IP Native L2/L3, mNFS+, FCoE)
3. Geo-Dist-Nodes w. FusionIO Stacked “Hot Cache Layer” (Tier2)
4. Geo-Dist-Nodes w. SAS SSDs (Tier3) + SAS Spinner (Tier4)
5. Optional A/B nodes w. SAS-HBA to LTO-N (Tier5)

Project Coherent NAND - CMPLX.io (2015-2017)

1. Coherence Grid DDR4 (Tier1) + Relational-Vector DBMS
2. Native ZFS controllers w. Multi-Protocol-Net Interlinks (IB, IP Native L2/L3, mNFS+, iSER)
3. Geo-Dist-Nodes w. NAND Stacked “Hot Cache Layer” (Tier2)
4. Geo-Dist-Nodes w. SAS3 SSDs (Tier3) + SAS3 Spinner (Tier4) [+ optional SAS3 HBA to LTO for Iron Mountain (Tier5)]
5. This architecture was a basis for one of my former startups, which was sold to private equity and built-upon for their HFT clusters under very tight NDA + Confidentiality Contract

Project Coherent IO-Max Storage Architectures v1, 2, 3

These storage architectures share core foundations, with iterations upward as Non-Volatile Memory was introduced on a wider scale into my designs. The initial design for v1 featured the first generation of Intel Optane 3D-XPoint (Series 100), v2 with Series-200, and now CXL taking the place of Optane PMem SKUs.

The details of each design from v1 and v2 may be expanded upon later, simply for historical purposes, and I have other things to do right now… but if you’ve made it this far into the post and want to see the specification ADRs and related research, which is RAG-pipelined and iteratively deployed on real hardware for revisions and regression testing to validate LLM Inference usage & advancements with the now-opensource Coherence-CE version… scroll down for a link to v3’s repo.

• Project Coherent IO-Max v1, (2020-2023)
• Project Coherent IO-Max v2, (2023-2025)
• Project Coherent IO-Max v3,CXL (2026)

Let’s inspect one of my favorites lately, Qwen3-235B. This is not a model that one commonly sees running outside of large compute clusters and supercomputers, because the 235 billion parameters require substantially more resources than most infrastructures have available. Often when we see Qwen3 in-use on not-supercomputers, it’s the 32B variant, which can run well enough on a 16GB+ vRAM capable GPU. Querying the models on Huggingface and one will see there are 32B on the low-side all the way to 480B for the supercomputers and research clusters, and many customized ones built from the originals.

Parameter quantity aside, as we’re not focused on that at the moment, let’s look at the primary high-level differences, as well as low-level technical differences, between LLM variants like “Coder”, “Thinking”, “Instruct”. Over at the Cerebras Inference Cloud we can use the following types.

High Level Differences

• Base Model: Generalist, no task specialization.
• Coder: Focuses on programming efficiency (e.g., auto-completion, error fixing).
• Instruct: Prioritizes user safety and conversational alignment (e.g., chatbots).
• Thinking: Targets structured reasoning (e.g., math proofs, logical puzzles).

Low Level Differences

• Tokenizer: Coder extends vocabulary for code symbols (e.g., def, ->), while others reuse the base tokenizer.
• Fine-Tuning: Instruct uses RLHF for alignment; Thinking uses process-based rewards for intermediate steps.
• Decoding: Thinking enforces iterative step generation via modified KV-cache management, unlike autoregressive defaults.
• Loss Functions: Coder weights syntax validity; Thinking penalizes incorrect reasoning paths early.

Use Case Mapping

• Need code generation? → Coder
• Building a chatbot? → Instruct
• Solving multi-step problems? → Thinking
• Researching foundational capabilities? → Base

Visualizing Decisions via Flowchart

Visualizes the evolution from base model to specialized variants, emphasizing technical divergence points, which you can see in the live rendering panel over at Mermaid.Live

graph TD
    A[Qwen3-235B Base Model] -->|Trained on general web text| B[Qwen3-235B]
    A -->|+ Code corpus & tokenizer extension| C[Qwen3-235B-Coder]
    A -->|+ SFT/RLHF on instructions| D[Qwen3-235B-Instruct]
    A -->|+ Reasoning datasets & iterative decoding| E[Qwen3-235B-Thinking]

    B --> B1["Standard cross-entropy loss\nNeutral inference"]
    C --> C1["Code-specific tokens\nSyntax-aware embeddings"]
    D --> D1["RLHF alignment\nSafety filters"]
    E --> E1["Reasoning head\nStep-by-step CoT output"]

    classDef base fill:#e6f7ff,stroke:#1890ff;
    classDef variant fill:#f6ffed,stroke:#52c41a;
    class A,B base;
    class C,D,E variant;