Data caching at The Edge isn’t always from Fastter or Cuntter or Cloudstter or whichever CDN happens to be serving your IG, Safeway, NFL, Reddit, Anthropology, or 338% of the internet’s concurrent traffic being pointless traversals like jQuery minified & brotli-fied and sent back and forth endlessly through Internet Exchanges.

Edge caching is not limited to Anthropic or OAI or whoever else’s Load-Balancer API-side “prompt cache (not yet! after LB!)” packets which “those people” are constantly complaining about, nor the hyperscalers who whitebox big CDN networks under their own headline IP.. those are not what I’m talking about when I talk about Edge Caches.

These little embedded devices which collect climate metrics, air quality data, assembly robotics, municipal awareness and telemetry data, and everything else from a very long list… they need read-heavy churn-heavy persistent storage on a filesystem optimized for tiny sized industrial NAND.

More examples for the ones of all sizes making modern life possible: that ATM you went to, the parking lot video surveillance systems, traffic light coordinator nodes, telco edge ONT routers and OOBs with SDNs, and a thousand other examples abound that are functionally ignored.

Also, robots. And factory hardware. And factories making robots that make tiny factories? Hopefully so.

Embeds Reduce CAPEX & OPEX Efficiently

So, those embedded devices usually are designed for efficiency, while not too often designed to be hammering on NAND for write-heavy storage workloads. This has caveats: devices explicitly designed for write heavy I/O on purpose, and therefore not what I’m referring to, and cases where the provisioning and/or configuration stages of fleet deployment are improperly aligned to hardware realities.

Why? Low wattage, moderate caches, and a desire to not burn out NAND when the device is located in a difficult-to-get-to location for service techs working on hardware replacement.

Most of all - we want efficient storage for embedded devices so that their operationally collected data can be stored in memory with a ECC write-back cache with “operationally consistent” garbage collection and non-invasive wear-leveling, all while the transactions are paged/queued to F2FS before async’ing upstream to a centralized data collector for aggregation.

Predator Drones Too?

Yes and no, they send their comms to LEOSAT before humanity’s “furthest from the menial and meaningless non-Elysium people” edge devices cache, aggregate, and compute roll-up analytics with inline encryption and compression via DPU offload streaming.. all the way down to LANDSAT.

Edge devices have their own little ecosystem separate from the manner in which datacenter storage network-enabled hardware operates. All by design, and always room for improvement.

Many people consider this boring tech, and would rather move fast and break things — your time is limited, people are sick of perfectly workable systems breaking all the fuckin time and they’re tired of being A/B tested on in production — doing those things makes your company’s testing policy look like an asshole. Everyone, stop being assholes. Thanks!

These Things - Why Should We Care?

You like your cell phone? How about GPS? SAT-NAV? Skylink? Drones operating up above withour your awareness? How about sector antennas for the police and emergency services? Embedded systems are streaming, caching, processing, and relaying data all over the globe and up into LEO constellations.

You don’t have to care, and you shouldn’t have to care about FSFS or drones unless you work with that side of tech, or its adjacent use cases - like living in the modern world. Everyone should care because knowledge is fundamentally important, if for no other reason at all. Do you know how the light-switch on the wall works? Same deal. Stop making excuses for saying you don’t know or don’t care or you will be left behind. No one is saving us but ourselves, and we must have knowledge to do so.

Recent Adventures with Standards-Definitions

I love automation. In part this is because I prefer efficiency over repetitive manual bullshit. Efficiently handling requirements means more time to do what I love:

• Hanging out by the pool with a tablet or waterproof e-ink reader.
  Way back in 2018 my e-book tally was 62 novels over the calendar year. That number has dropped off recently for trivial reasons, but the point remains that I love to read, and I love to lay out by the pool.
• What I do not love:
  Wasting time when I could be at the pool. ♥‿♥

there’s only room for two hats in my life - and this is the one for sunning

• What I’ll never not love:
  The poem on my torso, dedicated to the manner in which my departed twin lived & loved.

Standards for NAND? Standards for SD-NAND!

To the point, here’s a script that had to be blasted out a while back, while imaging so many too many industrial-spec SD/TF-cards for a fleet of distibuted data collectors. These fun little RK3588 based systems have a NPU and only an eMMC drive for the OS.

While the eMMC in question has decent specs, if it fails then it’s sadly more expensive to de-solder and re-solder and re-image the OS on eMMC, and so on these systems we use the SD NAND for the following data:

• Ephemeral scratch space and logs (eg: /var/log, /var/tmp, /tmp - nope, these go to tmpfs)
• Cache directories which are aptly named .cache or .Cache סּ_סּ
• Files which must be staged locally but have no business being on the OS mount points.
• Several other use cases that I don’t feel like describing right now; so here’s the discussion for those who love to know more more more.

### F2FS - Embedded Filesystem Performance Options

- Ref: https://docs.kernel.org/filesystems/f2fs.html

#### Option Flags and Reasoning

##### For High-Churn I/O
This is less ideal, but sometimes a `tmpfs` overlay isn't an option, so it's better than burning out the eMMC. A good industrial SD card has ECC, wear-leveling, checksum validations, write-amplification controls, etc. Don't care for those features? You may one day...

Example Writes: data persistent log-sync, package builds on-device, container writable layers, time-series compressible metrics, telemetry buffers, or small-file caches, opt for the non-default `-o 10` flag. 

| Option                      | Rationale              |
| --------------------------- | -----------------------|
| `-l DATA`                   | Stable label for `/etc/fstab`; avoids relying on device enumeration order. UUID is an alternative, but requires programmatic parsing of `blkid` output, or manually specifying a UUID to pass to the FS creation process.. or you can simpy use a GPT label. |
| `-o 5`                      | Reserves 5% of the F2FS volume for the F2FS cleaner. `mkfs.f2fs` documents this as hidden overprovision space used by the cleaner. |
| `-t 1`                      | Performs discard/TRIM at format time; defaults discard to enabled, but some card readers/cards mishandle it; use `-t 0` if format stalls or logs errors |
| `extra_attr`                | Required by several modern F2FS features. |
| `inode_checksum`            | Adds inode metadata checksums; useful for removable flash and unattended systems. |
| `sb_checksum`               | Adds superblock checksum support. |
| `flexible_inline_xattr`     | Modern xattr layout; useful with `extra_attr` and future feature expansion. |
| `inode_crtime`              | Adds inode creation-time support; low cost, useful for forensic/debug visibility. |
| `lost_found`                | Creates lost+found support for recovery workflows. |
| `compression`               | Good for compressible write-heavy data; poor fit for already-compressed media, model files, archives, database pages, and random-write workloads. |
| `compress_algorithm=zstd:3` | Validate performance of zstd compression options before using for production purposes. |
| `compress_log_size=%u`      | Support configuring compress cluster size. The size will be `4KB * (1 << %u)`. The default and minimum sizes are 16KB. |
| `checkpoint_merge`          | Self-describing (or check the refs)  |
| `nocheckpoint_merge`        | Disable checkpoint merge feature. |

- `checkpoint=%s[:%u[%]]`: Set to "disable" to turn off checkpointing. Set to "enable" to re-enable checkpointing. Is enabled by default. 

- `checkpoint_merge`
When checkpoint is enabled, this can be used to create a kernel daemon and make it to merge concurrent checkpoint requests as much as possible to eliminate redundant checkpoint issues. Plus, we can eliminate the sluggish issue caused by slow checkpoint operation when the checkpoint is done in a process context in a cgroup having low i/o budget and cpu shares.

- `compress_algorithm=%s`
- `compress_algorithm=lzo-rle`

Control compress algorithm, currently f2fs supports "lzo", "lz4", "zstd" and "lzo-rle" algorithm.

- `compress_algorithm=%s:%d`

Control compress algorithm and its compress level.. only "lz4" and "zstd" support compress level config and those algos may be in your kernel config and may not.. but you'll find out quickly if you don't already know the answer.

========= =========== algorithm level range ========= =========== lz4 3 - 16 zstd 1 - 22 ========= ===========

- `compress_log_size=%u`
Support configuring compress cluster size. The size will be `4KB * (1 << %u)`. The default and minimum sizes are 16KB.

- `compress_extension=%s`

Support adding specified extension, so that f2fs can enable compression on those corresponding files.
If all files with `.ext` has high compression rate, we can set the `.ext` on compression extension list and enable compression on these file by default rather than to enable it via ioctl.

For other files, we can still enable compression via ioctl. Note that, there is one reserved special extension `*`, it can be set to enable compression for all files.

- `nocompress_extension=%s`

Support adding specified extension, so that f2fs can disable compression on those corresponding files, just contrary to compression extension.
If you know exactly which files cannot be compressed, you can use this.

> The same extension name can’t appear in both compress and nocompress extension at the same time.

If the compress extension specifies all files, the types specified by the nocompress extension will be treated as special cases and will not be compressed.
After add nocompress_extension, the priority should be: `dir_flag < comp_extention,nocompress_extension < comp_file_flag,no_comp_file_flag`.

- `compress_chksum`
Support verifying chksum of raw data in compressed cluster.

- `compress_mode=%s`
Control file compression mode: "fs" and "user". In "fs" mode (default), f2fs does automatic compression on the compression enabled files. In "user" mode, f2fs disables the automaic compression and gives the user discretion of choosing the target file and the timing.

The user can do manual compression/decompression on the compression enabled files using ioctls.

- `compress_cache`
Support to use address space of a filesystem managed inode to cache compressed block, in order to improve cache hit ratio of random read.

- `inlinecrypt`
When possible, encrypt/decrypt the contents of encrypted files using the blk-crypto framework rather than filesystem-layer encryption.
This allows the use of inline encryption hardware. The on-disk format is unaffected.

- `atgc`
Enable age-threshold garbage collection, it provides high effectiveness and efficiency on background GC. No one likes garbage.

- `discard_unit=%s`
Control discard unit, the argument can be "block", "segment" and "section", issued discard command’s offset/size will be aligned to the unit, by default, "`discard_unit=block`" is set, so that small discard functionality is enabled.

For blkzoned device, "`discard_unit=section`" will be set by default, it is helpful for large sized (fucccck off with SMR, less fucks to ZNS) devices to reduce memory cost by getting rid of fs metadata supports small discard (ouch, ever get burned by SMR? yeah... just don't bother).

- `age_extent_cache`
Enable an age extent cache based on rb-tree. It records data block update frequency of the extent per inode, in order to provide better temperature hints for data block allocation.

- `errors=%s`
Specify f2fs behavior on critical errors. This supports modes: "`panic`", "`continue`" and "`remount-ro`", respectively, trigger panic immediately, continue without doing anything, and remount the partition in read-only mode.

By default it uses "`continue`" mode, not my choice.

====================== ===============
mode continue remount-ro panic ====================== ===============
access ops normal normal N/A syscall errors -EIO -EROFS N/A mount option rw ro N/A pending dir write keep keep N/A pending non-dir write drop keep N/A pending node write drop keep N/A pending meta write keep keep N/A ====================== ===============

- `nat_bits`
Enable nat_bits feature to enhance full/empty nat blocks access, by default it’s disabled.

- `lookup_mode=%s`
Control the directory lookup behavior for case-folded directories. This option has no effect on directories that do not have the casefold feature enabled.

======================================== Value Description

perf (Default) Enforces a hash-only lookup. The linear search fallback is always disabled, ignoring the on-disk flag. compat Enables the linear search fallback for compatibility with directory entries created by older kernel that used a different case-folding algorithm. This mode ignores the on-disk flag. auto F2FS determines the mode based on the on-disk SB_ENC_NO_COMPAT_FALLBACK_FL flag.

----

#### Benchmarking F2FS Compression

Always validate the compression algo used before using those options, as performance of one algo vs anther can vary widely across different CPU generation and architectures, as well as for hosts with algo/hash compression offloading engines. We can cover benchmarking these in another script.

> lzo, lz4, zstd, lzo-rle algorithms

##### Create Partitions

DEV=/dev/sdX sgdisk –zap-all “$DEV” sgdisk -a 2048 -n 1:2048:-16G -t 1:8300 -c 1:DATA “$DEV” partprobe “$DEV”

##### Create FS

mkfs.f2fs -f
-l DATA
-o 10
-t 1
-O extra_attr,inode_checksum,sb_checksum,flexible_inline_xattr,inode_crtime,lost_found,compression,compress_algorithm=zstd:3 “${DEV}1”

#### FS Tab Entry

##### Conservative defaults

LABEL=DATA /srv/data f2fs rw,noatime,nodiratime,lazytime,nofail,background_gc=on,gc_merge,checkpoint_merge,flush_merge,mode=adaptive,fsync_mode=posix,nodiscard 0 0

##### Compression-capable variant

LABEL=DATA /srv/data f2fs rw,noatime,nodiratime,lazytime,nofail,background_gc=on,gc_merge,checkpoint_merge,flush_merge,mode=adaptive,fsync_mode=posix,nodiscard,compress_algorithm=lz4,compress_chksum,compress_mode=fs 0 0

#### Selectively Mark Compressible Directories
Do not enable compression broadly for LLM model files, videos, archives, SQLite databases, or opaque binary blobs unless benchmarks show a benefit.

mkdir -p /srv/data/_logs /srv/data/_caches.compressible chattr +c /srv/data/_logs /srv/data/_cache.compressible

So, Maybe This is Scriptable?

It sure is, or by Ansible playbook, or Salt sync, perhaps even a CFengine golden image.

That’s a lot of blah blah blah. How about that script?

Efficiency, indeed. Here ya go. The benchmarking script is for another day, or you can check out this workload primer about the filesystem from USENIX (w/ link to slide deck), and a benchmarking script which has been customized for F2FS.

#!/usr/bin/env bash
# -*- mode: sh; sh-basic-offset: 4; sh-shell: bash; coding: utf-8; indent-tabs-mode: nil; -*-
# ----------------------------------------------------------------------- #
set -euo pipefail
# ----------------------------------------------------------------------- #
# Purpose: POSIX-syntax wrapper to partition and format a non-boot SD/microSD card as F2FS with optional compression.
# NAND Profile: embedded I/O, kernel 6.1+, ECC industrial spec controller.
# WARNING: this destroys data on the selected block device, so be aware.
# ----------------------------------------------------------------------- #

usage() {
    cat <<'USAGE'
Usage:
  format-f2fs-sd-data.sh -d /dev/DEVICE [-l LABEL] [-r RESERVE_GIB] [-o F2FS_OP_PERCENT] [-c]

Examples:
# ----------------------------------------------------------------------- #
  ./format-f2fs-sd-data.sh -d /dev/sdX -l DATA -r 16 -o 5
  ./format-f2fs-sd-data.sh -d /dev/mmcblk0 -l DATA -r 24 -o 8 -c

Options:
# ----------------------------------------------------------------------- #
  -d  Whole block device to erase and partition: /dev/sdX -OR- /dev/mmcblk0
  -l  F2FS label. Default: DATA
  -r  GiB left unpartitioned at end for card FTL spare area. Default: 16
  -o  F2FS hidden overprovision percentage for the F2FS cleaner. Default: 5
  -c  Enable F2FS compression feature at format time. Mount options and chattr are still required.

Dependencies:
# ----------------------------------------------------------------------- #
  sgdisk, partprobe, mkfs.f2fs, lsblk
# ----------------------------------------------------------------------- #
USAGE
}

# Basic Variables
# ----------------------------------------------------------------------- #
LABEL=DATA
RESERVE_GIB=16
OP_PERCENT=5
COMPRESS=0
DEVICE=

# Define ARGV Mappings
# ----------------------------------------------------------------------- #
while getopts 'd:l:r:o:ch' opt; do
    case "$opt" in
        d) DEVICE=$OPTARG ;;
        l) LABEL=$OPTARG ;;
        r) RESERVE_GIB=$OPTARG ;;
        o) OP_PERCENT=$OPTARG ;;
        c) COMPRESS=1 ;;
        h) usage; exit 0 ;;
        *) usage; exit 2 ;;
    esac
done

# Sanity Checks
# ----------------------------------------------------------------------- #
if [ -z "$DEVICE" ]; then
    usage >&2
    exit 2
fi

case "$DEVICE" in
    /dev/sd?|/dev/sd??|/dev/mmcblk[0-9]|/dev/nvme[0-9]n[0-9]) ;;
    *)
        echo "Refusing suspicious device path: $DEVICE" >&2
        exit 2
        ;;
esac

# Validate Deps & Check Limits
# ----------------------------------------------------------------------- #
for cmd in sgdisk partprobe mkfs.f2fs lsblk; do
    if ! command -v "$cmd" >/dev/null 2>&1; then
        echo "Missing required command: $cmd" >&2
        exit 1
    fi
done

if [ ! -b "$DEVICE" ]; then
    echo "Not a block device: $DEVICE" >&2
    exit 1
fi

case "$RESERVE_GIB" in
    ''|*[!0-9]*) echo "Reserve GiB must be an integer" >&2; exit 2 ;;
esac
case "$OP_PERCENT" in
    ''|*[!0-9]*) echo "F2FS OP percentage must be an integer" >&2; exit 2 ;;
esac

case "$DEVICE" in
    /dev/mmcblk*|/dev/nvme*) PART=${DEVICE}p1 ;;
    *) PART=${DEVICE}1 ;;
esac

# Define Default F2FS Options
# ----------------------------------------------------------------------- #
FEATURES='extra_attr,inode_checksum,sb_checksum,flexible_inline_xattr,inode_crtime,lost_found'
if [ "$COMPRESS" -eq 1 ]; then
    FEATURES=${FEATURES},compression
fi

# Make Sure...
# ----------------------------------------------------------------------- #
echo "About to DESTROY all data on: $DEVICE"
lsblk -o NAME,SIZE,MODEL,SERIAL,TRAN,RM,RO,FSTYPE,MOUNTPOINTS "$DEVICE" || true
echo
echo "New partition: $PART"
echo "Label: $LABEL"
echo "Unpartitioned reserve at end: ${RESERVE_GIB} GiB"
echo "F2FS OP: ${OP_PERCENT}%"
echo "F2FS features: $FEATURES"
echo
printf 'Type YES to continue: '
read ans
if [ "$ans" != YES ]; then
    echo "Aborted."
    exit 1
fi

# Best Effort Unmount for Existing Partitions (please don't need this)
# ----------------------------------------------------------------------- #
if command -v findmnt >/dev/null 2>&1; then
    findmnt -rn -S "$DEVICE" -o TARGET 2>/dev/null | while IFS= read -r mp; do
        [ -n "$mp" ] && umount "$mp" || true
    done
fi

# And then...
# ----------------------------------------------------------------------- #
sgdisk --zap-all "$DEVICE"
sgdisk -a 2048 -n "1:2048:-${RESERVE_GIB}G" -t 1:8300 -c "1:${LABEL}" "$DEVICE"
partprobe "$DEVICE" || true
sleep 2

if [ ! -b "$PART" ]; then
    echo "Partition node did not appear: $PART" >&2
    lsblk "$DEVICE" >&2 || true
    exit 1
fi

# OhhhShitHandlers
# ----------------------------------------------------------------------- #
# NOTE: -t 1 performs initial discard when supported; if it's hanging here, rerun manually with -t 0
mkfs.f2fs -f -l "$LABEL" -o "$OP_PERCENT" -t 1 -O "$FEATURES" "$PART" && \
sync

# Display Block Devs
# ----------------------------------------------------------------------- #
lsblk -o NAME,SIZE,MODEL,SERIAL,TRAN,RM,RO,FSTYPE,LABEL,UUID,PARTUUID,MOUNTPOINTS "$DEVICE"

echo "[COMPLETE]"
# ----------------------------------------------------------------------- #
# EOF #
# ----------------------------------------------------------------------- #

More details later… sleepy.

 # NeoCortex Standards, Repo Storage, ZFS Dataset, M70 QAT Fast-Track Plan

  ## Summary

  Implement a non-destructive NeoCortex follow-up that locks
  these policies into code, tests, and docs:

  - Repos live under /opt/repos/<repo>; worktrees live under
    /opt/repos/worktrees/<repo>/<branch>.
  - /root is not a repo/worktree destination; /root/OMGFUCKED is
    temporary quarantine only.
  - New repos must be created from
    yukon-systems/YukonSYS-Standard-Definitions.
  - Every repo must use Git LFS and the standards repo
    .gitattributes byte-for-byte.
  - ZFS dataset creation for /opt/repos becomes reusable
    Ansible/policy logic, based on
    /root/bin/ZFS-M70-Forge_zroot-dataset_opt-repos.sh.
  - M70 QAT rollout uses canary first: build OpenSSL+QAT and
    OpenZFS/QAT binpkgs, validate on M70 canary, then promote.

  ## Key Changes

  - Standards and repo-placement policy
      - Update NeoCortex root policy docs and tests to forbid
        new repo/worktree paths under /root, except documented
        temporary quarantine paths under /root/OMGFUCKED.
      - Update tools/import-source-repos.py defaults from
        /root/<repo> to /opt/repos/<repo>, with
        /root/OMGFUCKED/<repo> accepted only as an explicit
        transitional override.
      - Add a repo-location audit script/test that fails active
        operational docs/scripts using /root/<repo> or
        /root/.config/superpowers/worktrees as live paths.
  - Git LFS and .gitattributes
      - Replace NeoCortex root .gitattributes with the standards
        repo version byte-for-byte.
      - Add a validation script that checks:
          - git lfs version works,
          - filter.lfs.required=true,
          - origin LFS locksverify=true,
          - root .gitattributes hash equals standards source,
          - LFS pre-push hook exists.
      - Document that non-compliant repos must be retrofitted
        before migration/import.
  - ZFS dataset automation
      - Convert
        /root/bin/ZFS-M70-Forge_zroot-dataset_opt-repos.sh into
        reusable NeoCortex Ansible/defaults for managed
        datasets.
      - First managed dataset profile: zroot/opt/repos mounted
        at /opt/repos with the script’s options, including
        compression=zstd-fast-1, dedup=sha256,verify,
        recordsize=256K, xattr=sa, POSIX ACLs, atime=off, and
        reservation=32M.
      - Add a second reusable pattern for per-user home
        datasets, but do not block on the NAS/NFS 777
        investigation; include a permission preflight that
        creates a test directory with umask 0027 and fails if
        mode becomes 0777.
  - M70 QAT binpkg fast-track
      - Extend the existing M70 canary stage4/binpkg lane to
        build a QAT package set:
          - QAT firmware/kernel module readiness package policy,
          - OpenSSL QAT engine/provider package path,
          - OpenZFS package built with QAT support,
          - rollback stock OpenSSL/OpenZFS binpkgs.
      - Add package/use/profile fragments under the existing
        Gentoo stage4 profile area instead of ad hoc host edits.
      - Add canary validation:
          - lspci -nnk sees QAT device,
          - qat_c3xxx and intel_qat load,
          - OpenZFS module parameters expose QAT controls,
          - openssl speed before/after artifacts are captured,
          - root pool imports and mounts after reboot,
          - rollback binpkg path is tested before wider
            promotion.
      - Do not promote to all M70 nodes until canary evidence is
        committed.

  ## Test Plan

  - Local static checks:
      - tests/shell/run-tests.sh
      - new repo-location standards test
      - new Git LFS standards test
      - new ZFS dataset defaults test
      - new M70 QAT package-policy test
      - YAML parse for all workflows/playbooks
      - git diff --check
  - Runtime/dry-run checks:
      - Ansible syntax-check for dataset roles/playbooks.
      - Dry-run dataset plan for /opt/repos confirms current
        zroot/opt/repos matches expected options.
      - M70 canary dry-run emits QAT binpkg build plan without
        mutating packages.
  - Canary acceptance:
      - QAT binpkgs build successfully.
      - M70 canary reboots with root pool intact.
      - QAT modules and OpenZFS QAT controls are present.
      - Benchmark and rollback evidence are recorded under
        NeoCortex audit docs.


  ## Assumptions

  - Canonical repo clone layout is /opt/repos/<repo>.
  - Worktree layout is /opt/repos/worktrees/<repo>/<branch>.
  - /root/OMGFUCKED remains temporary quarantine, not an
    approved steady-state repo location.
  - QAT rollout gate is canary first, not all-M70 immediate.
  - OpenZFS QAT implementation details must be verified against
    current OpenZFS/Gentoo package behavior during
    implementation; source basis includes OpenZFS QAT docs and
    module-parameter docs:
      - https://openzfs.org/wiki/ZFS_Hardware_Acceleration_with_QAT
      - https://openzfs.github.io/openzfs-docs/Performance%20and%20Tuning/Module%20Parameters.html

─ Worked for 22m 41s
───────────────────────────────────────────────────────────
  Implement this plan?
  1. Yes, implement this plan          Switch to Default and
  start coding.
  2. Yes, clear context and implement  Fresh thread. Context:
  53% used.
› 3. No, stay in Plan mode             Continue planning with
the model.



-----------------------
  ## Assumptions

  - Canonical repo clone layout is /opt/repos/<repo>.
  - Worktree layout is /opt/repos/worktrees/<repo>/<branch>.
  - /root/OMGFUCKED remains temporary quarantine, not an
    approved steady-state repo location.
  - QAT rollout gate is canary first, not all-M70 immediate.
  - OpenZFS QAT implementation details must be verified against
    current OpenZFS/Gentoo package behavior during
    implementation; source basis includes OpenZFS
    QAT docs and module-parameter docs:
      - https://openzfs.org/wiki/ZFS_Hardware_Acceleration_with_QAT
      - https://openzfs.github.io/openzfs-docs/Performance%20and%20Tuning/Module%20Parameters.html

─ Worked for 22m 41s
───────────────────────────────────────────────────────────

› 1) We also need to include the control plane fixes and
validations so that mutli-workers comms are fixed. 2) We need to
leverage the SLURM cluster nodes and
  additional distcc build hosts, 3) X12again & FMT2 are
  effectively idle and can be blasted into submission to fast
  track all of these changes. 4) Once the
  M70-canary passes, we need to roll it out to K10 (tear-it-down
  AAA) and validate that type of OS netbooting. 5) Add those
  aspects to the Plans and we're
  looking better. 6) Finally, wrap up ALL changes necessary to
  complete the NeoCortex monorepo migration.

───────────────────────────────────────────────────────────

• I’ll revise the plan around the expanded end-state, but first
I’m doing a read-only pass over NeoCortex’s control-plane,
SLURM/distcc, X12/FMT2/K10, and
  migration-status artifacts so the replacement plan is grounded
  instead of generic.

───────────────────────────────────────────────────────────

• Proposed Plan

  # NeoCortex Standards-Complete Monorepo Migration Plan

  ## Summary

  - Make /opt/repos/<repo> the only canonical repo location; no 
  new checkouts or random repo files under any home directory, 
  including /root.
  - Finish standards adoption: all new repos must originate from 
  YukonSYS-Standard-Definitions; NeoCortex gets retrofitted 
  instead of restarted unless
    retrofit verification fails.
  - Complete NeoCortex monorepo migration with standards baseline, 
  Git LFS policy, control-plane comms fixes, ZFS dataset policy, 
  SLURM/distcc acceleration, M70 QAT canary, FMT2/X12again build 
  capacity, and K10 netboot AAA rollout validation.

  ## Key Changes

  - Repository standards
      - Merge standards policy PR work, then make NeoCortex 
      root .gitattributes byte-for-byte identical to the standards 
      repo version.
      - Update active NeoCortex import/migration tooling to default 
      source paths to /opt/repos/<repo>.
      - Preserve historical /root/... references only in immutable 
      audit/provenance docs; active scripts, runbooks, tests, and 
      templates must not depend on
        home-directory repos.
      - Use worktrees under /opt/repos/worktrees/<repo>/<branch>.
  - NeoCortex monorepo completion
      - Finish all migration blockers already tracked in 
      NeoCortex: M70 stage4/QCOW validation, Kata/container 
      runtime, FMT2 ephemeral VM lane, Gentoo/OpenRC
        replacement for temporary Rocky bootstrap, Thor replacement 
        path, and optional history graft/freeze decision.
      - Treat NeoCortex as authoritative after completion: source 
      repos become frozen compatibility references or are 
      history-grafted into NeoCortex with
        documented provenance.
      - Add/repair migration validation so PR completion proves 
      standards baseline, Git LFS, repo location policy, shell 
      tests, YAML parsing, and migration
        manifests.
  - Control-plane and multi-worker comms
      - Bring all control-plane fixes into NeoCortex and validate 
      Forge/FCP worker communication end to end.
      - Add tests for FCP envelope validation, replay/nonce handling, 
      approval ceilings, routing scope, transport policy, ntfy bridge 
      behavior, and multi-worker request/response flows.
      - Replace temporary Rocky bootstrap control-plane host with the 
      compliant Gentoo/OpenRC image once that image passes validation.
  - ZFS, NFS, and storage policy
      - Use the provided /opt/repos ZFS dataset options as the canonical 
      dataset profile for repo storage.
      - Add reusable dataset/playbook logic for repo datasets and 
      per-user home datasets.
      - Add NFS/floating-home validation that detects the NAS NFSv4.2 
      0777 directory-mode bug under expected 0027 umask and blocks 
      insecure rollout.
  - Build acceleration and hardware rollout
      - Use SLURM plus distcc for M70 stage4/QAT/OpenZFS builds.
      - Treat X12again and FMT2 as available fast-track build/validation 
      capacity.
      - Build binpkg path for OpenSSL + QAT + OpenZFS on M70 canary first; 
      non-QAT nodes may consume the same binpkgs with QAT unused.
      - After M70 canary passes, roll the validated path to K10 and 
      run destructive AAA/netboot validation with rollback evidence 
      captured.

  ## Test Plan

  - Run NeoCortex shell/YAML validation, including migration, 
  standards-baseline, Git LFS, .gitattributes, and no-active-home-repo-path 
  checks.
  - Run control-plane/FCP tests for envelope schema, replay rejection, 
  worker routing, ntfy bridge, and multi-worker comms.
  - Run Ansible/playbook syntax checks for ZFS datasets, NFS home policy, 
  SLURM/distcc build orchestration, M70 canary, FMT2/X12again lanes, and 
  K10 netboot
    AAA.
  - Execute staged hardware validation:
      1. SLURM/distcc dry-run.
      2. M70 canary build and QCOW/stage validation.
      3. FMT2/X12again accelerated build/ephemeral VM smoke.
      4. K10 netboot/AAA destructive validation after M70 success.

  ## Assumptions

  - Retrofit NeoCortex rather than restart the repo unless standards 
  validation exposes unrecoverable divergence.
  - Historical audit files may retain old /root/... paths as provenance; 
  active operational content may not.
  - M70 canary is the release gate for QAT/OpenZFS/binpkg rollout.
  - K10 destructive AAA/netboot validation is in scope only after M70 
  canary passes and rollback evidence is captured.

─ Worked for 2m 18s ───────────────────────────────────────────────────────────

Choose your failure mode and you will find it quickly.

Please Skip the Philosophy

A valid request. Perhaps you don’t care about this verbosity of thought, or considerations therein, or maybe you’re aware of the reprisals and follies and want to move on to the engineering discussion? If so, simply scroll down to the section titled, “How Does This Relate to Memory”. Note that in doing so you will miss a blurb from Office Space.

Endeavor to Endure the Philosophy

A fair degree of the human experience involves a process by which those of “early-phase cognition” attempt to self-obfuscate their “desire for desire”, attempting to live a life of nebulous and fleeting happiness, non-blissfully and unaware that it is this desire to feel and experience desire itself which has cornered their experience into a lifetime of endless searching, yearning, wanting, never satiated, eventually growing old and stale and losing their imaginative or creative or physical edge.

This cycle of The Great Chase is one basis of Sisiphean frustration; one which - until it is surpassed - which requires being acknowledged, accepted, and integrated into their forward movement

Some understand that to exist with desire, that all-controlling force, seemingly inescapable force which presses down upon the rock, forever trapping your perspective into the wheel of the unknown yet forever traveled, repeating mistakes which others warned against, refusing awareness of one’s own misfortunate mentality and guiding-line towards an inevitable failure; experiences from which all lessons are never to be learned.

Go ahead. Do it. Don’t learn from others. Do everything yourself, do it with the veracity by which you have become convinced is correct - do everything as though this is the first time anyone has ever tried to achieve the new shiny thing. Surely, this must be the path towards truth, self-realization, self-actualization, self-assurance, and that desperation of desire for freedom from desire to desire everything… and then… surely success is ensured?

No, that modality of mentality would achieve its opposite effect, and it would be a waste of everyone’s time; because despite the ego-driven main-character syndrome which has become so performatively boring in the past two decades of social media, it’s not all about You, or Me, it’s about Everyone alive and even those who we have loved who died.

Mr. Gibbons

Peter Gibbons:
It’s not just about me and my dream of doing nothing. It’s about all of us… Michael, we don’t have a lot of time on this earth!

We weren’t meant to spend it this way. Human beings were not meant to sit in little cubicles staring at computer screens all day, filling out useless forms and listening to eight different bosses drone on about about mission statements.

Michael Bolton:
I told those fudge-packers I liked Michael Bolton’s music.

Peter Gibbons:
Oh, That is not right, Michael.

Purposeless self-serving inefficiency which is predicated on assured failure wastes time that humanity does not have - we do not have time for oppositional defiance induced mediocrity. Stop the gamification.

The Mindset of Being Simply Wrong

Instead of repeating the pattern described above, grow the fuck up and learn how to effectively learn from others, to learn from the past, to learn using a reinforcement framework of concept-action-processing. “Thems may be big words”, but this is not a complicated concept. One does not have to choose the path of stupidity; ignorance - we’re all ignorant in all manner of ways and that’s inescapable; however when one ignores the data, ignores the lesson, and chooses the path before them via oppositional defiance, that’s called “being stupid”. It’s stupid not to learn from others.

How Does This Relate to Memory

Memory is everything. The conscious experience, the neural network running inside our brains, the execution of “The Default Mode Network”, requires effective memory - short term a well as consolidation to long-term memory.

So, when the industry was informed of the untimely discontinuation of Intel’s Optane NVDIMMs… we were all screwed. Until Q2 2026. Now it’s Q2 and finally CXL non-volatile memory is available in sufficiently equivalent density, performance, and lifespan to exceed Optane’s role, and it is available in PCIe Gen-4 and Gen-5 and soon… Gen-6.

An Introduction to CXL

• It’s not HBM, it’s not 3D-Xpoint, it’s not SRAM, nor vRAM
• CXL is packaged in PCIe, U.3, and EDSFF formats
• CXL offers dense capacity, low-latency, and exceptional I/O
• CXL is made for Ai/HPC/OLTP memory acceleration
• CXL is the new hotness, it is the new standard, it is performance

HPC Cluster Storage Acceleration

CXL is being built into (not next-gen, now-gen) storage appliances and network protocol accelerator systems used in High Performance Computing, Ai/ML which leverages HPC tech, and Supercomputers.

CXL in Telco DSP as Protocol Caches

CXL is perfectly suited for digital signals processing on a range of network protocols and their intra-chassis and inter-chassis RDMA driven networks. There’s one person who I know that hates it when I bring up RDMA - truly an example of being willfully ignorant. Moving on…

CXL Memory in Telco ATAC Chassis

• 14 slots in 19-inch rack ≈→ ∆ {$N total trays per 42U ≈ μ19}
• 16 slots in 23-inch ETSI rack ≈→ ∆ {$N total trays per 42U ≈ μ23}
• Chassis heights from 4U to 15U ≈↑ ∆ [[ μ4, μ19, μ23 ]]
• Telco grade Power & Cooling Efficiency & Redundancy ++
• Mil-DoD Compliant Physical & Operational Security ++
• Industrial NEBS Environmental Operating Ranges ++
• Zero-Downtime Serviceability & Upgrade/Maint ++
• Intra+Inter-Chassis Redundancy (Modular Scale) ++
• Geo-Deploy Designed for regional XOR continental mesh ++

So, Why Should We Listen to Her?

Becausee I’ve been building and iterating these types of storage architectures for nearly twenty years of my career. If you want to do it all from scratch or buy this type of storage performance for 10x the cost, go ahead; the market exists and you can be the big winner.

However, if you don’t want to pay a premium for non-OpenSource storage clusters which are capable of serving and scaling from 10s to 100s of Petabytes, then this is the direction by which engineering solutions arrive at the datacenter. My storage designs range from Oracle’s Exadata & Exascale clusters which have been adapted to run non-standard protocols, to fully purpose-built data-acceleration clusters which use COTS building blocks to provide similar performance and scale at a fraction of the cost (not including consulting, engineering scope, and top-to-bottom fully-automated storage architecture deployments).

As networking has improved, NAND designs have advanced, and costs have reduced on $/TB and $/Watt… yet always… always the storage sizes grow in density and capabilities. This is natural and expected.

Project Coherent Storage, an Evolution

Six primary iterations, tracking the advancement of NAND acceleration, Network Acceleration, Protocol Offloading, Non-Volarile RAM, and now we see CXL capable of being deploying into reference architectures by which all others may be measured.

• Project Coherent Storage v1, (2010-2012)
• Project Coherent Storage v2, (2011-2015)
• Project Coherent NAND, (CMPLX.io) (2015-2017)
• Project Coherent IO-Max v1, w/ Optane PMem (2020-2024)
• Project Coherent IO-Max v2, w/ Optane PMem (2024-2025)
• Project Coherent Storage v3, w/ CXL AIC/U3 (2026)

Development Details - Concise Timeline

A little bit of engineering nerdery for those who care… and btw, the community myth that ZFS cannot fulfill a role as a multi-node clustered filesystem are wrong.

What About the “ZFS Can’t Cluster” Myth?

The “Clustered ZFS” operating modality for multi-controller interactions (not limited to A/B active/passive or even active/active with modified pool-async hooks) requires advanced network hardware and engineering experience which most are not aware of, and so the myth perpetuates; but it is wrong and it has been wrong for as long as SAS drives have had dual-heads and shelves have had 4+ SFF connectors ready for multi-multipath-chaining - and enterprise NVMe drives are also available in this arrangement for the same purpose.

Project Coherent Storage v1 (2010-2012)

1. Coherence Grid (DDR3 (Tier1)) + Custom Relational-Vector DBMS
2. Sunfire ZFS controllers w. Multi-Protocol-Net Interlinks (IB, IPoIB, mNFS+, FC, FCoE)
3. Geo-Dist-Nodes w. DDR3-PCIe SuperCap “Hot Cache Layer” (Tier2), w. Sunfire X4600/M2 @ 64 DIMMs Hotswap
4. Geo-Dist-Nodes w. SAS SSDs (Tier3) + SAS Spinner (Tier4)
5. Controller A/B+ nodes w. SAS-HBA to LTO-N (Tier5)

Project Coherent Storage v2 (2011-2015)

1. Coherence Grid (DDR3 (Tier1)) + Relational-Vector DBMS
2. Sunfire ZFS controllers w. Multi-Protocol-Net Interlinks (IB, IP Native L2/L3, mNFS+, FCoE)
3. Geo-Dist-Nodes w. FusionIO Stacked “Hot Cache Layer” (Tier2)
4. Geo-Dist-Nodes w. SAS SSDs (Tier3) + SAS Spinner (Tier4)
5. Optional A/B nodes w. SAS-HBA to LTO-N (Tier5)

Project Coherent NAND - CMPLX.io (2015-2017)

1. Coherence Grid DDR4 (Tier1) + Relational-Vector DBMS
2. Native ZFS controllers w. Multi-Protocol-Net Interlinks (IB, IP Native L2/L3, mNFS+, iSER)
3. Geo-Dist-Nodes w. NAND Stacked “Hot Cache Layer” (Tier2)
4. Geo-Dist-Nodes w. SAS3 SSDs (Tier3) + SAS3 Spinner (Tier4) [+ optional SAS3 HBA to LTO for Iron Mountain (Tier5)]
5. This architecture was a basis for one of my former startups, which was sold to private equity and built-upon for their HFT clusters under very tight NDA + Confidentiality Contract

Project Coherent IO-Max Storage Architectures v1, 2, 3

These storage architectures share core foundations, with iterations upward as Non-Volatile Memory was introduced on a wider scale into my designs. The initial design for v1 featured the first generation of Intel Optane 3D-XPoint (Series 100), v2 with Series-200, and now CXL taking the place of Optane PMem SKUs.

The details of each design from v1 and v2 may be expanded upon later, simply for historical purposes, and I have other things to do right now… but if you’ve made it this far into the post and want to see the specification ADRs and related research, which is RAG-pipelined and iteratively deployed on real hardware for revisions and regression testing to validate LLM Inference usage & advancements with the now-opensource Coherence-CE version… scroll down for a link to v3’s repo.

• Project Coherent IO-Max v1, (2020-2023)
• Project Coherent IO-Max v2, (2023-2025)
• Project Coherent IO-Max v3,CXL (2026)

Adjusted Query: how much storage space would the user deploy to an existing multi-Petabyte array, with consideration to physical limitations in systems design which prioritize two common architectural decision targets: More or Faster?

(1) maximizing raw-block capacity
(2) maximizing I/O performance

TL;DR

Between 4-5PB is the simplest answer, but those are just numbers. Could be 10x either, but that’s not realistic - it’s conjecture. The full picture requires insight into physical limits of the transfer time, storage space, rack space, and rotational speeds involved; spacetime could be analyzed by the right minds if this were JPL ^_^ .. instead of limited to hardware I’ve acquired over a small handful of years.

Hypothetical or Reality

Let’s say that I contacted the datacenter and ordered additional power circuits to light-up additional drive shelves, which are already installed (minus the box of drive trays which are in a massive box on a parts shelf in my home office). Ok, reality checks out there.

The circuit drops would go to the central storage rack which is currently at 84% power limit. I know this because power is monitored and validated via a combination of Check_MK + Prometheus + SNMP exporters, which poll power stats remotely from the cabinet’s APC ATS, the rack’s per-port metered and switched outlet PDUs, and then if I really want to I can look at the DC’s circuit stats (via MRTG) which are tracked for the cabinet. Reality checks out there decently enough. Moving on…

The Hypothetical

First we’ll cover some raw numbers and theoretical maximums by looking at two high-level approaches required for “More or Faster”.

The astute reader may wonder why we’re not discussing >100TB QLC NVMe drives or ultra-dense E2 ruler-style 2U chassis, and the answer is that those calculations are generally similar in equation workflow (eg: how many things of X can we put into Y at A,B,C performance/size/latency), however those hardware types are not currently racked in my real-world systems at an interesting scale. However, migrating to denser NVMe arrangements will likely be part of an eventual down-scaling forklift operation whenever the SAS3 arrays are no longer useful for my personal labs or colo bills.

Production is another story, where large format SAS3 is still relevant and being deployed and improved over time; in the last several years we’re seen drive capacity double, along with some innovative designs. It’s not often considered exciting, but it lasts A LONG TIME, and the NSA loves archiving the internet on this medium over in their Utah storage complex (overheard, surely).

So, back to the math…

Decision: Optimize for Faster (prioritize IOPs + bw)

Aggregate per-rack w/ 10x shelves of raw 4K block = 3,824 TB == 3.72 PB

Decision: Optimize for RAW Storage Capacity

How about the same hardware focused on max-total raw space with less regard to per-drive IOPs (perhaps we offset the latency reduction per-drive via larger L1ARC and L2ARC buffers on the head-nodes, which becomes a loaded tangent)… then the rack could be upgraded to host 4.85PB @ $800/m MRC ($400/m cabinet + $400/m amperage)**.

We’ll also save the conversation about concessions made at time of contract signing where I might / maybe / surely have saved on recurring costs by locking-in a multi-year multi-rack contract. Sales team, thanks!

What is this MRC, MRR?

MRC in this situation is the ‘Monthly Recurring Cost’ for power + rack. This value does not include the cost of acquired hardware, which is variable on supply-chain sourcing and other factors. I’m also not going to describe potential MRC or MRR offsets which factor into profit margin potential and/or influence EBITDA.

Ok, what is MRR and EBID..? Fine, at least with MRR it’s ‘Monthly Recurring Revenue’, and one can read all about EBITDA elsewhere.

What about Flash-Cache or Optane or Non-Volatile blah blah blah

Sure, those are considerations, where applicable. Some footnote notes…

• It’s prudent to deploy storage controllers with SLC NVMe or PMem/NVDIMM persistent cache (which are often a baseline performance requirement). These are sized anywhere from cumulative 1-12TB per-controller, depending on available PCIe lanes and/or NV-DIMM slots.
• It’s a good idea to make scoping decisions like that ahead of time, instead of later realizing that the controllers’ lanes are maxed out or all DIMM slots are used (or you bought 128GB NVDIMMs instead of 512GB, thinking “8x 128GB should be fine for PMem” or the other fun one about ignoring (or being unaware) about GHz speed reduction by running 4-DPC in a 2-DPC max-perf system.
• Those determinations may be either workload dependent, or in my personal life sometimes it qualifies as the “because I wanted to” type of maximalist methodology, or the “this is relevant for future proofing my own rationalization on fun hardware”, or could be considered a baseline in pre-sales because “they always want more performance eventually; just bake it in from the start”. It varies, but remember where you started and where any future requirements for performance headroom exists.

Where’s the Math?

Validation on spec baselines can be tedious, but anyway… the DE3-24C and DE3-24P have very shiny and well made Dual-Redundant IOMs with 4x SAS3 SFF-8643 connectors.

Specs for our SAS3 Fabric with DE3-24 Shelves

• One SAS-3 lane (PHY): 12 Gbit/s ⇒ 1.5 GB/s (raw line-rate, decimal GB)
• External SAS x4 port connector: 4 PHY × 12 Gbit/s ⇒ 48 Gbit/s ⇒ 6 GB/s
• Each DE3-24{C/P} has 2× IOM, with each IOM having 4× SFF-8643 ports

Drive Specs - Seagate EXOS & Oracle SSD

Scenario A – Maximum RAW Block Space

Before going into the theoretical weeds, let’s look at some per-shelf media limits.

1) Oracle DE3-24P shelf w/ 24TB Spinners**:** 24 × 285 MB/s = 6.84 GB/s (media-limited)

2) Oracle DE3-24C shelf w/ SSDs:

• Read: 24 × 2150 MB/s = 51.60 GB/s → capped by fabric at 36 GB/s (N=3)
• Write: 24 × 1980 MB/s = 47.52 GB/s → capped by fabric at 36 GB/s (N=3)

Scenario B: Maximum Performance

Hardware has limits, and we’re looking at 8× DE3-24P w/ 18TB Mach.2 HDD + 2× DE3-24C w/ 7.68TB SSD (N=3)

Aggregate sequential bandwidth (N=3)

• Mach.2 18TB HDD shelf: 24 × 554 MB/s = 13.296 GB/s (media-limited; below 36 GB/s fabric cap)
• SSD shelf: remains fabric-capped at 36 GB/s read and write (N=3)

Aggregate Random R/W 4K IOPS (N=3)

Fabric-derived 4K IOPS cap per shelf (upper bound):
36 GB/s @ 4KB = approx 8.79M IOPS

• SSD shelf
  • Read: 24 × 450k = 10.8M IOPs → capped to 8.79M IOPs
  • Write: 24 × 95k = 2.28M IOPs → media-limited (below fabric cap)
• Mach.2 HDD shelf (4K @ QD16)
  • Read: 24 × 304 = 7,296 IOPs
  • Write: 24 × 560 = 13,440 IOPs

Let’s See the Deltas!

What’s “a delta”? Sometimes it’s the headwaters of a bay, where the river distributes across a flood-plane into many separately-unified channels, eventually giving way to deeper waters and…. no no not that kind.

Delta as a value in storage is often used as a marker to show the differential of two functions which indicates directional change. At least, that’s what comes from my brain while I sit here having an ice cream sandwich, but it has some alternate terminology which applies irrespective of ice cream hour:

Delta uppercase Δ, lowercase δ
Uppercase: a change of any changeable quantity
Lowercase: the central difference for a function

~ The discriminant of a polynomial equation or symmetric difference of two sets
~ The determinant of the matrix of coefficients of a set of linear equations

Scenario B vs Scenario A (with N=3)

• Total sequential BW: 126.72 → 178.368 GB/s (increase +51.648 GB/s)
  • Primary determinant: per-shelf BW 6.84 → 13.296 GB/s (Mach.2 dual-actuator).
• SSD tier remains fabric-capped at 36 GB/s per SSD shelf under N=3; increasing N (or adding more independent uplinks with additional HBAs) is what changes that cap.

1. Faster array = less space @ higher IOPs and BW == 178 GB/s @ 3.8PB
2. Larger array = more space @ lower IOPs and BW == 126 GB/s @ 4.9PB

Equations w/ decimal units: 1 TB = 1000 GB

• jaja usually we work in 1024 (but I’m tired today, so not right now)

t_1TB(s)      = 1000 / BW_GBps
t_full(s)     = (Capacity_TB * 1000) / BW_GBps
t_full(hours) = t_full(s) / 3600
BW_density    = BW_GBps / Capacity_TB   (GB/s per TB)
              = 1000 * BW_density       (MB/s per TB)

So… what’s the answer? More, Faster?

Is bigger is better? Sometimes. Sometimes you need it faster. Sometimes you want both.

Oracle bought Sun to get the ZFS file system. They have been porting ZFS features into Btrfs. The head of the Ext4 file system project, an Oracle employee, said that Ext4 was just a temporary stepping stone on the way to Btrfs.

A primary motivation for Oracle’s acquisition of Sun was to obtain and control the IP for the MySQL database project, which had been obliterating a large amount of Oracle DB customer licensing and dominating the OSS database landscape.

A primary motivation for Oracle’s acquisition of Sun was to obtain their hardware platform for running Oracle DB, Oracle J2EE applications, and their vast middleware stacks. This was everything related to the UltraSparc architecture, Sun Fire line of Opteron based servers, all of the high performance storage technology, and all of Sun’s enterprise contracts with the US military, US gov, various nation states, and all manner of other high-end “serious business” service compute providers.

Sun created Java, nearly all of Oracle’s entire stack requires Java, and Larry wanted that as a foundational piece of IP. At the time of the acquisition, the world ran on Java and it was the most profitable and most widely deployed programming language as well as server runtime environment (JDK, JRE). This aspect is also related to various IP lawsuits around Java, see Oracle vs Google later on.

ReiserFS features were ported into Btrfs, not ZFS features. ZFS has always dominated enterprise storage, never btrfs, never ReiserFS. ZFS in an entirely different class of quality, performance, scalability, reliability, and global market penetration.

RedHat was tired of supporting Btrfs after too many customers reported critical data-loss incidents, which could only serve to harm the company’s reputation for stability and technical excellence.

ZFS continues to be amazing, Btrfs continues to suck, Oracle still has their own original ZFS source tree separate from the OpenZFS project, Oracle still sells UltraSparc architecture, Oracle still owns Java IP, Oracle still doesn’t prioritize Btrfs, nothing important runs on Btrfs.

The only recent news about btrfs and Redhat, which invalidates the original claims about Redhat vs Oracle conspiracy, is that Fedora Linux 33 has Btrfs as a default option along with XFS and Ext4, and we all know that Fedora is upstream development for RedHat Enterprise Linux.

Some timeline details:

• 2007, Btrfs concepts and originated at IBM
• 2008, rip-off of ReiserFS features started by developer at Oracle, who was a former dev of ReiserFS (before Hans Reiser murdered wife, tried to cover it up, landed in prison, and ReiserFS project basically died too)
• Btrfs continues to be experimental for years, cannot complete with ZFS on any important metrics
• 2008, Oracle Exadata v1 - a specialized hardware platform for running Oracle databases on HP-UX with Itanium procs
• 2009, Oracle Exadata v2 - now running on Sun hardware, using modified Sun Fire systems with AMD Opteron procs
• 2010, Oracle acquired Sun which had recently acquired MySQL
• 2011, Redhat offers Btrfs as experimental in RHEL 6 beta
• 2012, Btrfs main developer leaves Oracle for Fusion-IO
• 2012, Oracle and SUSE offer Btrfs in their distro for prod use
• 2013, Btrfs main developer leaves Fusion-IO for Facebook
• 2017, Redhat removes Btrfs from inclusion to RHEL 8, switching to XFS (originally from SGI)
• 2018, Redhat acquisition by IBM

Some enjoyable references:

• en.wikipedia.org/wiki/Btrf…
• en.wikipedia.org/wiki/Acqu…
• en.wikipedia.org/wiki/Hans…
• en.wikipedia.org/wiki/ZFS
• en.wikipedia.org/wiki/XFS
• en.wikipedia.org/wiki/Orac…
• www.oracle.com/webfolder…
• flashdba.com/history-o…