Data caching at The Edge isn’t always from Fastter or Cuntter or Cloudstter or whichever CDN happens to be serving your IG, Safeway, NFL, Reddit, Anthropology, or 338% of the internet’s concurrent traffic being pointless traversals like jQuery minified & brotli-fied and sent back and forth endlessly through Internet Exchanges.

Edge caching is not limited to Anthropic or OAI or whoever else’s Load-Balancer API-side “prompt cache (not yet! after LB!)” packets which “those people” are constantly complaining about, nor the hyperscalers who whitebox big CDN networks under their own headline IP.. those are not what I’m talking about when I talk about Edge Caches.

These little embedded devices which collect climate metrics, air quality data, assembly robotics, municipal awareness and telemetry data, and everything else from a very long list… they need read-heavy churn-heavy persistent storage on a filesystem optimized for tiny sized industrial NAND.

More examples for the ones of all sizes making modern life possible: that ATM you went to, the parking lot video surveillance systems, traffic light coordinator nodes, telco edge ONT routers and OOBs with SDNs, and a thousand other examples abound that are functionally ignored.

Also, robots. And factory hardware. And factories making robots that make tiny factories? Hopefully so.

Embeds Reduce CAPEX & OPEX Efficiently

So, those embedded devices usually are designed for efficiency, while not too often designed to be hammering on NAND for write-heavy storage workloads. This has caveats: devices explicitly designed for write heavy I/O on purpose, and therefore not what I’m referring to, and cases where the provisioning and/or configuration stages of fleet deployment are improperly aligned to hardware realities.

Why? Low wattage, moderate caches, and a desire to not burn out NAND when the device is located in a difficult-to-get-to location for service techs working on hardware replacement.

Most of all - we want efficient storage for embedded devices so that their operationally collected data can be stored in memory with a ECC write-back cache with “operationally consistent” garbage collection and non-invasive wear-leveling, all while the transactions are paged/queued to F2FS before async’ing upstream to a centralized data collector for aggregation.

Predator Drones Too?

Yes and no, they send their comms to LEOSAT before humanity’s “furthest from the menial and meaningless non-Elysium people” edge devices cache, aggregate, and compute roll-up analytics with inline encryption and compression via DPU offload streaming.. all the way down to LANDSAT.

Edge devices have their own little ecosystem separate from the manner in which datacenter storage network-enabled hardware operates. All by design, and always room for improvement.

Many people consider this boring tech, and would rather move fast and break things — your time is limited, people are sick of perfectly workable systems breaking all the fuckin time and they’re tired of being A/B tested on in production — doing those things makes your company’s testing policy look like an asshole. Everyone, stop being assholes. Thanks!

These Things - Why Should We Care?

You like your cell phone? How about GPS? SAT-NAV? Skylink? Drones operating up above withour your awareness? How about sector antennas for the police and emergency services? Embedded systems are streaming, caching, processing, and relaying data all over the globe and up into LEO constellations.

You don’t have to care, and you shouldn’t have to care about FSFS or drones unless you work with that side of tech, or its adjacent use cases - like living in the modern world. Everyone should care because knowledge is fundamentally important, if for no other reason at all. Do you know how the light-switch on the wall works? Same deal. Stop making excuses for saying you don’t know or don’t care or you will be left behind. No one is saving us but ourselves, and we must have knowledge to do so.

Recent Adventures with Standards-Definitions

I love automation. In part this is because I prefer efficiency over repetitive manual bullshit. Efficiently handling requirements means more time to do what I love:

• Hanging out by the pool with a tablet or waterproof e-ink reader.
  Way back in 2018 my e-book tally was 62 novels over the calendar year. That number has dropped off recently for trivial reasons, but the point remains that I love to read, and I love to lay out by the pool.
• What I do not love:
  Wasting time when I could be at the pool. ♥‿♥

there’s only room for two hats in my life - and this is the one for sunning

• What I’ll never not love:
  The poem on my torso, dedicated to the manner in which my departed twin lived & loved.

Standards for NAND? Standards for SD-NAND!

To the point, here’s a script that had to be blasted out a while back, while imaging so many too many industrial-spec SD/TF-cards for a fleet of distibuted data collectors. These fun little RK3588 based systems have a NPU and only an eMMC drive for the OS.

While the eMMC in question has decent specs, if it fails then it’s sadly more expensive to de-solder and re-solder and re-image the OS on eMMC, and so on these systems we use the SD NAND for the following data:

• Ephemeral scratch space and logs (eg: /var/log, /var/tmp, /tmp - nope, these go to tmpfs)
• Cache directories which are aptly named .cache or .Cache סּ_סּ
• Files which must be staged locally but have no business being on the OS mount points.
• Several other use cases that I don’t feel like describing right now; so here’s the discussion for those who love to know more more more.

### F2FS - Embedded Filesystem Performance Options

- Ref: https://docs.kernel.org/filesystems/f2fs.html

#### Option Flags and Reasoning

##### For High-Churn I/O
This is less ideal, but sometimes a `tmpfs` overlay isn't an option, so it's better than burning out the eMMC. A good industrial SD card has ECC, wear-leveling, checksum validations, write-amplification controls, etc. Don't care for those features? You may one day...

Example Writes: data persistent log-sync, package builds on-device, container writable layers, time-series compressible metrics, telemetry buffers, or small-file caches, opt for the non-default `-o 10` flag. 

| Option                      | Rationale              |
| --------------------------- | -----------------------|
| `-l DATA`                   | Stable label for `/etc/fstab`; avoids relying on device enumeration order. UUID is an alternative, but requires programmatic parsing of `blkid` output, or manually specifying a UUID to pass to the FS creation process.. or you can simpy use a GPT label. |
| `-o 5`                      | Reserves 5% of the F2FS volume for the F2FS cleaner. `mkfs.f2fs` documents this as hidden overprovision space used by the cleaner. |
| `-t 1`                      | Performs discard/TRIM at format time; defaults discard to enabled, but some card readers/cards mishandle it; use `-t 0` if format stalls or logs errors |
| `extra_attr`                | Required by several modern F2FS features. |
| `inode_checksum`            | Adds inode metadata checksums; useful for removable flash and unattended systems. |
| `sb_checksum`               | Adds superblock checksum support. |
| `flexible_inline_xattr`     | Modern xattr layout; useful with `extra_attr` and future feature expansion. |
| `inode_crtime`              | Adds inode creation-time support; low cost, useful for forensic/debug visibility. |
| `lost_found`                | Creates lost+found support for recovery workflows. |
| `compression`               | Good for compressible write-heavy data; poor fit for already-compressed media, model files, archives, database pages, and random-write workloads. |
| `compress_algorithm=zstd:3` | Validate performance of zstd compression options before using for production purposes. |
| `compress_log_size=%u`      | Support configuring compress cluster size. The size will be `4KB * (1 << %u)`. The default and minimum sizes are 16KB. |
| `checkpoint_merge`          | Self-describing (or check the refs)  |
| `nocheckpoint_merge`        | Disable checkpoint merge feature. |

- `checkpoint=%s[:%u[%]]`: Set to "disable" to turn off checkpointing. Set to "enable" to re-enable checkpointing. Is enabled by default. 

- `checkpoint_merge`
When checkpoint is enabled, this can be used to create a kernel daemon and make it to merge concurrent checkpoint requests as much as possible to eliminate redundant checkpoint issues. Plus, we can eliminate the sluggish issue caused by slow checkpoint operation when the checkpoint is done in a process context in a cgroup having low i/o budget and cpu shares.

- `compress_algorithm=%s`
- `compress_algorithm=lzo-rle`

Control compress algorithm, currently f2fs supports "lzo", "lz4", "zstd" and "lzo-rle" algorithm.

- `compress_algorithm=%s:%d`

Control compress algorithm and its compress level.. only "lz4" and "zstd" support compress level config and those algos may be in your kernel config and may not.. but you'll find out quickly if you don't already know the answer.

========= =========== algorithm level range ========= =========== lz4 3 - 16 zstd 1 - 22 ========= ===========

- `compress_log_size=%u`
Support configuring compress cluster size. The size will be `4KB * (1 << %u)`. The default and minimum sizes are 16KB.

- `compress_extension=%s`

Support adding specified extension, so that f2fs can enable compression on those corresponding files.
If all files with `.ext` has high compression rate, we can set the `.ext` on compression extension list and enable compression on these file by default rather than to enable it via ioctl.

For other files, we can still enable compression via ioctl. Note that, there is one reserved special extension `*`, it can be set to enable compression for all files.

- `nocompress_extension=%s`

Support adding specified extension, so that f2fs can disable compression on those corresponding files, just contrary to compression extension.
If you know exactly which files cannot be compressed, you can use this.

> The same extension name can’t appear in both compress and nocompress extension at the same time.

If the compress extension specifies all files, the types specified by the nocompress extension will be treated as special cases and will not be compressed.
After add nocompress_extension, the priority should be: `dir_flag < comp_extention,nocompress_extension < comp_file_flag,no_comp_file_flag`.

- `compress_chksum`
Support verifying chksum of raw data in compressed cluster.

- `compress_mode=%s`
Control file compression mode: "fs" and "user". In "fs" mode (default), f2fs does automatic compression on the compression enabled files. In "user" mode, f2fs disables the automaic compression and gives the user discretion of choosing the target file and the timing.

The user can do manual compression/decompression on the compression enabled files using ioctls.

- `compress_cache`
Support to use address space of a filesystem managed inode to cache compressed block, in order to improve cache hit ratio of random read.

- `inlinecrypt`
When possible, encrypt/decrypt the contents of encrypted files using the blk-crypto framework rather than filesystem-layer encryption.
This allows the use of inline encryption hardware. The on-disk format is unaffected.

- `atgc`
Enable age-threshold garbage collection, it provides high effectiveness and efficiency on background GC. No one likes garbage.

- `discard_unit=%s`
Control discard unit, the argument can be "block", "segment" and "section", issued discard command’s offset/size will be aligned to the unit, by default, "`discard_unit=block`" is set, so that small discard functionality is enabled.

For blkzoned device, "`discard_unit=section`" will be set by default, it is helpful for large sized (fucccck off with SMR, less fucks to ZNS) devices to reduce memory cost by getting rid of fs metadata supports small discard (ouch, ever get burned by SMR? yeah... just don't bother).

- `age_extent_cache`
Enable an age extent cache based on rb-tree. It records data block update frequency of the extent per inode, in order to provide better temperature hints for data block allocation.

- `errors=%s`
Specify f2fs behavior on critical errors. This supports modes: "`panic`", "`continue`" and "`remount-ro`", respectively, trigger panic immediately, continue without doing anything, and remount the partition in read-only mode.

By default it uses "`continue`" mode, not my choice.

====================== ===============
mode continue remount-ro panic ====================== ===============
access ops normal normal N/A syscall errors -EIO -EROFS N/A mount option rw ro N/A pending dir write keep keep N/A pending non-dir write drop keep N/A pending node write drop keep N/A pending meta write keep keep N/A ====================== ===============

- `nat_bits`
Enable nat_bits feature to enhance full/empty nat blocks access, by default it’s disabled.

- `lookup_mode=%s`
Control the directory lookup behavior for case-folded directories. This option has no effect on directories that do not have the casefold feature enabled.

======================================== Value Description

perf (Default) Enforces a hash-only lookup. The linear search fallback is always disabled, ignoring the on-disk flag. compat Enables the linear search fallback for compatibility with directory entries created by older kernel that used a different case-folding algorithm. This mode ignores the on-disk flag. auto F2FS determines the mode based on the on-disk SB_ENC_NO_COMPAT_FALLBACK_FL flag.

----

#### Benchmarking F2FS Compression

Always validate the compression algo used before using those options, as performance of one algo vs anther can vary widely across different CPU generation and architectures, as well as for hosts with algo/hash compression offloading engines. We can cover benchmarking these in another script.

> lzo, lz4, zstd, lzo-rle algorithms

##### Create Partitions

DEV=/dev/sdX sgdisk –zap-all “$DEV” sgdisk -a 2048 -n 1:2048:-16G -t 1:8300 -c 1:DATA “$DEV” partprobe “$DEV”

##### Create FS

mkfs.f2fs -f
-l DATA
-o 10
-t 1
-O extra_attr,inode_checksum,sb_checksum,flexible_inline_xattr,inode_crtime,lost_found,compression,compress_algorithm=zstd:3 “${DEV}1”

#### FS Tab Entry

##### Conservative defaults

LABEL=DATA /srv/data f2fs rw,noatime,nodiratime,lazytime,nofail,background_gc=on,gc_merge,checkpoint_merge,flush_merge,mode=adaptive,fsync_mode=posix,nodiscard 0 0

##### Compression-capable variant

LABEL=DATA /srv/data f2fs rw,noatime,nodiratime,lazytime,nofail,background_gc=on,gc_merge,checkpoint_merge,flush_merge,mode=adaptive,fsync_mode=posix,nodiscard,compress_algorithm=lz4,compress_chksum,compress_mode=fs 0 0

#### Selectively Mark Compressible Directories
Do not enable compression broadly for LLM model files, videos, archives, SQLite databases, or opaque binary blobs unless benchmarks show a benefit.

mkdir -p /srv/data/_logs /srv/data/_caches.compressible chattr +c /srv/data/_logs /srv/data/_cache.compressible

So, Maybe This is Scriptable?

It sure is, or by Ansible playbook, or Salt sync, perhaps even a CFengine golden image.

That’s a lot of blah blah blah. How about that script?

Efficiency, indeed. Here ya go. The benchmarking script is for another day, or you can check out this workload primer about the filesystem from USENIX (w/ link to slide deck), and a benchmarking script which has been customized for F2FS.

#!/usr/bin/env bash
# -*- mode: sh; sh-basic-offset: 4; sh-shell: bash; coding: utf-8; indent-tabs-mode: nil; -*-
# ----------------------------------------------------------------------- #
set -euo pipefail
# ----------------------------------------------------------------------- #
# Purpose: POSIX-syntax wrapper to partition and format a non-boot SD/microSD card as F2FS with optional compression.
# NAND Profile: embedded I/O, kernel 6.1+, ECC industrial spec controller.
# WARNING: this destroys data on the selected block device, so be aware.
# ----------------------------------------------------------------------- #

usage() {
    cat <<'USAGE'
Usage:
  format-f2fs-sd-data.sh -d /dev/DEVICE [-l LABEL] [-r RESERVE_GIB] [-o F2FS_OP_PERCENT] [-c]

Examples:
# ----------------------------------------------------------------------- #
  ./format-f2fs-sd-data.sh -d /dev/sdX -l DATA -r 16 -o 5
  ./format-f2fs-sd-data.sh -d /dev/mmcblk0 -l DATA -r 24 -o 8 -c

Options:
# ----------------------------------------------------------------------- #
  -d  Whole block device to erase and partition: /dev/sdX -OR- /dev/mmcblk0
  -l  F2FS label. Default: DATA
  -r  GiB left unpartitioned at end for card FTL spare area. Default: 16
  -o  F2FS hidden overprovision percentage for the F2FS cleaner. Default: 5
  -c  Enable F2FS compression feature at format time. Mount options and chattr are still required.

Dependencies:
# ----------------------------------------------------------------------- #
  sgdisk, partprobe, mkfs.f2fs, lsblk
# ----------------------------------------------------------------------- #
USAGE
}

# Basic Variables
# ----------------------------------------------------------------------- #
LABEL=DATA
RESERVE_GIB=16
OP_PERCENT=5
COMPRESS=0
DEVICE=

# Define ARGV Mappings
# ----------------------------------------------------------------------- #
while getopts 'd:l:r:o:ch' opt; do
    case "$opt" in
        d) DEVICE=$OPTARG ;;
        l) LABEL=$OPTARG ;;
        r) RESERVE_GIB=$OPTARG ;;
        o) OP_PERCENT=$OPTARG ;;
        c) COMPRESS=1 ;;
        h) usage; exit 0 ;;
        *) usage; exit 2 ;;
    esac
done

# Sanity Checks
# ----------------------------------------------------------------------- #
if [ -z "$DEVICE" ]; then
    usage >&2
    exit 2
fi

case "$DEVICE" in
    /dev/sd?|/dev/sd??|/dev/mmcblk[0-9]|/dev/nvme[0-9]n[0-9]) ;;
    *)
        echo "Refusing suspicious device path: $DEVICE" >&2
        exit 2
        ;;
esac

# Validate Deps & Check Limits
# ----------------------------------------------------------------------- #
for cmd in sgdisk partprobe mkfs.f2fs lsblk; do
    if ! command -v "$cmd" >/dev/null 2>&1; then
        echo "Missing required command: $cmd" >&2
        exit 1
    fi
done

if [ ! -b "$DEVICE" ]; then
    echo "Not a block device: $DEVICE" >&2
    exit 1
fi

case "$RESERVE_GIB" in
    ''|*[!0-9]*) echo "Reserve GiB must be an integer" >&2; exit 2 ;;
esac
case "$OP_PERCENT" in
    ''|*[!0-9]*) echo "F2FS OP percentage must be an integer" >&2; exit 2 ;;
esac

case "$DEVICE" in
    /dev/mmcblk*|/dev/nvme*) PART=${DEVICE}p1 ;;
    *) PART=${DEVICE}1 ;;
esac

# Define Default F2FS Options
# ----------------------------------------------------------------------- #
FEATURES='extra_attr,inode_checksum,sb_checksum,flexible_inline_xattr,inode_crtime,lost_found'
if [ "$COMPRESS" -eq 1 ]; then
    FEATURES=${FEATURES},compression
fi

# Make Sure...
# ----------------------------------------------------------------------- #
echo "About to DESTROY all data on: $DEVICE"
lsblk -o NAME,SIZE,MODEL,SERIAL,TRAN,RM,RO,FSTYPE,MOUNTPOINTS "$DEVICE" || true
echo
echo "New partition: $PART"
echo "Label: $LABEL"
echo "Unpartitioned reserve at end: ${RESERVE_GIB} GiB"
echo "F2FS OP: ${OP_PERCENT}%"
echo "F2FS features: $FEATURES"
echo
printf 'Type YES to continue: '
read ans
if [ "$ans" != YES ]; then
    echo "Aborted."
    exit 1
fi

# Best Effort Unmount for Existing Partitions (please don't need this)
# ----------------------------------------------------------------------- #
if command -v findmnt >/dev/null 2>&1; then
    findmnt -rn -S "$DEVICE" -o TARGET 2>/dev/null | while IFS= read -r mp; do
        [ -n "$mp" ] && umount "$mp" || true
    done
fi

# And then...
# ----------------------------------------------------------------------- #
sgdisk --zap-all "$DEVICE"
sgdisk -a 2048 -n "1:2048:-${RESERVE_GIB}G" -t 1:8300 -c "1:${LABEL}" "$DEVICE"
partprobe "$DEVICE" || true
sleep 2

if [ ! -b "$PART" ]; then
    echo "Partition node did not appear: $PART" >&2
    lsblk "$DEVICE" >&2 || true
    exit 1
fi

# OhhhShitHandlers
# ----------------------------------------------------------------------- #
# NOTE: -t 1 performs initial discard when supported; if it's hanging here, rerun manually with -t 0
mkfs.f2fs -f -l "$LABEL" -o "$OP_PERCENT" -t 1 -O "$FEATURES" "$PART" && \
sync

# Display Block Devs
# ----------------------------------------------------------------------- #
lsblk -o NAME,SIZE,MODEL,SERIAL,TRAN,RM,RO,FSTYPE,LABEL,UUID,PARTUUID,MOUNTPOINTS "$DEVICE"

echo "[COMPLETE]"
# ----------------------------------------------------------------------- #
# EOF #
# ----------------------------------------------------------------------- #

For those not familiar, the SoW is a consulting agreement between contractor and employer, which defines the scope and timeline and requirements etc to “do the needful” in a broader sense of everyone’s favorite phrase from the ‘ol “Follow the Sun” SLA support model from the early pre-Cloud era (sorry cloudbabies it’s from when you were single digits old or not yet born).

This specific SoW has been used in prior endeavors which eventually follow the typical enterprise hardware lifecycle in no specific order [1]: forklifts, retros, rebuilds, expansion, and… words.. so many words.

Labs last a long time, so it’s important to build things correctly the first time - not waste time hacking together whatever compromised decision trees were necessary due to last-minute planning (or no planning at all) and little to no experience with building labs. Remember, if you are a specialized engineer who excels in one or two areas, that does not imply domain specific knowledge from adjacent or entirely different sides of engineering worlds. I’m a systems architect .. I don’t build jet engines (though it’s probably fun), so I do what I’m good at. Otherwise to say, “stay in your lane or learn to love losing - but protect that ego at all costs and never admit fault ever!” (yes, I’m bitter and for exceptionally valid reasons).

So, until the pre-IPO runway runs out, or the post-IPO environment bores people to tears and they jump ship with their ISOs and RSUs and START ALL OVER AGAIN! (I have definitely never not done this exact thing several times never for sure for sure? sure.)

Successful Labs Require People and Hardware

Vaporware labs don’t have hardware. Real labs do not revolve around PPTX slide decks, or Slack Canvas, or Sharepoint whatever, or ephemeral Confluence pages that change ownership by a click of a button. Real labs have real hardware and real engineers and architects involved at every critical stage to ensure that standards are being used, implemented, tested, and validated.

If you think I’m wrong, go ahead, do it all by slide deck, by half-baked simulators that don’t even use SST and parallelization correctly, with lots of Big Discussions for the Big People you want to impress to get ahead instead of making quality hardware.

See how far the Peter Principle gets you - chances are, if you’re surrounded by “Yes Men” and “C-Suite but never Technical” boardrooms, you will get far enough to be proven the fraud that you are.

The Peter principle is a concept in management .. which observes that people in a hierarchy tend to rise to “a level of respective incompetence”: employees are promoted based on their success in previous jobs until they reach a level at which they are no longer competent, as skills in one job do not necessarily translate to another.

Having risen rapidly, since you’re up high now - you can destroy entire companies by making extremely stupid choices “to save money” by not spec’ing hot-swap trays for NVMe drives on 1U systems which were never designed for the wattage you’re running - and were not spec’d for the BTU and LFM required to keep the systems from burning out 100G optics..

Just when you think it’s all good, um why are there only seven drives in this chassis that can run with eight?

• Oh it’s because the 8th drive would draw too much power and overload the non-redundant PSUs. What happens when a system which should have N+1 has only N?
• The PSUs could be in redundant mode but Billy didn’t want to authorize the enablement of active/active mode, and instead has demanded that his lacky make the systems run the PSUs in “combined mode” to try to get more power to that 8th drive… but no… not possible.
• So all of the fancy new systems can’t have eight drives like the original spec REQUIRED in order to fulfill global fleet machine generation scaling models. As in, you lose 1/8th of your drives on every new system for so many thousands of systems. 1/8th, for those not inclined to fractional conversions, is
• So now all of those systems will be missing ~2-16-64TB of NVMe drives (multiply by number of nodes per rack, per pod, per cage, per site, per region, per global fleet). That’s a lot of expected high-speed storage that is now not being deployed.
• Hopefully no contracts were made to customers based on “Total NVMe Raw Storage per <unit/region/etc>” variables, which now renders the deliverable impossible.
• What can be done? Those contracts have to either be revised and compensated (yeah good f’in luck), or a massive workaround must be implemented to compensate - at additional cost to the company.

Why does that matter to building labs?

Because any lab worth their salt would have raised immediate red flags and halted the entire ill-devised scope. Then they would use standardized and periodically revised for betterment:

• provide technical backing evidence
• scaling costs, IOP/s cost, power assessments, baseline to STDDEV ratio, then appropriation wear-cycle equations
• All that industry jargon that managerializes into the ether for all to repeat.
• “keep me honest” you may say, so then.. are you often lying? Ill-informed, unprepared, completely clueless?
• Maybe not, hopefully not, but analytical people will see you that way and eventually the Peter Principle catches up and BAM! You’re hit by a and your entire team is fired, and so you move on with your RSUs, talk yourself up, and try to ruin those who smote you from afar.

Congrats, Billy, you’ll never change - and that is why we test in labs - because things change.

The Contract Statement of Work

Always get the scope in writing with sign-off. Scope changes mid-stream, not awesome.

Statement of Work

Consultant Services Engagement

This Statement of Work ("SOW") is entered into between [Client Legal
Name] ("Client") and [Consultant Legal Name] ("Consultant"), effective
as of [Effective Date], and is governed by the [Master Consulting
Agreement / Professional Services Agreement] dated [Agreement Date] (the
"Agreement"). In the event of any conflict between this SOW and the
Agreement, the Agreement will control.

  -----------------------------------------------------------------------
  Client                              [Client Legal Name]
  ----------------------------------- -----------------------------------
  Consultant                          [Consultant Legal Name]

  SOW Effective Date                  [Effective Date]

  Initial Term                        Six (6) months from the SOW
                                      Effective Date

  Target Level of Effort              Forty (40) hours per week,
                                      approximately 1,040 hours over the
                                      initial term

  Primary Stakeholders                VP, OCTO, Eng-Org, App-Team, and
                                      Sales-Org

  Commercial Terms                    Billing rate, invoicing, payment
                                      terms, and approved expenses as set
                                      forth in the Agreement and/or
                                      applicable order form
  -----------------------------------------------------------------------

1. Purpose and Objectives

The Consultant will provide technical leadership, architecture,
implementation, documentation, mentoring, and subject matter expertise
to establish standardized HPC benchmarking, workload automation, and
telemetry/analytics capabilities, while also improving cross-functional
execution and the overall developer experience for CORP products and
associated collateral.

- Define, document, and deploy a modular "Industry Expectations
  Benchmark Suite" leveraging HPC, SPEC, Top500, AI/ML GPU leaderboards,
  and SNIA methodologies to support rigorous adherence and consistent
  results using standard performance testing tools.

- Define and deploy a standardized HPC "Workload Automation Pipeline"
  utilizing SLURM, Jenkins, Celery, RabbitMQ ("RMQ"), ClickHouse,
  Ansible, and related tooling.

- Define and deploy a standardized HPC "Telemetry + Analytics
  Aggregation + Data Visualization Processing Pipeline" utilizing
  Check_MK, SNMP, Elasticsearch + Kibana, Prometheus, InfluxDB, Grafana,
  and related tooling.

- Assess current organizational working patterns across Eng-Org,
  App-Team, and Sales-Org and provide a quarterly review to the VP and
  OCTO.

- Assess the current CORP collateral surface area, including SDKs,
  documentation, libraries, proofs of concept, and reference designs, as
  an ongoing initiative to improve the developer experience.

- Author white papers, research documents, and related technical
  materials, with a feedback loop to validate and improve internal and
  external documentation.

- Provide mentoring for junior through senior application and systems
  engineers, plus reasonable availability as an SME across storage,
  systems, database, and AI/ML domains.

- Provide ad hoc consultation to support Sales-Org customer scoping,
  implementation, and integration needs.

2. Term and Working Model

The engagement will begin on the SOW Effective Date and continue for an
initial six (6) month term. The Consultant will generally allocate forty
(40) hours per week to the engagement, subject to mutually agreed
priorities, holiday schedules, and reasonable scheduling adjustments.

The parties acknowledge that some workstreams in this SOW are ongoing or
iterative in nature. Accordingly, the Client and Consultant will jointly
prioritize and sequence the work so that the highest-value deliverables
are completed within the available time and level of effort.

Services may be provided remotely or through other mutually agreed
working arrangements. Any onsite work, travel, or work materially
outside the target level of effort will require prior written approval.

3. Scope of Services

3.1 Modular "Industry Expectations Benchmark Suite"

The Consultant will define, document, and deploy a modular benchmark
suite intended to measure representative performance, reproducibility,
and adherence to industry-standard methodologies across compute,
storage, and AI/ML use cases.

Illustrative deliverables:

- Benchmark taxonomy and workload catalog covering applicable HPC, SPEC,
  Top500-style, AI/ML GPU leaderboard-relevant, and SNIA-aligned
  methodologies.

- Benchmark execution standards, parameterization guidance, and runbooks
  designed to improve repeatability and result consistency.

- Result templates and reporting conventions for capturing baseline
  measurements and comparative findings.

- Initial implementation and/or deployment in designated Client
  environment(s), subject to infrastructure readiness and access.

- Validation summary describing observed results, known limitations, and
  next-step recommendations.

3.2 Standardized HPC "Workload Automation Pipeline"

The Consultant will define and deploy a standardized HPC workload
automation architecture using SLURM, Jenkins, Celery, RMQ/RabbitMQ,
ClickHouse, Ansible, and related tools as appropriate to the Client
environment.

Illustrative deliverables:

- Reference architecture and component interaction design for job
  submission, orchestration, execution, retry handling, and result
  capture.

- Deployment-ready configurations, automation artifacts, or initial
  implementation in designated environment(s).

- Operational workflows for benchmark or workload scheduling, execution,
  data collection, and reporting.

- Runbooks and handoff documentation sufficient to support repeatable
  operation and internal adoption.

3.3 Standardized HPC "Telemetry + Analytics Aggregation + Data
Visualization Processing Pipeline"

The Consultant will define and deploy a standardized observability and
analytics processing pipeline spanning infrastructure telemetry,
log/event aggregation, time-series monitoring, analytics storage, and
visualization.

Illustrative deliverables:

- Reference architecture covering Check_MK, SNMP, Elasticsearch +
  Kibana, Prometheus, InfluxDB, Grafana, and related integrations as
  applicable.

- Recommended data model, retention approach, dashboard structure, and
  operating practices for standardized reporting and analysis.

- Initial dashboards, visualizations, and/or deployment artifacts for
  designated environment(s).

- Documentation for data sources, ingestion paths, dashboards, and
  operational ownership considerations.

3.4 Cross-Functional Collaboration Assessment

The Consultant will assess current organizational working patterns
across Eng-Org, App-Team, and Sales-Org in order to identify friction
points, handoff gaps, role ambiguity, communication issues, and
opportunities to improve execution.

Illustrative deliverables:

- Current-state assessment and concise findings memo.

- Recommendations for cadence, decision ownership, communication flow,
  and escalation patterns.

- Quarterly review readout(s) presented to the VP and OCTO during the
  engagement term.

3.5 CORP Collateral Surface Area and Developer Experience

The Consultant will assess the current ‘surface area’ of CORP collateral,
including SDKs, documentation, libraries, proofs of concept, and
reference designs, as an iterative stretch initiative with the overall
goal of improving the developer experience.

Illustrative deliverables:

- Inventory and gap analysis of current CORP technical collateral.

- Prioritized recommendations and/or backlog themes to improve
  consistency, usability, completeness, and customer/integrator
  experience.

- Feedback loop for validating and improving internal and external
  documentation and supporting artifacts.

- Periodic progress updates and next-step roadmap recommendations within
  the engagement window.

3.6 White Papers, Research Documents, and Documentation Improvement

The Consultant will author white papers, research documents, technical
notes, and similar written materials as prioritized by the Client, and
will help create a feedback loop to validate and improve internal and
external documentation quality.

Illustrative deliverables:

- White papers, technical briefs, research documents, architecture
  notes, or similar written outputs as mutually prioritized.

- Document review comments, revision recommendations, and technical
  fact-checking input.

- Structured feedback loop to capture lessons learned and improve future
  documentation quality.

3.7 Mentoring and Knowledge Transfer

The Consultant will provide mentoring support for junior through senior
application and systems engineers, with an emphasis on practical
technical development and hands-on knowledge transfer.

Illustrative deliverables:

- Office hours, design reviews, pair-working sessions, workshops, or
  other mentoring interactions as appropriate.

- Guidance on benchmarking, automation, observability, storage, systems,
  database, and AI/ML topics relevant to the engagement.

- Knowledge transfer materials or summary notes where appropriate to
  support broader team adoption.

3.8 SME Availability and Sales-Org Consultation

The Consultant will maintain reasonable availability to organizational
staff in applicable subject-matter areas of focus and will provide ad
hoc consultation for Sales-Org customer scoping, implementation, and
integration needs, as capacity permits within the contracted level of
effort.

Illustrative deliverables:

- Advisory support for storage, systems, database, and AI/ML questions
  arising during the engagement.

- Participation in internal and customer-facing scoping discussions,
  solution reviews, or implementation planning sessions, when requested
  by the Client.

- Technical input intended to improve solution fit, implementation
  readiness, and integration clarity; provided as advisory support
  rather than as a standalone managed service or customer support
  commitment.

4. Indicative Delivery Plan

The following plan is illustrative and may be reprioritized by mutual
agreement based on Client needs, environment readiness, and emerging
business priorities.

  -----------------------------------------------------------------------
          Period          Primary Focus           Illustrative Outputs
  ----------------------- ----------------------- -----------------------
          Month 1         Discovery and Baseline  Stakeholder alignment,
                                                  current-state review,
                                                  environment access,
                                                  priority mapping,
                                                  benchmark/workload
                                                  definition, and
                                                  architecture framing.

        Months 2-3        Design and Initial      Benchmark suite v1,
                          Deployment              automation pipeline
                                                  design and initial
                                                  deployment artifacts,
                                                  telemetry pipeline
                                                  design, and initial
                                                  dashboards or data
                                                  paths.

        Months 4-5        Validation and          Validation runs,
                          Expansion               documentation
                                                  expansion,
                                                  collaboration
                                                  assessment findings,
                                                  collateral review,
                                                  white papers/research
                                                  outputs, and mentoring
                                                  support.

          Month 6         Handoff and Executive   Runbooks,
                          Review                  recommendations,
                                                  backlog or roadmap
                                                  items, knowledge
                                                  transfer, and
                                                  quarterly/executive
                                                  review presentation(s)
                                                  to the VP and OCTO.
  -----------------------------------------------------------------------

5. Governance and Reporting

- The Client will designate a primary engagement owner or point of
  contact responsible for prioritization, coordination, and consolidated
  feedback.

- The parties will maintain a regular working cadence, which may include
  weekly or biweekly syncs, technical reviews, design reviews, and
  status reporting as appropriate.

- The Consultant will provide concise updates on progress, risks,
  blockers, and recommended next actions.

- A quarterly review readout will be provided to the VP and OCTO during
  the engagement term.

6. Assumptions, Dependencies, and Constraints

- The Client will provide timely access to relevant systems,
  environments, credentials, documentation, repositories, data, and
  subject-matter stakeholders reasonably required for the services.

- The Client is responsible for providing or approving the hardware,
  software, licenses, network access, security approvals, and
  change-management windows required to implement or deploy
  deliverables.

- Any production deployment, change control, or release activity remains
  subject to Client approval processes and operational safeguards.

- Where this SOW references industry-standard benchmarks or
  leaderboard-adjacent methodologies, any formal external publication or
  third-party submission will remain subject to separate Client approval
  and third-party rules.

- The Consultant’s work product may include recommendations, reference
  architectures, configurations, code or scripts, runbooks, dashboards,
  presentations, reports, and live working sessions; not every
  workstream will result in a standalone written deliverable.

- Because certain activities are advisory and ongoing in nature, the
  parties will periodically rebalance priorities to remain within the
  contracted time and level of effort.

- Travel, onsite presence, after-hours support, and work materially
  beyond the initial term or weekly level of effort are excluded unless
  separately approved in writing.

7. Out of Scope

- Managed services, long-term production operations ownership, or 24x7
  operational support.

- Hardware procurement, datacenter installation, or unrelated
  infrastructure build-out not directly tied to the services described
  in this SOW.

- Binding commitments to end customers, contractual promises to third
  parties, or customer project delivery obligations not expressly
  authorized by the Client.

- Ongoing maintenance, enhancement, or support of deliverables after the
  end of the engagement term, except as separately agreed in writing.

8. Acceptance and Change Control

Commercial terms, invoicing, payment terms, intellectual property,
confidentiality, warranties, limitation of liability, and any formal
acceptance procedures will be governed by the Agreement.

Where the Agreement does not specify a separate acceptance procedure,
the Client will review deliverables promptly and provide consolidated
feedback within five (5) business days after delivery. Deliverables will
be deemed accepted when they materially conform to the scope described
in this SOW, subject to any documented punch-list items or agreed
revisions.

Any material change to scope, deliverables, timeline, or level of effort
will require written approval by both parties.

9. Signatures

The parties acknowledge and agree to this SOW as of the dates written
below.

By: ________________________________ 
Name: ______________________________ 


[Consultant Legal Name]
Title: _______________________________
Date: _______________________________
By: ________________________________ 

[Client Legal Name]
Name: ______________________________
Title: _______________________________
Date: _______________________________

Choose your failure mode and you will find it quickly.

Please Skip the Philosophy

A valid request. Perhaps you don’t care about this verbosity of thought, or considerations therein, or maybe you’re aware of the reprisals and follies and want to move on to the engineering discussion? If so, simply scroll down to the section titled, “How Does This Relate to Memory”. Note that in doing so you will miss a blurb from Office Space.

Endeavor to Endure the Philosophy

A fair degree of the human experience involves a process by which those of “early-phase cognition” attempt to self-obfuscate their “desire for desire”, attempting to live a life of nebulous and fleeting happiness, non-blissfully and unaware that it is this desire to feel and experience desire itself which has cornered their experience into a lifetime of endless searching, yearning, wanting, never satiated, eventually growing old and stale and losing their imaginative or creative or physical edge.

This cycle of The Great Chase is one basis of Sisiphean frustration; one which - until it is surpassed - which requires being acknowledged, accepted, and integrated into their forward movement

Some understand that to exist with desire, that all-controlling force, seemingly inescapable force which presses down upon the rock, forever trapping your perspective into the wheel of the unknown yet forever traveled, repeating mistakes which others warned against, refusing awareness of one’s own misfortunate mentality and guiding-line towards an inevitable failure; experiences from which all lessons are never to be learned.

Go ahead. Do it. Don’t learn from others. Do everything yourself, do it with the veracity by which you have become convinced is correct - do everything as though this is the first time anyone has ever tried to achieve the new shiny thing. Surely, this must be the path towards truth, self-realization, self-actualization, self-assurance, and that desperation of desire for freedom from desire to desire everything… and then… surely success is ensured?

No, that modality of mentality would achieve its opposite effect, and it would be a waste of everyone’s time; because despite the ego-driven main-character syndrome which has become so performatively boring in the past two decades of social media, it’s not all about You, or Me, it’s about Everyone alive and even those who we have loved who died.

Mr. Gibbons

Peter Gibbons:
It’s not just about me and my dream of doing nothing. It’s about all of us… Michael, we don’t have a lot of time on this earth!

We weren’t meant to spend it this way. Human beings were not meant to sit in little cubicles staring at computer screens all day, filling out useless forms and listening to eight different bosses drone on about about mission statements.

Michael Bolton:
I told those fudge-packers I liked Michael Bolton’s music.

Peter Gibbons:
Oh, That is not right, Michael.

Purposeless self-serving inefficiency which is predicated on assured failure wastes time that humanity does not have - we do not have time for oppositional defiance induced mediocrity. Stop the gamification.

The Mindset of Being Simply Wrong

Instead of repeating the pattern described above, grow the fuck up and learn how to effectively learn from others, to learn from the past, to learn using a reinforcement framework of concept-action-processing. “Thems may be big words”, but this is not a complicated concept. One does not have to choose the path of stupidity; ignorance - we’re all ignorant in all manner of ways and that’s inescapable; however when one ignores the data, ignores the lesson, and chooses the path before them via oppositional defiance, that’s called “being stupid”. It’s stupid not to learn from others.

How Does This Relate to Memory

Memory is everything. The conscious experience, the neural network running inside our brains, the execution of “The Default Mode Network”, requires effective memory - short term a well as consolidation to long-term memory.

So, when the industry was informed of the untimely discontinuation of Intel’s Optane NVDIMMs… we were all screwed. Until Q2 2026. Now it’s Q2 and finally CXL non-volatile memory is available in sufficiently equivalent density, performance, and lifespan to exceed Optane’s role, and it is available in PCIe Gen-4 and Gen-5 and soon… Gen-6.

An Introduction to CXL

• It’s not HBM, it’s not 3D-Xpoint, it’s not SRAM, nor vRAM
• CXL is packaged in PCIe, U.3, and EDSFF formats
• CXL offers dense capacity, low-latency, and exceptional I/O
• CXL is made for Ai/HPC/OLTP memory acceleration
• CXL is the new hotness, it is the new standard, it is performance

HPC Cluster Storage Acceleration

CXL is being built into (not next-gen, now-gen) storage appliances and network protocol accelerator systems used in High Performance Computing, Ai/ML which leverages HPC tech, and Supercomputers.

CXL in Telco DSP as Protocol Caches

CXL is perfectly suited for digital signals processing on a range of network protocols and their intra-chassis and inter-chassis RDMA driven networks. There’s one person who I know that hates it when I bring up RDMA - truly an example of being willfully ignorant. Moving on…

CXL Memory in Telco ATAC Chassis

• 14 slots in 19-inch rack ≈→ ∆ {$N total trays per 42U ≈ μ19}
• 16 slots in 23-inch ETSI rack ≈→ ∆ {$N total trays per 42U ≈ μ23}
• Chassis heights from 4U to 15U ≈↑ ∆ [[ μ4, μ19, μ23 ]]
• Telco grade Power & Cooling Efficiency & Redundancy ++
• Mil-DoD Compliant Physical & Operational Security ++
• Industrial NEBS Environmental Operating Ranges ++
• Zero-Downtime Serviceability & Upgrade/Maint ++
• Intra+Inter-Chassis Redundancy (Modular Scale) ++
• Geo-Deploy Designed for regional XOR continental mesh ++

So, Why Should We Listen to Her?

Becausee I’ve been building and iterating these types of storage architectures for nearly twenty years of my career. If you want to do it all from scratch or buy this type of storage performance for 10x the cost, go ahead; the market exists and you can be the big winner.

However, if you don’t want to pay a premium for non-OpenSource storage clusters which are capable of serving and scaling from 10s to 100s of Petabytes, then this is the direction by which engineering solutions arrive at the datacenter. My storage designs range from Oracle’s Exadata & Exascale clusters which have been adapted to run non-standard protocols, to fully purpose-built data-acceleration clusters which use COTS building blocks to provide similar performance and scale at a fraction of the cost (not including consulting, engineering scope, and top-to-bottom fully-automated storage architecture deployments).

As networking has improved, NAND designs have advanced, and costs have reduced on $/TB and $/Watt… yet always… always the storage sizes grow in density and capabilities. This is natural and expected.

Project Coherent Storage, an Evolution

Six primary iterations, tracking the advancement of NAND acceleration, Network Acceleration, Protocol Offloading, Non-Volarile RAM, and now we see CXL capable of being deploying into reference architectures by which all others may be measured.

• Project Coherent Storage v1, (2010-2012)
• Project Coherent Storage v2, (2011-2015)
• Project Coherent NAND, (CMPLX.io) (2015-2017)
• Project Coherent IO-Max v1, w/ Optane PMem (2020-2024)
• Project Coherent IO-Max v2, w/ Optane PMem (2024-2025)
• Project Coherent Storage v3, w/ CXL AIC/U3 (2026)

Development Details - Concise Timeline

A little bit of engineering nerdery for those who care… and btw, the community myth that ZFS cannot fulfill a role as a multi-node clustered filesystem are wrong.

What About the “ZFS Can’t Cluster” Myth?

The “Clustered ZFS” operating modality for multi-controller interactions (not limited to A/B active/passive or even active/active with modified pool-async hooks) requires advanced network hardware and engineering experience which most are not aware of, and so the myth perpetuates; but it is wrong and it has been wrong for as long as SAS drives have had dual-heads and shelves have had 4+ SFF connectors ready for multi-multipath-chaining - and enterprise NVMe drives are also available in this arrangement for the same purpose.

Project Coherent Storage v1 (2010-2012)

1. Coherence Grid (DDR3 (Tier1)) + Custom Relational-Vector DBMS
2. Sunfire ZFS controllers w. Multi-Protocol-Net Interlinks (IB, IPoIB, mNFS+, FC, FCoE)
3. Geo-Dist-Nodes w. DDR3-PCIe SuperCap “Hot Cache Layer” (Tier2), w. Sunfire X4600/M2 @ 64 DIMMs Hotswap
4. Geo-Dist-Nodes w. SAS SSDs (Tier3) + SAS Spinner (Tier4)
5. Controller A/B+ nodes w. SAS-HBA to LTO-N (Tier5)

Project Coherent Storage v2 (2011-2015)

1. Coherence Grid (DDR3 (Tier1)) + Relational-Vector DBMS
2. Sunfire ZFS controllers w. Multi-Protocol-Net Interlinks (IB, IP Native L2/L3, mNFS+, FCoE)
3. Geo-Dist-Nodes w. FusionIO Stacked “Hot Cache Layer” (Tier2)
4. Geo-Dist-Nodes w. SAS SSDs (Tier3) + SAS Spinner (Tier4)
5. Optional A/B nodes w. SAS-HBA to LTO-N (Tier5)

Project Coherent NAND - CMPLX.io (2015-2017)

1. Coherence Grid DDR4 (Tier1) + Relational-Vector DBMS
2. Native ZFS controllers w. Multi-Protocol-Net Interlinks (IB, IP Native L2/L3, mNFS+, iSER)
3. Geo-Dist-Nodes w. NAND Stacked “Hot Cache Layer” (Tier2)
4. Geo-Dist-Nodes w. SAS3 SSDs (Tier3) + SAS3 Spinner (Tier4) [+ optional SAS3 HBA to LTO for Iron Mountain (Tier5)]
5. This architecture was a basis for one of my former startups, which was sold to private equity and built-upon for their HFT clusters under very tight NDA + Confidentiality Contract

Project Coherent IO-Max Storage Architectures v1, 2, 3

These storage architectures share core foundations, with iterations upward as Non-Volatile Memory was introduced on a wider scale into my designs. The initial design for v1 featured the first generation of Intel Optane 3D-XPoint (Series 100), v2 with Series-200, and now CXL taking the place of Optane PMem SKUs.

The details of each design from v1 and v2 may be expanded upon later, simply for historical purposes, and I have other things to do right now… but if you’ve made it this far into the post and want to see the specification ADRs and related research, which is RAG-pipelined and iteratively deployed on real hardware for revisions and regression testing to validate LLM Inference usage & advancements with the now-opensource Coherence-CE version… scroll down for a link to v3’s repo.

• Project Coherent IO-Max v1, (2020-2023)
• Project Coherent IO-Max v2, (2023-2025)
• Project Coherent IO-Max v3,CXL (2026)

Adjusted Query: how much storage space would the user deploy to an existing multi-Petabyte array, with consideration to physical limitations in systems design which prioritize two common architectural decision targets: More or Faster?

(1) maximizing raw-block capacity
(2) maximizing I/O performance

TL;DR

Between 4-5PB is the simplest answer, but those are just numbers. Could be 10x either, but that’s not realistic - it’s conjecture. The full picture requires insight into physical limits of the transfer time, storage space, rack space, and rotational speeds involved; spacetime could be analyzed by the right minds if this were JPL ^_^ .. instead of limited to hardware I’ve acquired over a small handful of years.

Hypothetical or Reality

Let’s say that I contacted the datacenter and ordered additional power circuits to light-up additional drive shelves, which are already installed (minus the box of drive trays which are in a massive box on a parts shelf in my home office). Ok, reality checks out there.

The circuit drops would go to the central storage rack which is currently at 84% power limit. I know this because power is monitored and validated via a combination of Check_MK + Prometheus + SNMP exporters, which poll power stats remotely from the cabinet’s APC ATS, the rack’s per-port metered and switched outlet PDUs, and then if I really want to I can look at the DC’s circuit stats (via MRTG) which are tracked for the cabinet. Reality checks out there decently enough. Moving on…

The Hypothetical

First we’ll cover some raw numbers and theoretical maximums by looking at two high-level approaches required for “More or Faster”.

The astute reader may wonder why we’re not discussing >100TB QLC NVMe drives or ultra-dense E2 ruler-style 2U chassis, and the answer is that those calculations are generally similar in equation workflow (eg: how many things of X can we put into Y at A,B,C performance/size/latency), however those hardware types are not currently racked in my real-world systems at an interesting scale. However, migrating to denser NVMe arrangements will likely be part of an eventual down-scaling forklift operation whenever the SAS3 arrays are no longer useful for my personal labs or colo bills.

Production is another story, where large format SAS3 is still relevant and being deployed and improved over time; in the last several years we’re seen drive capacity double, along with some innovative designs. It’s not often considered exciting, but it lasts A LONG TIME, and the NSA loves archiving the internet on this medium over in their Utah storage complex (overheard, surely).

So, back to the math…

Decision: Optimize for Faster (prioritize IOPs + bw)

Aggregate per-rack w/ 10x shelves of raw 4K block = 3,824 TB == 3.72 PB

Decision: Optimize for RAW Storage Capacity

How about the same hardware focused on max-total raw space with less regard to per-drive IOPs (perhaps we offset the latency reduction per-drive via larger L1ARC and L2ARC buffers on the head-nodes, which becomes a loaded tangent)… then the rack could be upgraded to host 4.85PB @ $800/m MRC ($400/m cabinet + $400/m amperage)**.

We’ll also save the conversation about concessions made at time of contract signing where I might / maybe / surely have saved on recurring costs by locking-in a multi-year multi-rack contract. Sales team, thanks!

What is this MRC, MRR?

MRC in this situation is the ‘Monthly Recurring Cost’ for power + rack. This value does not include the cost of acquired hardware, which is variable on supply-chain sourcing and other factors. I’m also not going to describe potential MRC or MRR offsets which factor into profit margin potential and/or influence EBITDA.

Ok, what is MRR and EBID..? Fine, at least with MRR it’s ‘Monthly Recurring Revenue’, and one can read all about EBITDA elsewhere.

What about Flash-Cache or Optane or Non-Volatile blah blah blah

Sure, those are considerations, where applicable. Some footnote notes…

• It’s prudent to deploy storage controllers with SLC NVMe or PMem/NVDIMM persistent cache (which are often a baseline performance requirement). These are sized anywhere from cumulative 1-12TB per-controller, depending on available PCIe lanes and/or NV-DIMM slots.
• It’s a good idea to make scoping decisions like that ahead of time, instead of later realizing that the controllers’ lanes are maxed out or all DIMM slots are used (or you bought 128GB NVDIMMs instead of 512GB, thinking “8x 128GB should be fine for PMem” or the other fun one about ignoring (or being unaware) about GHz speed reduction by running 4-DPC in a 2-DPC max-perf system.
• Those determinations may be either workload dependent, or in my personal life sometimes it qualifies as the “because I wanted to” type of maximalist methodology, or the “this is relevant for future proofing my own rationalization on fun hardware”, or could be considered a baseline in pre-sales because “they always want more performance eventually; just bake it in from the start”. It varies, but remember where you started and where any future requirements for performance headroom exists.

Where’s the Math?

Validation on spec baselines can be tedious, but anyway… the DE3-24C and DE3-24P have very shiny and well made Dual-Redundant IOMs with 4x SAS3 SFF-8643 connectors.

Specs for our SAS3 Fabric with DE3-24 Shelves

• One SAS-3 lane (PHY): 12 Gbit/s ⇒ 1.5 GB/s (raw line-rate, decimal GB)
• External SAS x4 port connector: 4 PHY × 12 Gbit/s ⇒ 48 Gbit/s ⇒ 6 GB/s
• Each DE3-24{C/P} has 2× IOM, with each IOM having 4× SFF-8643 ports

Drive Specs - Seagate EXOS & Oracle SSD

Scenario A – Maximum RAW Block Space

Before going into the theoretical weeds, let’s look at some per-shelf media limits.

1) Oracle DE3-24P shelf w/ 24TB Spinners**:** 24 × 285 MB/s = 6.84 GB/s (media-limited)

2) Oracle DE3-24C shelf w/ SSDs:

• Read: 24 × 2150 MB/s = 51.60 GB/s → capped by fabric at 36 GB/s (N=3)
• Write: 24 × 1980 MB/s = 47.52 GB/s → capped by fabric at 36 GB/s (N=3)

Scenario B: Maximum Performance

Hardware has limits, and we’re looking at 8× DE3-24P w/ 18TB Mach.2 HDD + 2× DE3-24C w/ 7.68TB SSD (N=3)

Aggregate sequential bandwidth (N=3)

• Mach.2 18TB HDD shelf: 24 × 554 MB/s = 13.296 GB/s (media-limited; below 36 GB/s fabric cap)
• SSD shelf: remains fabric-capped at 36 GB/s read and write (N=3)

Aggregate Random R/W 4K IOPS (N=3)

Fabric-derived 4K IOPS cap per shelf (upper bound):
36 GB/s @ 4KB = approx 8.79M IOPS

• SSD shelf
  • Read: 24 × 450k = 10.8M IOPs → capped to 8.79M IOPs
  • Write: 24 × 95k = 2.28M IOPs → media-limited (below fabric cap)
• Mach.2 HDD shelf (4K @ QD16)
  • Read: 24 × 304 = 7,296 IOPs
  • Write: 24 × 560 = 13,440 IOPs

Let’s See the Deltas!

What’s “a delta”? Sometimes it’s the headwaters of a bay, where the river distributes across a flood-plane into many separately-unified channels, eventually giving way to deeper waters and…. no no not that kind.

Delta as a value in storage is often used as a marker to show the differential of two functions which indicates directional change. At least, that’s what comes from my brain while I sit here having an ice cream sandwich, but it has some alternate terminology which applies irrespective of ice cream hour:

Delta uppercase Δ, lowercase δ
Uppercase: a change of any changeable quantity
Lowercase: the central difference for a function

~ The discriminant of a polynomial equation or symmetric difference of two sets
~ The determinant of the matrix of coefficients of a set of linear equations

Scenario B vs Scenario A (with N=3)

• Total sequential BW: 126.72 → 178.368 GB/s (increase +51.648 GB/s)
  • Primary determinant: per-shelf BW 6.84 → 13.296 GB/s (Mach.2 dual-actuator).
• SSD tier remains fabric-capped at 36 GB/s per SSD shelf under N=3; increasing N (or adding more independent uplinks with additional HBAs) is what changes that cap.

1. Faster array = less space @ higher IOPs and BW == 178 GB/s @ 3.8PB
2. Larger array = more space @ lower IOPs and BW == 126 GB/s @ 4.9PB

Equations w/ decimal units: 1 TB = 1000 GB

• jaja usually we work in 1024 (but I’m tired today, so not right now)

t_1TB(s)      = 1000 / BW_GBps
t_full(s)     = (Capacity_TB * 1000) / BW_GBps
t_full(hours) = t_full(s) / 3600
BW_density    = BW_GBps / Capacity_TB   (GB/s per TB)
              = 1000 * BW_density       (MB/s per TB)

So… what’s the answer? More, Faster?

Is bigger is better? Sometimes. Sometimes you need it faster. Sometimes you want both.