Data caching at The Edge isn’t always from Fastter or Cuntter or Cloudstter or whichever CDN happens to be serving your IG, Safeway, NFL, Reddit, Anthropology, or 338% of the internet’s concurrent traffic being pointless traversals like jQuery minified & brotli-fied and sent back and forth endlessly through Internet Exchanges.

Edge caching is not limited to Anthropic or OAI or whoever else’s Load-Balancer API-side “prompt cache (not yet! after LB!)” packets which “those people” are constantly complaining about, nor the hyperscalers who whitebox big CDN networks under their own headline IP.. those are not what I’m talking about when I talk about Edge Caches.

These little embedded devices which collect climate metrics, air quality data, assembly robotics, municipal awareness and telemetry data, and everything else from a very long list… they need read-heavy churn-heavy persistent storage on a filesystem optimized for tiny sized industrial NAND.

More examples for the ones of all sizes making modern life possible: that ATM you went to, the parking lot video surveillance systems, traffic light coordinator nodes, telco edge ONT routers and OOBs with SDNs, and a thousand other examples abound that are functionally ignored.

Also, robots. And factory hardware. And factories making robots that make tiny factories? Hopefully so.

Embeds Reduce CAPEX & OPEX Efficiently

So, those embedded devices usually are designed for efficiency, while not too often designed to be hammering on NAND for write-heavy storage workloads. This has caveats: devices explicitly designed for write heavy I/O on purpose, and therefore not what I’m referring to, and cases where the provisioning and/or configuration stages of fleet deployment are improperly aligned to hardware realities.

Why? Low wattage, moderate caches, and a desire to not burn out NAND when the device is located in a difficult-to-get-to location for service techs working on hardware replacement.

Most of all - we want efficient storage for embedded devices so that their operationally collected data can be stored in memory with a ECC write-back cache with “operationally consistent” garbage collection and non-invasive wear-leveling, all while the transactions are paged/queued to F2FS before async’ing upstream to a centralized data collector for aggregation.

Predator Drones Too?

Yes and no, they send their comms to LEOSAT before humanity’s “furthest from the menial and meaningless non-Elysium people” edge devices cache, aggregate, and compute roll-up analytics with inline encryption and compression via DPU offload streaming.. all the way down to LANDSAT.

Edge devices have their own little ecosystem separate from the manner in which datacenter storage network-enabled hardware operates. All by design, and always room for improvement.

Many people consider this boring tech, and would rather move fast and break things — your time is limited, people are sick of perfectly workable systems breaking all the fuckin time and they’re tired of being A/B tested on in production — doing those things makes your company’s testing policy look like an asshole. Everyone, stop being assholes. Thanks!

These Things - Why Should We Care?

You like your cell phone? How about GPS? SAT-NAV? Skylink? Drones operating up above withour your awareness? How about sector antennas for the police and emergency services? Embedded systems are streaming, caching, processing, and relaying data all over the globe and up into LEO constellations.

You don’t have to care, and you shouldn’t have to care about FSFS or drones unless you work with that side of tech, or its adjacent use cases - like living in the modern world. Everyone should care because knowledge is fundamentally important, if for no other reason at all. Do you know how the light-switch on the wall works? Same deal. Stop making excuses for saying you don’t know or don’t care or you will be left behind. No one is saving us but ourselves, and we must have knowledge to do so.

Recent Adventures with Standards-Definitions

I love automation. In part this is because I prefer efficiency over repetitive manual bullshit. Efficiently handling requirements means more time to do what I love:

• Hanging out by the pool with a tablet or waterproof e-ink reader.
  Way back in 2018 my e-book tally was 62 novels over the calendar year. That number has dropped off recently for trivial reasons, but the point remains that I love to read, and I love to lay out by the pool.
• What I do not love:
  Wasting time when I could be at the pool. ♥‿♥

there’s only room for two hats in my life - and this is the one for sunning

• What I’ll never not love:
  The poem on my torso, dedicated to the manner in which my departed twin lived & loved.

Standards for NAND? Standards for SD-NAND!

To the point, here’s a script that had to be blasted out a while back, while imaging so many too many industrial-spec SD/TF-cards for a fleet of distibuted data collectors. These fun little RK3588 based systems have a NPU and only an eMMC drive for the OS.

While the eMMC in question has decent specs, if it fails then it’s sadly more expensive to de-solder and re-solder and re-image the OS on eMMC, and so on these systems we use the SD NAND for the following data:

• Ephemeral scratch space and logs (eg: /var/log, /var/tmp, /tmp - nope, these go to tmpfs)
• Cache directories which are aptly named .cache or .Cache סּ_סּ
• Files which must be staged locally but have no business being on the OS mount points.
• Several other use cases that I don’t feel like describing right now; so here’s the discussion for those who love to know more more more.

### F2FS - Embedded Filesystem Performance Options

- Ref: https://docs.kernel.org/filesystems/f2fs.html

#### Option Flags and Reasoning

##### For High-Churn I/O
This is less ideal, but sometimes a `tmpfs` overlay isn't an option, so it's better than burning out the eMMC. A good industrial SD card has ECC, wear-leveling, checksum validations, write-amplification controls, etc. Don't care for those features? You may one day...

Example Writes: data persistent log-sync, package builds on-device, container writable layers, time-series compressible metrics, telemetry buffers, or small-file caches, opt for the non-default `-o 10` flag. 

| Option                      | Rationale              |
| --------------------------- | -----------------------|
| `-l DATA`                   | Stable label for `/etc/fstab`; avoids relying on device enumeration order. UUID is an alternative, but requires programmatic parsing of `blkid` output, or manually specifying a UUID to pass to the FS creation process.. or you can simpy use a GPT label. |
| `-o 5`                      | Reserves 5% of the F2FS volume for the F2FS cleaner. `mkfs.f2fs` documents this as hidden overprovision space used by the cleaner. |
| `-t 1`                      | Performs discard/TRIM at format time; defaults discard to enabled, but some card readers/cards mishandle it; use `-t 0` if format stalls or logs errors |
| `extra_attr`                | Required by several modern F2FS features. |
| `inode_checksum`            | Adds inode metadata checksums; useful for removable flash and unattended systems. |
| `sb_checksum`               | Adds superblock checksum support. |
| `flexible_inline_xattr`     | Modern xattr layout; useful with `extra_attr` and future feature expansion. |
| `inode_crtime`              | Adds inode creation-time support; low cost, useful for forensic/debug visibility. |
| `lost_found`                | Creates lost+found support for recovery workflows. |
| `compression`               | Good for compressible write-heavy data; poor fit for already-compressed media, model files, archives, database pages, and random-write workloads. |
| `compress_algorithm=zstd:3` | Validate performance of zstd compression options before using for production purposes. |
| `compress_log_size=%u`      | Support configuring compress cluster size. The size will be `4KB * (1 << %u)`. The default and minimum sizes are 16KB. |
| `checkpoint_merge`          | Self-describing (or check the refs)  |
| `nocheckpoint_merge`        | Disable checkpoint merge feature. |

- `checkpoint=%s[:%u[%]]`: Set to "disable" to turn off checkpointing. Set to "enable" to re-enable checkpointing. Is enabled by default. 

- `checkpoint_merge`
When checkpoint is enabled, this can be used to create a kernel daemon and make it to merge concurrent checkpoint requests as much as possible to eliminate redundant checkpoint issues. Plus, we can eliminate the sluggish issue caused by slow checkpoint operation when the checkpoint is done in a process context in a cgroup having low i/o budget and cpu shares.

- `compress_algorithm=%s`
- `compress_algorithm=lzo-rle`

Control compress algorithm, currently f2fs supports "lzo", "lz4", "zstd" and "lzo-rle" algorithm.

- `compress_algorithm=%s:%d`

Control compress algorithm and its compress level.. only "lz4" and "zstd" support compress level config and those algos may be in your kernel config and may not.. but you'll find out quickly if you don't already know the answer.

========= =========== algorithm level range ========= =========== lz4 3 - 16 zstd 1 - 22 ========= ===========

- `compress_log_size=%u`
Support configuring compress cluster size. The size will be `4KB * (1 << %u)`. The default and minimum sizes are 16KB.

- `compress_extension=%s`

Support adding specified extension, so that f2fs can enable compression on those corresponding files.
If all files with `.ext` has high compression rate, we can set the `.ext` on compression extension list and enable compression on these file by default rather than to enable it via ioctl.

For other files, we can still enable compression via ioctl. Note that, there is one reserved special extension `*`, it can be set to enable compression for all files.

- `nocompress_extension=%s`

Support adding specified extension, so that f2fs can disable compression on those corresponding files, just contrary to compression extension.
If you know exactly which files cannot be compressed, you can use this.

> The same extension name can’t appear in both compress and nocompress extension at the same time.

If the compress extension specifies all files, the types specified by the nocompress extension will be treated as special cases and will not be compressed.
After add nocompress_extension, the priority should be: `dir_flag < comp_extention,nocompress_extension < comp_file_flag,no_comp_file_flag`.

- `compress_chksum`
Support verifying chksum of raw data in compressed cluster.

- `compress_mode=%s`
Control file compression mode: "fs" and "user". In "fs" mode (default), f2fs does automatic compression on the compression enabled files. In "user" mode, f2fs disables the automaic compression and gives the user discretion of choosing the target file and the timing.

The user can do manual compression/decompression on the compression enabled files using ioctls.

- `compress_cache`
Support to use address space of a filesystem managed inode to cache compressed block, in order to improve cache hit ratio of random read.

- `inlinecrypt`
When possible, encrypt/decrypt the contents of encrypted files using the blk-crypto framework rather than filesystem-layer encryption.
This allows the use of inline encryption hardware. The on-disk format is unaffected.

- `atgc`
Enable age-threshold garbage collection, it provides high effectiveness and efficiency on background GC. No one likes garbage.

- `discard_unit=%s`
Control discard unit, the argument can be "block", "segment" and "section", issued discard command’s offset/size will be aligned to the unit, by default, "`discard_unit=block`" is set, so that small discard functionality is enabled.

For blkzoned device, "`discard_unit=section`" will be set by default, it is helpful for large sized (fucccck off with SMR, less fucks to ZNS) devices to reduce memory cost by getting rid of fs metadata supports small discard (ouch, ever get burned by SMR? yeah... just don't bother).

- `age_extent_cache`
Enable an age extent cache based on rb-tree. It records data block update frequency of the extent per inode, in order to provide better temperature hints for data block allocation.

- `errors=%s`
Specify f2fs behavior on critical errors. This supports modes: "`panic`", "`continue`" and "`remount-ro`", respectively, trigger panic immediately, continue without doing anything, and remount the partition in read-only mode.

By default it uses "`continue`" mode, not my choice.

====================== ===============
mode continue remount-ro panic ====================== ===============
access ops normal normal N/A syscall errors -EIO -EROFS N/A mount option rw ro N/A pending dir write keep keep N/A pending non-dir write drop keep N/A pending node write drop keep N/A pending meta write keep keep N/A ====================== ===============

- `nat_bits`
Enable nat_bits feature to enhance full/empty nat blocks access, by default it’s disabled.

- `lookup_mode=%s`
Control the directory lookup behavior for case-folded directories. This option has no effect on directories that do not have the casefold feature enabled.

======================================== Value Description

perf (Default) Enforces a hash-only lookup. The linear search fallback is always disabled, ignoring the on-disk flag. compat Enables the linear search fallback for compatibility with directory entries created by older kernel that used a different case-folding algorithm. This mode ignores the on-disk flag. auto F2FS determines the mode based on the on-disk SB_ENC_NO_COMPAT_FALLBACK_FL flag.

----

#### Benchmarking F2FS Compression

Always validate the compression algo used before using those options, as performance of one algo vs anther can vary widely across different CPU generation and architectures, as well as for hosts with algo/hash compression offloading engines. We can cover benchmarking these in another script.

> lzo, lz4, zstd, lzo-rle algorithms

##### Create Partitions

DEV=/dev/sdX sgdisk –zap-all “$DEV” sgdisk -a 2048 -n 1:2048:-16G -t 1:8300 -c 1:DATA “$DEV” partprobe “$DEV”

##### Create FS

mkfs.f2fs -f
-l DATA
-o 10
-t 1
-O extra_attr,inode_checksum,sb_checksum,flexible_inline_xattr,inode_crtime,lost_found,compression,compress_algorithm=zstd:3 “${DEV}1”

#### FS Tab Entry

##### Conservative defaults

LABEL=DATA /srv/data f2fs rw,noatime,nodiratime,lazytime,nofail,background_gc=on,gc_merge,checkpoint_merge,flush_merge,mode=adaptive,fsync_mode=posix,nodiscard 0 0

##### Compression-capable variant

LABEL=DATA /srv/data f2fs rw,noatime,nodiratime,lazytime,nofail,background_gc=on,gc_merge,checkpoint_merge,flush_merge,mode=adaptive,fsync_mode=posix,nodiscard,compress_algorithm=lz4,compress_chksum,compress_mode=fs 0 0

#### Selectively Mark Compressible Directories
Do not enable compression broadly for LLM model files, videos, archives, SQLite databases, or opaque binary blobs unless benchmarks show a benefit.

mkdir -p /srv/data/_logs /srv/data/_caches.compressible chattr +c /srv/data/_logs /srv/data/_cache.compressible

So, Maybe This is Scriptable?

It sure is, or by Ansible playbook, or Salt sync, perhaps even a CFengine golden image.

That’s a lot of blah blah blah. How about that script?

Efficiency, indeed. Here ya go. The benchmarking script is for another day, or you can check out this workload primer about the filesystem from USENIX (w/ link to slide deck), and a benchmarking script which has been customized for F2FS.

#!/usr/bin/env bash
# -*- mode: sh; sh-basic-offset: 4; sh-shell: bash; coding: utf-8; indent-tabs-mode: nil; -*-
# ----------------------------------------------------------------------- #
set -euo pipefail
# ----------------------------------------------------------------------- #
# Purpose: POSIX-syntax wrapper to partition and format a non-boot SD/microSD card as F2FS with optional compression.
# NAND Profile: embedded I/O, kernel 6.1+, ECC industrial spec controller.
# WARNING: this destroys data on the selected block device, so be aware.
# ----------------------------------------------------------------------- #

usage() {
    cat <<'USAGE'
Usage:
  format-f2fs-sd-data.sh -d /dev/DEVICE [-l LABEL] [-r RESERVE_GIB] [-o F2FS_OP_PERCENT] [-c]

Examples:
# ----------------------------------------------------------------------- #
  ./format-f2fs-sd-data.sh -d /dev/sdX -l DATA -r 16 -o 5
  ./format-f2fs-sd-data.sh -d /dev/mmcblk0 -l DATA -r 24 -o 8 -c

Options:
# ----------------------------------------------------------------------- #
  -d  Whole block device to erase and partition: /dev/sdX -OR- /dev/mmcblk0
  -l  F2FS label. Default: DATA
  -r  GiB left unpartitioned at end for card FTL spare area. Default: 16
  -o  F2FS hidden overprovision percentage for the F2FS cleaner. Default: 5
  -c  Enable F2FS compression feature at format time. Mount options and chattr are still required.

Dependencies:
# ----------------------------------------------------------------------- #
  sgdisk, partprobe, mkfs.f2fs, lsblk
# ----------------------------------------------------------------------- #
USAGE
}

# Basic Variables
# ----------------------------------------------------------------------- #
LABEL=DATA
RESERVE_GIB=16
OP_PERCENT=5
COMPRESS=0
DEVICE=

# Define ARGV Mappings
# ----------------------------------------------------------------------- #
while getopts 'd:l:r:o:ch' opt; do
    case "$opt" in
        d) DEVICE=$OPTARG ;;
        l) LABEL=$OPTARG ;;
        r) RESERVE_GIB=$OPTARG ;;
        o) OP_PERCENT=$OPTARG ;;
        c) COMPRESS=1 ;;
        h) usage; exit 0 ;;
        *) usage; exit 2 ;;
    esac
done

# Sanity Checks
# ----------------------------------------------------------------------- #
if [ -z "$DEVICE" ]; then
    usage >&2
    exit 2
fi

case "$DEVICE" in
    /dev/sd?|/dev/sd??|/dev/mmcblk[0-9]|/dev/nvme[0-9]n[0-9]) ;;
    *)
        echo "Refusing suspicious device path: $DEVICE" >&2
        exit 2
        ;;
esac

# Validate Deps & Check Limits
# ----------------------------------------------------------------------- #
for cmd in sgdisk partprobe mkfs.f2fs lsblk; do
    if ! command -v "$cmd" >/dev/null 2>&1; then
        echo "Missing required command: $cmd" >&2
        exit 1
    fi
done

if [ ! -b "$DEVICE" ]; then
    echo "Not a block device: $DEVICE" >&2
    exit 1
fi

case "$RESERVE_GIB" in
    ''|*[!0-9]*) echo "Reserve GiB must be an integer" >&2; exit 2 ;;
esac
case "$OP_PERCENT" in
    ''|*[!0-9]*) echo "F2FS OP percentage must be an integer" >&2; exit 2 ;;
esac

case "$DEVICE" in
    /dev/mmcblk*|/dev/nvme*) PART=${DEVICE}p1 ;;
    *) PART=${DEVICE}1 ;;
esac

# Define Default F2FS Options
# ----------------------------------------------------------------------- #
FEATURES='extra_attr,inode_checksum,sb_checksum,flexible_inline_xattr,inode_crtime,lost_found'
if [ "$COMPRESS" -eq 1 ]; then
    FEATURES=${FEATURES},compression
fi

# Make Sure...
# ----------------------------------------------------------------------- #
echo "About to DESTROY all data on: $DEVICE"
lsblk -o NAME,SIZE,MODEL,SERIAL,TRAN,RM,RO,FSTYPE,MOUNTPOINTS "$DEVICE" || true
echo
echo "New partition: $PART"
echo "Label: $LABEL"
echo "Unpartitioned reserve at end: ${RESERVE_GIB} GiB"
echo "F2FS OP: ${OP_PERCENT}%"
echo "F2FS features: $FEATURES"
echo
printf 'Type YES to continue: '
read ans
if [ "$ans" != YES ]; then
    echo "Aborted."
    exit 1
fi

# Best Effort Unmount for Existing Partitions (please don't need this)
# ----------------------------------------------------------------------- #
if command -v findmnt >/dev/null 2>&1; then
    findmnt -rn -S "$DEVICE" -o TARGET 2>/dev/null | while IFS= read -r mp; do
        [ -n "$mp" ] && umount "$mp" || true
    done
fi

# And then...
# ----------------------------------------------------------------------- #
sgdisk --zap-all "$DEVICE"
sgdisk -a 2048 -n "1:2048:-${RESERVE_GIB}G" -t 1:8300 -c "1:${LABEL}" "$DEVICE"
partprobe "$DEVICE" || true
sleep 2

if [ ! -b "$PART" ]; then
    echo "Partition node did not appear: $PART" >&2
    lsblk "$DEVICE" >&2 || true
    exit 1
fi

# OhhhShitHandlers
# ----------------------------------------------------------------------- #
# NOTE: -t 1 performs initial discard when supported; if it's hanging here, rerun manually with -t 0
mkfs.f2fs -f -l "$LABEL" -o "$OP_PERCENT" -t 1 -O "$FEATURES" "$PART" && \
sync

# Display Block Devs
# ----------------------------------------------------------------------- #
lsblk -o NAME,SIZE,MODEL,SERIAL,TRAN,RM,RO,FSTYPE,LABEL,UUID,PARTUUID,MOUNTPOINTS "$DEVICE"

echo "[COMPLETE]"
# ----------------------------------------------------------------------- #
# EOF #
# ----------------------------------------------------------------------- #

In the worst of circumstances, Fedora notified me of a firmware update, which was a hands-off type of “reboot to install” EFI blob which bricked my Thinkpad X1 Nano laptop just last week. That’s an expensive failure and one which has no simple resolution (many hours, many iso to img to efi/cap conversions, and maybe soon a WSOP-8 NAND reprogramming).

I’ve been all-in on the RedHat infra train since 1999, having used it at global backbone providers, satellite infra providers, hpc clusters, media conglomerates, gaming networks, streaming services, cloud providers and hyperscalers, and just about everything in-between… and while I have my qualms on certain technical aspects and corporate strategy over those years, I still consider it one of my favorite Linux distos (top five at least), and it’s my first choice for enterprise Linux. So, I’m not just hating on the ecosystem here, I’m a vested user who’s been disappointed too many times by the direction

Unfortunately, with Fedora, I stand behind my position that Redhat has been too hands-off in conducting standards requirement alignment, in requiring a base level of stability upon which serious engineering development can be done, without having to clean up other people’s messes every few weeks.

It’s just not a great dev platform distro anymore, and yet the org markets the name as if it were a valid OS for all manner of production uses. IoT use? Never, no thanks. Edge devices? Never again. Security? Forget about it.

It’s not a “bleeding-edge issue” - as that’s Rawhide’s territory where stability of any sort is not expected. The evolving issue with Fedora is that it is uncommonly unstable for a “dev env”, and one which ships far too much untested code and pushes the equivalent of “crowd source fail-tests” where they expect the users to “just tell us if it doesn’t work”. Their release engineering and load-testing seems to occur with zero multivariate matrices sequences for all manner of performance, validation, and functional tests. I suppose they consider unit testing to be “full coverage”, but it is not. Happy to be corrected here, though I will remain disappointed and avoid any further deployments or firmware updates from Fedora.

Oracle bought Sun to get the ZFS file system. They have been porting ZFS features into Btrfs. The head of the Ext4 file system project, an Oracle employee, said that Ext4 was just a temporary stepping stone on the way to Btrfs.

A primary motivation for Oracle’s acquisition of Sun was to obtain and control the IP for the MySQL database project, which had been obliterating a large amount of Oracle DB customer licensing and dominating the OSS database landscape.

A primary motivation for Oracle’s acquisition of Sun was to obtain their hardware platform for running Oracle DB, Oracle J2EE applications, and their vast middleware stacks. This was everything related to the UltraSparc architecture, Sun Fire line of Opteron based servers, all of the high performance storage technology, and all of Sun’s enterprise contracts with the US military, US gov, various nation states, and all manner of other high-end “serious business” service compute providers.

Sun created Java, nearly all of Oracle’s entire stack requires Java, and Larry wanted that as a foundational piece of IP. At the time of the acquisition, the world ran on Java and it was the most profitable and most widely deployed programming language as well as server runtime environment (JDK, JRE). This aspect is also related to various IP lawsuits around Java, see Oracle vs Google later on.

ReiserFS features were ported into Btrfs, not ZFS features. ZFS has always dominated enterprise storage, never btrfs, never ReiserFS. ZFS in an entirely different class of quality, performance, scalability, reliability, and global market penetration.

RedHat was tired of supporting Btrfs after too many customers reported critical data-loss incidents, which could only serve to harm the company’s reputation for stability and technical excellence.

ZFS continues to be amazing, Btrfs continues to suck, Oracle still has their own original ZFS source tree separate from the OpenZFS project, Oracle still sells UltraSparc architecture, Oracle still owns Java IP, Oracle still doesn’t prioritize Btrfs, nothing important runs on Btrfs.

The only recent news about btrfs and Redhat, which invalidates the original claims about Redhat vs Oracle conspiracy, is that Fedora Linux 33 has Btrfs as a default option along with XFS and Ext4, and we all know that Fedora is upstream development for RedHat Enterprise Linux.

Some timeline details:

• 2007, Btrfs concepts and originated at IBM
• 2008, rip-off of ReiserFS features started by developer at Oracle, who was a former dev of ReiserFS (before Hans Reiser murdered wife, tried to cover it up, landed in prison, and ReiserFS project basically died too)
• Btrfs continues to be experimental for years, cannot complete with ZFS on any important metrics
• 2008, Oracle Exadata v1 - a specialized hardware platform for running Oracle databases on HP-UX with Itanium procs
• 2009, Oracle Exadata v2 - now running on Sun hardware, using modified Sun Fire systems with AMD Opteron procs
• 2010, Oracle acquired Sun which had recently acquired MySQL
• 2011, Redhat offers Btrfs as experimental in RHEL 6 beta
• 2012, Btrfs main developer leaves Oracle for Fusion-IO
• 2012, Oracle and SUSE offer Btrfs in their distro for prod use
• 2013, Btrfs main developer leaves Fusion-IO for Facebook
• 2017, Redhat removes Btrfs from inclusion to RHEL 8, switching to XFS (originally from SGI)
• 2018, Redhat acquisition by IBM

Some enjoyable references:

• en.wikipedia.org/wiki/Btrf…
• en.wikipedia.org/wiki/Acqu…
• en.wikipedia.org/wiki/Hans…
• en.wikipedia.org/wiki/ZFS
• en.wikipedia.org/wiki/XFS
• en.wikipedia.org/wiki/Orac…
• www.oracle.com/webfolder…
• flashdba.com/history-o…