Skip to content
Frostyard docs

Manage system extensions

A system extension is an EROFS image that adds files to /usr only. Every Frostyard bootc image carries a definition for each extension in the catalog, switched off. The extension itself is not in the image: Updex turns it on and downloads it. This guide covers the whole lifecycle of one extension: find it, enable it, check it, update it, roll it back, and remove it.

Before you start

Section titled “Before you start”

You need:

  • Root access to enable, update, and remove an extension. Checks that change nothing run without sudo.
  • Network access to https://repository.frostyard.org/ext/.
  • Free space in /var for the download and the installed image.

This guide follows Updex v2.1.3. Check your version:

Terminal window
updex --version

The output shows the version. An older Updex can lack a command or print different messages.

Find an extension

Section titled “Find an extension”

See what Updex found:

Terminal window
updex components
updex features list

The first table lists each component, the source directory that provides it, and its feature count. The second lists each feature, its description, its enabled state, its origin, and its transfers.

Limit a command to one component:

Terminal window
updex features list --component=docker

The table lists only the Docker feature.

The extension catalog lists every published extension. Your image ships its definitions in /usr/lib/sysupdate.<name>.d/, and it can enable only the features it has files for. If updex features list does not show an extension, your image cannot enable it.

Enable an extension

Section titled “Enable an extension”

Enable a feature for the next update:

Terminal window
sudo updex features enable <name>

Replace <name> with a feature name from updex features list. The output says Feature '<name>' enabled. and then Run 'updex features update' to download extensions. The command writes one drop-in, /etc/sysupdate.<name>.d/<name>.feature.d/00-updex.conf, which sets Enabled=true. It does not download the image.

Enable it and download it now:

Terminal window
sudo updex features enable <name> --now

The output says Feature '<name>' enabled. and lists the downloaded files. With --now, Updex writes the drop-in, downloads the image, checks its SHA256 hash against the signed manifest, creates the link, and refreshes the merge.

The refresh also creates the users, groups, and directories that the extension declares, before systemd reloads. Its units can then start at once, with no reboot.

Check that it works

Section titled “Check that it works”

Check the merge:

Terminal window
systemd-sysext status

The output lists the merged extensions. <name> is in the list.

The live link is /var/lib/extensions/<name>.raw. It points to the chosen image file in /var/lib/extensions.d/, named <name>_<version>_<os>_<arch>.raw. The transfer file sets CurrentSymlink=<name>.raw, which would put a second link in /var/lib/extensions.d/. Updex deletes that one on install.

If the extension has a service, check it with the unit name from its page:

Terminal window
systemctl status <name>.service

The unit is active. The extension ships a preset that enables the service and a multi-user.target.d/10-<name>.conf drop-in with Upholds=. Some extensions uphold a small setup unit that starts the service, for example k3s-sysext-setup.service, because systemd retries an upheld unit whose start condition fails. Only a Type=oneshot unit with RemainAfterExit=yes can carry a condition and be upheld. The drop-in is new to systemd after the post-merge daemon-reload, so it starts the unit when multi-user.target activates. A preset alone would not work: at boot, PID 1 scans unit files before the extension is merged.

At boot, reload-sysext.service runs systemd-sysusers, systemd-tmpfiles --create --graceful, a systemd-sysctl replay of the inotify, fs.aio-max-nr, and kernel.keys limits, and systemctl daemon-reload after the merge. A refresh by Updex (--now, or features update without --no-refresh) also runs systemd-sysusers and systemd-tmpfiles --create, but not the systemd-sysctl replay: an extension’s kernel limits apply at the next boot. With an Updex older than v2.1.3, a refresh creates no users or groups, and a unit that needs one fails until you reboot.

If a unit shows failed with start-limit-hit, systemd stopped starting it. Run sudo systemctl reset-failed <unit>, then sudo systemctl start <unit>, or reboot.

Frostyard transfers set Verify=true. Updex reads SHA256SUMS from the transfer’s source and checks the detached signature in SHA256SUMS.gpg against the systemd keyring. Only then does it trust the hashes in that file, and it checks the downloaded image against its hash. A missing signature or a bad one stops the command, and no image is installed.

The base image ships the Frostyard repository key at /usr/lib/systemd/import-pubring.gpg. Updex uses the first of these files that exists: /etc/systemd/import-pubring.gpg, then /usr/lib/systemd/import-pubring.gpg. If you create the /etc file, it must also hold the Frostyard key, or every Frostyard download fails verification.

If enable --now exits with a refresh error (refresh_error in JSON), the download worked, but the activation did not. If the output says activation did not finish, the extension is merged, but its users and groups can be missing: reboot. Otherwise Updex cannot tell what is merged: reboot. Updex v2.1.3 still says run 'systemd-sysext refresh' (or reboot) here. Do not refresh by hand: a sudo systemd-sysext refresh merges the extension, but it does not create the extension’s users and groups. After the reboot, check systemd-sysext status.

Update

Section titled “Update”

See what you can update to. This changes nothing:

Terminal window
updex features check

The table has a row for each enabled feature that has transfers, with the current version, the running and linked versions, the newest version, and UPDATE. UPDATE normally shows yes or no. It can also show:

  • error: Updex could not check that component. The command exits non-zero. Do not read an error row as no.
  • needs image >= <value>: a staged revision needs a newer base image. See An extension that needs a newer image.
  • held (rejected <version>) or held (by rollback of <feature>: <reason>): a rollback holds the extension. See Roll back.

The current version is the one that the link /var/lib/extensions/<name>.raw points to. If the newest image is already on disk but the link names an older one, UPDATE normally shows yes.

RUNNING is the version merged now. LINKED is the version that the next boot merges. features list shows both columns too. When they differ, the output ends with a PENDING REBOOT: line: the update is linked and waits for a reboot.

Install the newest versions:

Terminal window
sudo updex features update

Each row shows the status downloaded or up to date. Updex downloads and verifies each new image, switches the link, removes old versions, and refreshes the merge.

If the newest image is already on disk and only the link names an older one, Updex downloads nothing. It repoints the link and prints Linked staged version <version>. The row shows linked, and the refresh then activates the version. With --no-refresh, or when the refresh fails, the new version runs after the next reboot. If Updex prints refresh skipped: ..., it could not refresh safely, and the output says why: a revision needs a newer running image, or Updex cannot check one. The links stay, and the change takes effect at the next boot. Do not run systemd-sysext refresh by hand: it merges a revision that the running image may not support, and it does not create the extension’s users and groups. With --json, each feature result has refresh_skipped with this reason. See Features and components for the refresh rules.

Take the update at the next boot instead of now:

Terminal window
sudo updex features update --no-refresh

Updex downloads, verifies, and links the new version, but does not run systemd-sysext refresh. Nothing changes in the running system. The new version runs after the next reboot. --no-refresh is a global flag: the timer below uses it too.

Keep old versions:

Terminal window
sudo updex features update --no-vacuum

Without the option, Updex keeps the number of versions that the transfer sets in InstancesMax, which is 2 by default. It also keeps the linked image, the merged image, and a version that a rollback rejected.

Preview an update without a change:

Terminal window
sudo updex --dry-run features update

The output starts with [DRY RUN] and shows would download where it would change something.

A refresh swaps /usr under the running system, and a service can restart. Stop or drain workloads first if a restart would hurt them.

An extension that needs a newer image

Section titled “An extension that needs a newer image”

A new revision can need a newer base image. Updex then links nothing for that feature, keeps the old link, and keeps the downloaded image. The row shows not linked: needs image >= <value>. This is not an error, and the command exits 0.

Update the image. Install and update an image has the steps. The next Updex run links the staged revision. After a queued bootc rollback, Updex checks the revision against the rollback image. See Minimum image version for the rules.

Automatic updates

Section titled “Automatic updates”

Every Frostyard image enables updex-update.timer. A new install runs it from the first boot. A host that already runs Frostyard gets it from the second boot of the first image that carries the change.

The timer runs updex features update --no-refresh every 6 hours, with a random delay of up to 1 hour. A run missed while the host was off happens at the next boot. A run downloads, verifies, and links the new version. It never activates it: the new version runs after the next reboot, and nothing reboots the host for you. The service skips a run while the host is on battery. A host without a battery always runs.

No notice tells you that a run linked a new version. The desktop update notice (bootc-update-notify) covers image updates only, so no notice appears for an extension update. Check for one yourself.

Check that the timer is on:

Terminal window
systemctl is-enabled updex-update.timer

The output is enabled. To see the next run:

Terminal window
systemctl list-timers updex-update.timer

The table shows the next scheduled run of updex-update.timer.

Show the timer and the last run:

Terminal window
updex daemon status

The output looks like this:

Auto-update daemon: installed
  Enabled: true
  Active: true
  Source: packaged
  Unit file: <path>
  Schedule: <OnCalendar>
  Next run: <time>
  Last run: <time> (<result>)

There is one Unit file line for each unit. On a Frostyard image, Source is packaged and the unit files are under /usr/lib/systemd/system/. A masked timer shows Source: etc, Masked: yes, the mask /etc/systemd/system/updex-update.timer as its Unit file, Schedule: -, and Next run: none scheduled. <result> is the result of updex-update.service, for example success. With no run recorded since boot, the last line reads Last run: none recorded since boot. With no timer scheduled, Next run reads none scheduled. With no timer unit, the output is Auto-update daemon: not installed, then Run 'updex daemon enable' to enable automatic updates. The run log is in the journal:

Terminal window
journalctl -u updex-update.service

A run on an offline host, where no update source answers, logs updex: no update source is reachable; skipping this run and exits 0. The unit stays successful. A run fails when only some sources answer, when a source answers with an error, or when a download or check fails. The unit then shows failed until the next good run.

After a run links a new version, check it:

Terminal window
updex features check

LINKED shows a newer version than RUNNING, and the output ends with a line like this:

PENDING REBOOT: <n> sysext(s) linked to a version that is not running: <name>: <running> -> <linked>

Reboot to run the new version. After the reboot, RUNNING equals LINKED and the PENDING REBOOT line is gone. To look for a broken link, run:

Terminal window
find /var/lib/extensions -xtype l

The command prints nothing when every link works.

Turn the timer off for the whole host. --now also stops the timer that runs now, and the mask survives image updates:

Terminal window
sudo systemctl mask --now updex-update.timer

sudo updex daemon disable also stops the timer. The output says Auto-update daemon disabled. An image update can enable the timer again, so use the mask to keep it off. updex daemon disable does not remove a mask.

Turn the timer on again after a mask:

Terminal window
sudo systemctl unmask updex-update.timer
sudo updex daemon enable

With the packaged units, Updex writes no unit file. It enables and starts the timer. The output says Auto-update daemon enabled., Using the units shipped by the package; no unit file was written., Updates run every 6 hours while on AC power and download new versions., and Reboot required to activate downloaded extensions. To preview the change, run sudo updex --dry-run daemon enable. Nothing changes.

To stop updates of one extension, run sudo updex features disable <name>. See Automatic updates for the schedule, the lock, and the other details.

Roll back

Section titled “Roll back”

Go back to the previous version of an extension:

Terminal window
sudo updex features rollback <name>

Updex links the newest staged version that is older than the linked one. The rollback takes effect at the next boot. To apply it now with a refresh, add --now. To pick the version, add --to VERSION. If the older version is not on disk, the command fails and changes nothing.

Updex records the rejected version in /var/lib/updex/rollback/<name>. While the record exists, updex features update and the timer leave the extension alone. The row shows held by rollback: rejected <version> or held by rollback of <feature>: <reason>, and the command exits 0. When a newer version is published, the next update links it and deletes the record. To let updates link the newest version again now, delete the record:

Terminal window
sudo updex features rollback --clear <name>

The command changes no link. By default Updex keeps the two newest versions on disk (InstancesMax), plus the linked, the merged, and the rejected version. --no-vacuum keeps all of them. Your data in /var stays. See Rollback for the rules.

An image rollback with bootc rollback does not roll back extensions. The links and the images are in /var, which every image version shares, so the older image merges the same extension versions at its next boot. An extension is built against the image it was made for. A newer extension can fail on an older image. If one does, roll the extension back too, stop its updates, or remove it. While a bootc rollback is queued, Updex checks new revisions against the rollback image, so a revision that needs a newer image stays unlinked.

Stop updates or remove

Section titled “Stop updates or remove”

Stop future updates and leave the extension merged:

Terminal window
sudo updex features disable <name>

The output says Feature '<name>' disabled. Updex writes Enabled=false in the drop-in. It does not unmerge the extension. The image and the link stay. The next boot merges the version that the link names. LINKED in updex features list shows it. To go back to the version you run now, roll back first. The output also says Its extensions stay installed and merged. Run 'updex features disable <name> --now' to remove them.

Remove the extension now:

Terminal window
sudo updex features disable <name> --now --force

The output says Feature '<name>' disabled. and Extensions unmerged., lists the removed files, and ends with Warning: Reboot required for changes to take effect. Updex unmerges all extensions on the host, removes the image files and the link, and then merges the rest again. Reboot when the command finishes.

Updex refuses to remove a merged extension without --force:

Extension <name> (version <version>) is active. Removing requires --force and a reboot to take effect.

The version is the merged one. On systemd older than 258 it reads unknown. If Updex cannot read the merged state, it also refuses, and names the component with merge state unknown. <name> is the transfer name, the same as the feature name for Frostyard extensions.

An extension cannot provide files in /etc or /var. Your data in /var and your changes in /etc stay after a removal.

If the refresh at the end fails, the remaining extensions may stay unmerged. Run sudo systemd-sysext refresh, or reboot. If the output says activation did not finish, they are merged, but their users and groups can be missing: reboot.

How extensions are built

Section titled “How extensions are built”

An extension is an EROFS overlay (Format=sysext, Overlay=yes) on its build base: Base, or for desktop applications gui-base (Base plus the GUI libraries that every desktop image ships). It holds only what that build base does not have. A desktop-application extension needs those GUI libraries, so it is meant for Snow, Snowfield, and Sundog, not Floe.

  • The payload is /usr only. A package that installs to /opt is relocated to /usr/lib/<package> with /usr/bin symlinks, because /opt is a bind mount to /var/opt and a file there would be shadowed.
  • Default files for /etc stay in /usr/share/factory/etc, and tmpfiles.d copies them into /etc at boot, only when the file is missing. A build captures only the paths its rules name.
  • The file is <name>_<version>_<os>_<arch>.raw.zst. %w matches your OS version and %a your architecture. The transfer also accepts .raw.xz, .raw.gz, and .raw. Updex stores the image decompressed, as .raw. A file that an older Updex installed can keep a .raw.zst name. Its content is decompressed, and it works.
  • The version comes from the image’s key package (KEYPACKAGE). A Debian epoch 5:1.2.3 becomes 5+1.2.3 in the file name.
  • SYSEXT_REVISION=N appends +rN to the version, for a content fix at the same package version.
  • The image is a delta against the build base it was built on. A later Base that stops shipping a package the delta left out can break the merged extension.
  • Each build checks the paths in mkosi.images/<name>/required-paths.txt. A build that is missing one fails.
  • Each build also fails if the payload has files under /opt, a private key, or a file that Base owns. It also fails if an Upholds= drop-in names a unit with a start condition that is not a RemainAfterExit oneshot.

Troubleshooting

Section titled “Troubleshooting”
Symptom Check Fix
updex features list shows no features updex components If no component is listed, Updex found no definition files. Check that your image ships /usr/lib/sysupdate.<name>.d/.
features check shows UPDATE=yes but features update downloads nothing The output of features update says Linked staged version <version> The image was already on disk. Updex repointed the link and refreshed. If the output says refresh skipped, or you used --no-refresh, reboot to run the new version.
You want the old version back ls /var/lib/extensions.d/ Run sudo updex features rollback <name>, then reboot. See Roll back.
features rollback fails with version <version> of <name> is not on disk ls /var/lib/extensions.d/ Updex rolls back only to a staged image. Pick a version from the list with --to, or remove the extension.
features update shows held by rollback: rejected <version> or held by rollback of <feature>: <reason> ls /var/lib/updex/rollback/ A rollback holds the extension until a newer version is published. To update now, run sudo updex features rollback --clear <name>. If the message names another feature, its record holds the extension until that feature updates. To end the hold now, run sudo updex features rollback --clear <feature>.
features update shows not linked: needs image >= <value> bootc status The revision needs a newer base image. Update the image. The next run links it.
another updex operation is running Another updex command or the timer Wait, then run the command again. Only one operation that changes state runs at a time.
features check shows UPDATE=error The command output and the network Check the network and the published manifest. The command exits non-zero. Do not read the row as up to date.
enable --now exits non-zero after the download The output shows sysext refresh failed If the output says activation did not finish, reboot. Otherwise reboot too: a refresh by hand does not create the extension’s users and groups. Then check systemd-sysext status.
A service is not active after enable --now systemctl status <name>.service and systemd-sysext status If the unit shows start-limit-hit, run sudo systemctl reset-failed <name>.service, then start it. Otherwise reboot, so that reload-sysext.service runs the setup, then check again.
disable --now refuses with is active The refusal names the extension and its version Add --force, then reboot.
A signature check fails The command output Check the network and the published manifest. The publisher must provide SHA256SUMS.gpg. Never turn verification off.
Every Frostyard download fails verification ls /etc/systemd/import-pubring.gpg If the file exists, Updex uses it instead of the image’s key. Add the Frostyard key to it, or remove it.
updex-update.service shows failed journalctl -u updex-update.service A download or check failed, a source answered with an error, or only some sources answered. The unit stays failed until the next good run. A run where no source answers is skipped and does not fail.
The timer never runs updex daemon status If it shows Masked: yes, run sudo systemctl unmask updex-update.timer. If Enabled is false, run sudo updex daemon enable.

See also

Section titled “See also”

Sources

Section titled “Sources”

Reviewed 2026-10-30. snosi commit b2858ca0222814c395cf8248db98f3e096d57dc9; Updex v2.1.3 commit 2661e987faac986f2594d1d9bbbadf1bf3053f5b.