Arch Linux system maintenance and recovery

01 Package Updates

Always perform a full system upgrade. Partial upgrades (upgrading a subset of packages) are unsupported and routinely cause breakage on a rolling release.

# Full sync + upgrade
pacman -Syu

# Force refresh of all databases before upgrade
pacman -Syyu

Check for news before upgrading

Manual intervention is sometimes required before upgrading. Check https://archlinux.org/news/ or use informant / arch-news from the AUR to gate upgrades on unread news items.

Pacman flags reference

02 Cache Management

Pacman caches every downloaded package in /var/cache/pacman/pkg/. This directory grows unbounded and is never cleaned automatically.

# Remove all cached packages except the three most recent versions
paccache -r

# Keep only the most recent version
paccache -rk1

# Remove all cached packages not currently installed
paccache -ruk0

# Remove ALL cached packages (no rollback possible after this)
pacman -Scc

paccache is part of the pacman-contrib package. Enable its timer to automate weekly pruning:

systemctl enable --now paccache.timer

03 Orphan Packages

Orphans are packages installed as dependencies that are no longer required by any installed package. Accumulation is normal; clean them periodically.

# List orphans
pacman -Qdt

# Remove orphans and their unneeded dependencies (iterative)
pacman -Rns $(pacman -Qdtq)

# If the list is empty, pacman exits with an error — guard against it:
pacman -Qdtq | xargs -r sudo pacman -Rns

# Mark a package as explicitly installed (won't appear as orphan)
pacman -D --asexplicit python-requests

# Mark as dependency (will appear as orphan if nothing depends on it)
pacman -D --asdeps some-lib

04 Keyring & Signature Errors

Signature errors typically mean the local keyring is stale. This is common after a long period without updating or after reinstalling from old media.

# Update keyring package first, before anything else
pacman -S archlinux-keyring

# If pacman itself can't run due to key errors, initialize from scratch
pacman-key --init
pacman-key --populate archlinux

# Refresh keys from the keyserver
pacman-key --refresh-keys

Importing a specific key manually

pacman-key --recv-keys <KEY_ID>
pacman-key --lsign-key <KEY_ID>

05 Pacman Database Repair

The pacman local database lives at /var/lib/pacman/local/. Corruption is rare but possible after power loss during a transaction.

# Check local database consistency
pacman -Dk

# Check and fix — reinstall packages with broken DB entries
pacman -Dkk

# Rebuild the sync databases from scratch
rm -rf /var/lib/pacman/sync/*
pacman -Syy

Stale lock file

If pacman reports that the database is locked, it means another instance is running — or crashed, leaving a lock file behind.

# Verify no pacman process is running first
fuser /var/lib/pacman/db.lck

# If no process holds it, remove the lock
rm /var/lib/pacman/db.lck

Recovering from interrupted upgrades

# Find half-extracted packages (common after power loss)
find /var/lib/pacman -name "*.part" -delete

# Re-run upgrade; pacman will complete interrupted transactions
pacman -Syu

06 Partial Upgrade Recovery

A system in a partial upgrade state will exhibit broken shared libraries, missing symbols, or segfaults. The fix is always to complete the upgrade.

# Complete the upgrade — this is the first action in almost every scenario
pacman -Syu

# If a package conflict blocks the upgrade, remove the conflicting package first
pacman -Rdd conflicting-package
pacman -Syu

Diagnosing broken shared libraries

# Find binaries linked to missing libraries
ldd /usr/bin/some-broken-binary 2>&1 | grep "not found"

# Find all installed ELFs with missing deps (requires pax-utils)
findbrokenpkgs   # from AUR: findutils-broken

# Identify which package owns a missing .so
pkgfile libmissing.so.2   # requires pkgfile; run pkgfile -u first

Downgrading a package

# From cache (preferred)
pacman -U /var/cache/pacman/pkg/linux-6.8.1.arch1-1-x86_64.pkg.tar.zst

# From AUR helper (downgrade utility)
downgrade linux          # from AUR: downgrade

# Prevent a package from being upgraded
# Add to /etc/pacman.conf:
IgnorePkg = linux linux-headers

07 Systemd Service Failures

# Show all failed units
systemctl --failed

# Get full logs for a specific unit
journalctl -xeu servicename.service

# Show unit file, drop-in overrides, and current status
systemctl cat servicename.service
systemctl status servicename.service

# Reload daemon config after editing unit files
systemctl daemon-reload

Dependency and ordering issues

# Visualise unit startup order (requires systemd-analyze)
systemd-analyze critical-chain

# Identify units that slow boot
systemd-analyze blame

# Dump full dependency graph for a unit
systemctl list-dependencies servicename.service

Emergency / rescue targets

# Drop to rescue shell (single-user, mounts filesystems)
systemctl isolate rescue.target

# Drop to emergency shell (only root fs mounted read-only)
systemctl isolate emergency.target

# Kernel parameter equivalents (add to GRUB at boot):
systemd.unit=rescue.target
systemd.unit=emergency.target

08 Journal Management

The systemd journal can grow large. By default it is capped at 10% of filesystem size; configure it explicitly.

# Check current journal disk usage
journalctl --disk-usage

# Vacuum to a size limit
journalctl --vacuum-size=500M

# Vacuum entries older than a time period
journalctl --vacuum-time=2weeks

Persist the limit in /etc/systemd/journald.conf:

[Journal]
SystemMaxUse=500M
SystemKeepFree=200M
MaxRetentionSec=1month

# Apply config changes
systemctl restart systemd-journald

Useful journal filters

# Boot messages from current boot
journalctl -b

# Previous boot
journalctl -b -1

# Kernel messages only
journalctl -k

# Priority: err and above
journalctl -p err

# Follow live output
journalctl -f

09 Filesystem Checks

ext4

# Check and repair (answers yes automatically)
fsck.ext4 -fy /dev/sdXN

# Force check even if filesystem appears clean
fsck.ext4 -fyc /dev/sdXN

# Read-only check to assess damage before committing
fsck.ext4 -n /dev/sdXN

Btrfs

# Check (read-only by default)
btrfs check /dev/sdXN

# Scrub (online, verifies checksums against parity)
btrfs scrub start /mountpoint
btrfs scrub status /mountpoint

# Balance (redistributes data across devices, or after device add/remove)
btrfs balance start -dusage=50 /mountpoint

SMART monitoring

# Run a short self-test
smartctl -t short /dev/sdX

# View results
smartctl -a /dev/sdX | grep -A5 "SMART overall"

# Enable smartd daemon for ongoing monitoring
systemctl enable --now smartd

10 Chroot Recovery

When the system won't boot, boot from Arch live media and chroot into the installed system to perform repairs.

# 1. Identify your partitions
lsblk -f

# 2. Mount root partition
mount /dev/sdXN /mnt

# 3. Mount ESP if separate
mount /dev/sdXEFI /mnt/boot/efi

# 4. Use arch-chroot (handles bind-mounting /proc, /sys, /dev, /run)
arch-chroot /mnt

# Now inside chroot — run repairs:
pacman -Syu
mkinitcpio -P
grub-mkconfig -o /boot/grub/grub.cfg

Manual chroot (if arch-chroot is unavailable)

mount -t proc none /mnt/proc
mount -t sysfs none /mnt/sys
mount --bind /dev /mnt/dev
mount --bind /run /mnt/run
chroot /mnt /bin/bash

# Source profile for correct environment
source /etc/profile
export PS1="(chroot) $PS1"

Reinstalling all packages from chroot

If a system-wide corruption is suspected, reinstall every package:

pacman -Qnq | pacman -S -       # reinstall all native packages

11 Bootloader Recovery

GRUB

# From chroot after mounting EFI partition at /boot/efi:

# Reinstall GRUB to EFI (UEFI)
grub-install --target=x86_64-efi --efi-directory=/boot/efi --bootloader-id=GRUB

# Reinstall GRUB to MBR (BIOS/legacy)
grub-install --target=i386-pc /dev/sdX

# Regenerate config
grub-mkconfig -o /boot/grub/grub.cfg

systemd-boot

# Reinstall the EFI binary
bootctl install

# Update the installed binary to match current systemd version
bootctl update

# Inspect current state
bootctl status

Regenerating the initramfs

Required after kernel upgrades, driver changes, or if the system hangs during early userspace boot.

# Regenerate for all installed kernels
mkinitcpio -P

# Regenerate for a specific preset
mkinitcpio -p linux

# Verbose output for debugging hook failures
mkinitcpio -Pv 2>&1 | less

12 Snapshots & Rollbacks

Btrfs + Snapper

# Install and configure for root subvolume
snapper -c root create-config /

# Create a manual pre/post pair (e.g., around upgrades)
snapper -c root create --type pre --description "before upgrade"
pacman -Syu
snapper -c root create --type post --description "after upgrade"

# List snapshots
snapper -c root list

# Diff between two snapshots
snapper -c root diff 1..2

# Rollback to snapshot N (mounts it as new root on next boot)
snapper -c root rollback N

Snapper + pacman hook (snap-pac)

The snap-pac package installs pacman hooks that automatically create pre/post snapshots around every pacman transaction — no manual step needed.

pacman -S snap-pac

LVM thin snapshots (non-Btrfs)

# Create a snapshot of an LVM logical volume
lvcreate -s -n snap_root -L 10G /dev/vg0/root

# Mount snapshot read-only for inspection
mount -o ro /dev/vg0/snap_root /mnt/snap

# Rollback: merge snapshot back into origin
lvconvert --merge /dev/vg0/snap_root
# Reboot into origin LV for merge to complete