System Collapse: A Chaotic Experience with Linux and NVIDIA

During intensive testing with local language models (LLMs) running on an NVIDIA graphics card, the CachyOS (Linux) operating system completely collapsed after an abrupt reboot. The integration between gaming-oriented distributions, video drivers, and prolonged artificial intelligence workloads can cause severe freezing states and temporary storage caching corruption.

Given the inability to immediately recover the stability of the graphical environment, it was necessary to resort to deep analysis via the console, USB rescue drives, and advanced AI-assisted technical support tools to rebuild the boot system.

1. Initial Diagnostics and Boot Time Analysis

To identify which services or processes were slowing down the operating system's startup, the first diagnostic step consisted of analyzing component boot performance through the terminal.

• systemd-analyze blame: This command lists all system services (systemd) in an orderly fashion according to the exact time they took to initialize during boot. It immediately detects whether a specific daemon or driver is creating a bottleneck or blocking the startup process.
• systemctl status sddm and sudo systemctl restart sddm: These allow you to verify the status and restart the graphical display manager (in this case, SDDM). However, when the graphical interface fails completely due to video drivers, this alternative is usually insufficient if the core problem lies within the graphics hardware.
• systemctl --failed is used on Linux systems to list all systemd units that have failed
  • journalctl --directory=/mnt/var/log/journal -p err -b -1

2. System Rescue via Live USB and Chroot

Since the graphical environment failed to load and conventional access was unavailable, an external installation medium (Live USB) was used to virtually mount internal disk partitions and repair the root file system (formatted in Btrfs) from an independent command console.

• lsblk: Lists all storage devices and partitions available on the machine (such as NVMe drives), facilitating the exact identification of where the operating system is installed.
• sudo mount -o subvol=@ /dev/nvme1n1p2 /mnt: Mounts the main root Btrfs subvolume of the damaged system into the temporary directory /mnt of the rescue USB to manipulate its internal files.
• sudo mount /dev/nvme1n1p1 /mnt/boot/efi: Mounts the system's EFI boot partition at the corresponding path inside the mounted environment.
• sudo arch-chroot /mnt: Changes the root of the current command system to the mounted hard drive directory (chroot). It virtually "enters" the damaged system to execute repair commands as if the PC had booted normally.

3. Rebuilding the Boot Environment and GRUB Manager

Once inside the rescue environment, corruption issues in the startup images and the bootloader were corrected using native reconstruction tools:

• mkinitcpio -P: Rebuilds and packages initramfs images from scratch (the temporary root file system loaded into RAM when turning on the computer). This allows the system kernel to repackage the disk and file system drivers needed to read NVMe drives without freezing at boot.
  • This packages all disk drivers, file systems, and system services so that when you turn on the PC, the kernel can read your NVMe drives and mount your Btrfs partition without hanging.
  • Specifically, the hooks you see on screen perform essential tasks:
    ⁠autodetect⁠: Detects your motherboard's specific hardware and current components to include only necessary drivers and lighten the boot.
    ⁠microcode⁠: Loads manufacturer microcode updates for your Intel or AMD processor, fixing potential hardware-level stability bugs early.
    Once it finishes passing through all remaining hooks (like ⁠modconf⁠, ⁠block⁠, ⁠filesystems⁠, etc.), the process will be complete and return the command cursor.
  • It functions as a temporary and necessary RAM disk that the kernel uses right at startup to load basic hardware drivers, such as NVMe support and your Btrfs partition.
  • When that file gets corrupted or out of sync, the system freezes at the "Loading initramfs" message because the kernel becomes blind and doesn't know how to continue reading the hard drive. By running ⁠mkinitcpio -P⁠, we are telling it to repackage that boot bridge using real and clean component information.
• grub-install and grub-mkconfig:
  • grub-install --target=x86_64-efi --efi-directory=/boot/efi --bootloader-id=cachyos --recheck: Reinstalls GRUB bootloader files in the EFI partition so the motherboard correctly recognizes the operating system's startup path.
  • grub-mkconfig -o /boot/grub/grub.cfg: Scans the disk for installed kernels and generates an updated configuration file for GRUB, ensuring the boot menu recognizes all available systems (including Windows).

4. Editing Boot Parameters and Rescue Mode (TTY)

To prevent the system kernel from trying to load faulty NVIDIA graphical acceleration during startup, you can interrupt the GRUB menu by pressing the e key and adding debugging parameters to the end of the line starting with linux /boot/vmlinuz- :

• plymouth.enable=0: Disables the visual loading screen and system logo animation. By avoiding early use of graphical acceleration, you prevent freezing caused by video driver failures.
• nomodeset: Instructs the Linux kernel not to load dedicated graphics drivers (NVIDIA) during the initial boot stage, using a basic compatible video driver instead.
• 3: Forces the system to boot directly into pure text mode (multi-user console without a graphical desktop environment), facilitating advanced error diagnostics.

What you should do:

Turn on the PC and, when the GRUB menu appears (where you choose between CachyOS and Windows), select the CachyOS option but look for the one that says Fallback (for example, CachyOS Linux (fallback)) or press the ⁠E⁠ key on the normal line to edit boot parameters and add the word ⁠single⁠ or ⁠3⁠ to the end of the line starting with ⁠linux⁠, then press ⁠Ctrl + X⁠ to boot.

You can also try the other GRUB options:

• "Fallback" in GRUB:
  •  Go to Advanced options for CachyOS.
  • Select the option ending in (fallback initramfs).
  • Press ⁠Enter⁠.
• Force text/safe mode by editing GRUB in real time
  If the previous option also freezes on the black screen, do what was shown before.

Plymouth

5. Package Management and Definitive Solution

Once inside the command terminal without a graphical interface, package updating and repair relied on the system's official manager:

• sudo pacman -Syu: Synchronizes official repositories and updates all system packages to their most recent version.
  • • pacman: Invokes the native Arch Linux package manager.
    • -S (Sync): Indicates working with remote repositories to synchronize or install software.
    • -y (Refresh): Downloads and installs the most recent copy of the package database from the server.
    • -u (Sysupgrade): Compares local system package versions with the newly updated database and proceeds to update everything that has become obsolete.
• sudo pacman -S $(pacman -Ssq nvidia): Automatically searches and reinstalls all packages and dependencies related to graphics card drivers NVIDIA: NEVER EXECUTE THIS, as it would install packages you DO NOT need—all of them from the official repo.

OpenCode Analysis and AI-Assisted Diagnostics

During the operating system recovery process following the graphics driver failure, AI assistance tools (such as assistants based on code or OpenCode) were used with standard user privileges (without superuser permissions via sudo), ensuring a controlled analysis environment.

Perform a complete system diagnostic to identify the root cause of the startup performance issue. The system boots extremely slowly from GRUB to loading the login page, processes like Plymouth crawl, graphical interface elements load very slowly, and I can no longer load the graphical interface at all. The problem occurred when, from another session, I had been testing local LLMs based on a FastAPI project for hours, rebooted from the interface, and upon restarting it booted up very sluggishly. Check the system logs to see if anything turns up, as I suspect a conflict with nvidia-smi, mkinitcpio.conf configurations, or partition files to ensure everything is fine or whatever you consider to pinpoint the issue. I already ruled out a graphics issue by booting into Windows and running a game, and it runs perfectly. I suspect a conflict between Linux and NVIDIA drivers, a process left in a loop or corrupted, or some cache causing this behavior.

1. Presenting the Problem to the AI Assistant

To obtain an accurate diagnosis, a detailed report addressed to the AI agent was structured with the characteristics of the failure:

• Main symptoms: The system experienced extreme sluggishness from the GRUB screen to loading the login manager and graphical interface.
• Previous context: The failure occurred after an abrupt reboot prompted by prolonged sessions running local language models (LLMs) on an NVIDIA graphics card.
• Ruled-out actions: The physical proper functioning of the graphics hardware was verified by checking that the equipment booted and ran games smoothly under Windows.

2. Analysis and Error Correction Process

Initially, probabilistic model-based assistants tend to suggest hypotheses based on common configurations. In this case, the agent mistakenly interpreted a boot parameter manually entered in the GRUB menu (such as safe text mode) as a system anomaly.

Once the context was reoriented and system logs were analyzed, it was determined that the root cause lay in temporary corruption of proprietary NVIDIA driver packages caused by forced interruption of GPU processes.

3. Key Commands Used in Diagnostics and Repair

During the technical audit and the final solution of the problem, specialized system tools were used to examine resources and update affected components:

• systemd-analyze blame: Lists all active services ordered from longest to shortest startup duration, identifying boot bottlenecks.
• systemctl --failed: Displays only system service units that failed to start, useful for detecting looping blocked processes.
• nvidia-smi: Monitoring tool detailing the current state of the NVIDIA GPU, its temperature, and loaded drivers.
• journalctl -b -p warning: Filters the current boot log (-b) to display exclusively critical warnings and errors (-p warning).
  • journalctl -p err -b
  • journalctl -b -1 -p err
  •  
• sudo pacman -Syu: Synchronizes local repositories and fully updates all operating system packages.
  • Parameter -S (Sync): Manages package installation and updates from repositories.
  • Parameter y (Refresh): Downloads a recent copy of remote repository databases.
  • Parameter u (Sysupgrade): Updates all system packages with newer available versions.
• sudo pacman -S --overwrite '*' nvidia-utils opencl-nvidia nvidia-settings libva-nvidia-driver: Forces the reinstallation and overwriting of essential NVIDIA graphics components to fix damaged libraries.
  • Parameter --overwrite '*': Tells the package manager to overwrite any existing file on the system conflicting with new packages to install, ensuring a clean recovery.

Conclusion

Experience shows that while Artificial Intelligence tools (such as code-based assistants or OpenCode) offer valuable diagnostic routes to examine error logs and propose solutions, the user must maintain active technical judgment. Understanding the exact function of advanced commands like chroot, mkinitcpio, and manipulating parameters in GRUB makes the difference between successfully rescuing a complex operational environment or unnecessarily resorting to a full format.

Combining structured diagnostics via text terminals with AI-based code tool assistance made it possible to resolve a critical driver issue without formatting the equipment. Understanding system package behavior and advanced overwrite parameters ensures efficient responsiveness to complex failures in Linux environments.

This was the conclusion drawn by the AI, with which I agree:

That is an excellent question. In theory, restoring a Btrfs/Snapper checkpoint should return the system to the exact instant prior to the failure. However, in the case of NVIDIA GPUs and Linux booting, restoring a snapshot sometimes does not resolve the sluggishness or freezing due to three specific technical reasons:
1. GPU physical state/persistence: A restore point only changes files on the hard drive. If the GPU remained in a hardware fault state / VRAM locked by the LLM process, the video card will maintain that error at an electrical/firmware level until there is a power cycle or full PCI bus reset, regardless of what files the disk has.
2. The ⁠/boot⁠ or EFI partition is not always restored: Btrfs snapshots by default only save the root partition (⁠/⁠). Kernel startup files, boot environment images (⁠initramfs⁠), or the GRUB configuration in the EFI partition are usually left out of the snapshot. If there was a mismatch there, the root snapshot does not fix it.
3. Cached compiled modules: NVIDIA drivers compile specific modules against the kernel. If the snapshot restores the root but not the DKMS cache or ramdisk images, the startup kernel keeps trying to load an out-of-sync module.
Therefore, a clean reinstallation is the fastest and most definitive route at this point: format the root partition, write a brand new EFI/boot image, apply clean hardware detection for NVIDIA, and reconfigure GRUB recognizing Windows from scratch.