The System That Wouldn't Let Me In

Day 1: The Emerald City Problem

March 4, boot into Gentoo. Check the emerge logs.

1772607677:  *** Finished. Cleaning up...
1772607677:  *** exiting successfully.
1772607679:  *** terminating.

Wait. It... finished? After a hard lockup? After days of sitting frozen?

1,319 of the 1,556 packages installed. The emerge survived the crash, kept running somehow, and completed.

I was ready for a disaster. Instead, almost everything was fine.

(The "almost" is the problem.)

Day 2: Password Rejected

March 6. Try to log in.

SDDM prompts for password. Type the password. Rejected.

TTY login. Same password. Rejected.

But I know the password is right. I literally just typed it 30 seconds ago on the OpenSUSE partition. It works there. The password is correct.

So what's happening?

Time to chroot into Gentoo from OpenSUSE and see what's broken.

sudo mount /dev/nvme0n1p7 /mnt/gentoo
sudo chroot /mnt/gentoo /bin/bash

Check /etc/shadow:

argo:$6$random_hash_stuff:...

It's there. SHA512-CRYPT hash (the $6$ prefix). Standard, modern, should work.

Try to verify the hash manually... and it fails.

The Real Problem

libxcrypt — the library that handles password hashing and verification — got recompiled during the @world update. But the new version lost SHA512-CRYPT support.

It doesn't error. It doesn't warn. It doesn't say "I don't know how to verify this hash." It just returns "wrong password."

For every attempt. Every single time.

Think about how insidious that is: the library is supposed to verify a password. The password hash is valid. But the library was recompiled without support for that hashing algorithm. So it literally cannot check the hash. From its perspective, all passwords are wrong.

1,319 packages installed. One of them broke authentication. Not through corruption or a bug — through losing a feature that was needed.

The Diagnosis Rabbit Hole

I knew the password was right. I knew the hash format was valid. But the system kept rejecting me.

What if the crash corrupted something else? What if it's not libxcrypt, it's PAM config? What if it's the shadow file itself — maybe the emerge crash left it in an inconsistent state?

I started checking everything:

/etc/pam.d/system-auth — looked fine
/etc/shadow permissions — correct
Password hash format — valid SHA512-CRYPT
Dynamic linker — run ldconfig inside the chroot, maybe the LD cache is stale?

Each check worked. Each piece looked fine. But the sum of all the fine pieces still rejected my password.

That's when I realized: one of the libraries that other things depend on got silently broken. The old libxcrypt could verify SHA512. The new one can't. And the system just... quietly fails.

Initial Fixes

I can't log in, but I can fix things via chroot from OpenSUSE.

Disabled the display manager. Removed display-manager from Gentoo's default runlevel so it boots to TTY instead of SDDM. Logs in passwordless with PAM's nullok rule.
Enabled passwordless login. Modified /etc/shadow and PAM config so I can at least use the TTY.
Rebuilt the LD cache. Ran ldconfig inside the chroot. Maybe the dynamic linker was confused.
Backed up everything. Shadow file, PAM config, emerge log. I want rollback options if something gets worse.

The system now boots to a passwordless TTY login. Not great. Not production-ready. But I can actually use the system again.

The Deeper Problem

Here's what bothered me about this: I had 1,319 packages installed successfully. Everything should just work. But one package — libxcrypt — broke something fundamental: authentication.

And it only broke one specific case: systems using SHA512-CRYPT hashes. If you had MD5 passwords (deprecated but still exists) or bcrypt, you'd be fine. If you had just reset your password with a new algorithm, you'd be fine.

But I had old SHA512 hashes. And the new libxcrypt couldn't handle them.

This is the nightmare scenario in package management: A package compiles. It installs. It doesn't error. But the feature you depend on silently disappeared.

What I Learned (Immediately)

TTY mode for @world updates. Obviously. I know this. I should've done this.
Password hash validation script. Before deploying recovery changes, validate that the password system actually works. Don't assume.
Keep a dual-boot. OpenSUSE saved me. Without it, I'd be booting from a live USB every time I needed to fix things via chroot.
libxcrypt is finicky. Some version of it broke SHA512 support. This isn't a Gentoo bug — it's a "the library was recompiled and lost a feature" situation. Could happen to any system.

The March 9-10 Story

I'm writing this on March 3, but the real fix doesn't happen until March 9-10.

That's when I have time to write a comprehensive recovery script. The system boots on March 10, the recovery script runs, and then — finally — the system is back to normal.

But that's another story.

Today's Unresolved Question

I can log into a TTY now (passwordless). I can't log into the GUI. I can't use SSH. I can't do much of anything except sit at a prompt and type commands.

The system is broken in a very specific way: it can boot, but it can't authenticate. It has a working kernel, working filesystems, but the authentication layer is offline.

Tomorrow I'm going to write a script to fix this properly. Not a quick hack. A real recovery procedure that rebuilds libxcrypt with SHA512 support and resets the password so the system trusts it again.

Fourteen days from the crash to working login.

And it starts tomorrow.