Is the name of the vulnerability literally *sshole?
* ssh olé!
Yes, it’s actually a very creative name. It’s caused by the character * allowing anyone to log into an open SSH port. So it’s an amalgam of {*}{SSH}{port (hole)}.
The long story short is that if you don’t have a password set for an account, Alpine saves the user’s password hash as “*” because no password hash will ever match it. Should be safe, right? Any password will always automatically fail, because the hash algorithm cannot return “*” as a result. So the only way to log in as that user is via something besides a password.
And when musl (the underlying system that Alpine uses for SSH) gets a Null response from its password hash, it uses “*” to represent that null value. Because lots of programs throw a fit if they get a true null value. So instead of returning a null value, it just returns a *. Which shouldn’t match anything, right?
So an attacker forces the password hash to return a null value. Simple to do, because the password hash is capped at 256 characters. So just enter something with more than 256 characters. Musl takes that null value, turns it into a *, and passes that onto Alpine. Alpine takes this *, and compares it with the list of saved usernames and password hashes. And it finds that the user (who has no password set, and shouldn’t be allowed to log in with a password) has a password hash of “*”. This matches the result musl just provided, so the attacker is able to log into the system.
Ahaha! Thanks for explaining <3
Cyberpunk as fuck.
tldr: disable password auth in ssh.
Also name misses the point I think, it’s a musl-hole.
i mean assholes are muscle holes.
Yeah, but they are very specific kind of muscle hole. 🙃
Yeah, any application checking passwords would be vulnerable.
The only time I ever exposed my SSH port to the WAN, I got about 750 connection attempts from China and 250 attemps from Russia per hour.
One reason that I have my router ignore all connection attempts from Russia and China
Fail2ban with increasing jail time for repeat offenders usually takes care of the worst of it after a day or two
How do you know this? What tool or utility tells you this fact? I have a port exposed for a game server but realising I don’t know how many attempts are being made.
I was kinda new to linux so I used https://www.papertrail.com/ and hooked it up to my syslogs, and then went “oh… oh no… oh god no…” and immediately turned off SSH.
For ssh it’ll log out to the auth log at /var/log/auth.log
Among other things, change the default ssh port to something way higher.
Did it help you find hot milfs within a mile of your location?
Surprised me too, it’s even worse if you use a cloud host for a VPS since it’ll be a more known IP
Yet some people will still say with a straight face “tHeRe iS nO hYbRiD wAr!1!1”
Not in Ba Sing Se at least…
This was about 20 years ago btw. I know it’s gotten worse. But it started earlier than most think.
Yeah, BBC has a whole documentary about it. Plus the russian state department basically published it all in a propaganda manual. They don’t exactly try to hide this stuff…
yeah, while reading this i was going “What maniac exposes SSH to WAN anyway…?” the whole time. I fail to see any reason or point that can be justified exposing SSH to WAN.
I used to have a synology NAS that would sleep of death itself. Synology support asked me to port forward SSH to the open internet and set no password so they could pull logs over the next few weeks.
Yes, it was legit synology support not a scammer.
No, I haven’t bought synology ever again.
What do you use now? Qnap?
I build my own stuff now. I tried qnap after synology but it just sucked in different ways.
Not the person you responded to, but I’ll give my unsolicited advice: I switched to Unifi NAS. QNAP has had a lot of security nightmares, and a bad habit of EOLing devices.
The best option might actually be using an old workstation and going with something like TrueNAS, but I enjoy the convenience of a purpose built box.
One big difference between Unifi NAS and something like QNAP or Synology, is that the Unifi NAS only does NAS work, it doesn’t run containers or “apps”. You need separate compute if you want to run containers.
One thing I didn’t know before buying a UniFi Nas was it does not support raid0 or jbod unfortunately
The sounds like a big one. Doesn’t affect me, but you’d think a lot of people would want one of those.
Unifi actually sounds better. I’m the same and want my NAS to be optimized by someone smarter than me. I’ve been using qnap and don’t really care for half the crap they offer. They’re not as enshittified as Synology, but they’re getting there.
Any VPS provider just gives you an SSH IP and port. So that’s the default state, until you do something else, which I imagine many don’t, since key based authentication was always considered secure.
So I don’t know the reason, but that’s the standard practice with most providers, like Digital Ocean or Linode or whatever. You can call them maniacs but you shouldn’t be surprised at something so ubiquitous.
Speaking of which, what VPS provider would you recommend for private use? I just need something small and cheap for always on headscale, matrix, keyguard-app and the likes, but I have no idea where to start looking.
Literally anyone, I’m paying ~£40 a year for a fasthosts VPS, was kind of a what can I get in the next 5mins thing, rather than a deep dive in to features. I did have a VPS at one point that had some kind of Microsoft hypervisor that made me vomit in my mouth bit when I found out. Just watch out as the bottom end of the market is full of enshitification that you don’t notice at first until you want to change the rdns or some other basic thing
Ah ok thanks, I was hoping there was like a de facto King of the scene but it’s actually much better if it’s a toss-up.
Also, thanks for the warning about the w-word, and my condolences
99.999% of that is over IPv4. I leave IPv6 open to WAN for my own use and it’s not a problem.
Thanks for the intel
New friends from all over the world!
Yeah it’s crazy. Are they just constantly pinging every possible ip address on every possible port every second? I opened a port for 1 day and the traffic made me uncomfortable enough to rearchitecture my entire self host setup
Wait that means I have to expose a port. But I have zero exposed ports.
This post was created by the pangolin-gang.
I’ll expose my port to you.
Privileged port?
I downvoted because I laughed 🤣
Meaning router port, not firewall port? I struggle to know the difference.
Some people seem to think that by not opening a port in their router, and instead opening a reverse tunnel to a VPS which has an open port, it somehow makes them more secure
I’m not sure what a reverse tunnel or vps is so I’m good. Why wouldnt they just use tail scale.?
behold!
the average selfhost admin!
practically need to rotate logs every hour if you foolishly leave ssh open on a open network. usually a bastion with ssh listening to port 34000 or such is a good first step.
that said the conditions for this exploit are fascinating. no pam, shadow password set to *, using a alternative to glibc. The sacrifices to be more efficient feel on par with replacing a tank’s armor with paper mache and hope.
The best armor is to not get hit!
Can they please take a break from having vulnerabilities
nah, find and fix as many as possible.
It seems like the pattern emerging is that AI is great at finding these obscure vulnerabilities that are only exploitable on very particular systems with very particular, non-typical configurations, and only on nights where unicorns fart candycorn.
Sure, fix them, but the hype is too much. Also, disable password auth on SSH already. Jesus. Keys arnt that hard.
I think you’re buying snake oil, my friend. What is “AI”? Statistical tools have been used in security auditing for decades.
The people riding the AI bubble want you to think things are magically different right now, but we won’t know that until we can get some solid data over the next year or five to do a proper comparison.
More like AI is much faster at finding obscure vulnerabilities, because it doesn’t get tired of digging into the mind-numbing details of human-generated spaghetti code. Give a human the same kind of task, and they would die of boredom 12 hours into it.
We’re entering an age where AI is quickly cleaning up the stupid security holes that have been created by decades of human mistakes, and will continue to be this way until increased AI security reviews will become more standard. The tribals wielding spears will continue to get shot by superior firepower until they adapt to the same weapons and protection themselves.
“Cleaning up” seems a little generous. It’s finding them, but someone still has to patch them.
“Finding” is also rather generous, going by numbers from Anthropic and from the Linux kernel, things like Mythos are wrong 95% of the time.
They’re much easier to run though, compared to more reliable tools. So even being wrong 95% of the time wins out if you’ve got 100x more people running them.
Most vulnerability patches are a few lines of code. Finding it is the hard part.
Even then, the AI session that found it probably already has a solution to fix it.
In my experience, the LLM usually proposes a solution, but it’s a crapshoot whether it works well. And plenty of bugfixes go well beyond a few lines, especially in older code.
I use this shit on a daily basis at work. They work well, as long as you’re choosing the right models, making the right engineering decisions, and have a good set of baseline rules.
Yeah and this isn’t a new paradigm, in the sense that tools to find exploits have gotten better repeatedly and allowed those with enough knowledge to use the tools to find them, regardless of if they could manually craft the exploit themselves. There’s a multitude of bug bounties that have been claimed by those who knew how to let a reverse engineering tool like ghidra hammer away at something with variation for long enough. And the models are capable of novel exploits on their own, but generally they are just doing the same, hammering ghidra with an MCP at accelerated scale. So “only ai would have found this bug” is technically untrue and moot.
This example demonstrates my point though. It requires an extremely unusual system configuration that may or may not even exist in the real world it is so unusual. AI doesnt get tired, like you said, but its not about spaghetti code (both musl and openssh are gold-standard opposite of spaghetti code), its about trying every possible software combination in ways a human wouldnt bother.
It requires an extremely unusual system configuration that may or may not even exist in the real world it is so unusual.
You have to remember that the same AI agents finding the code are also able to exploit the code. So, if it’s some extremely rare race condition requiring some convoluted pattern, it’s still easy to exploit.
When I stack dominoes in a line on the floor and I push one of them on the outside as it is in the shape of a snake… they fall down in a prdictable pattern. I never knew that would happen until I was shown the first time. This knowlege was past down via print made by humans scraped by bots. For what and for who? Louder faster and dumber… on a foundation of jUsT gEt iT d0Ne… now you are fired. Loops, poops and dupes.
I agree about the AI, but desktop Linux has never been under haxxor pressure the same way windows or apple has. I also think the newer parts of the userbase are less technical than even ten years ago. Previously undocumented bugs found and squashed is great for everyone.
Have any of these new vulnerabilities been desktop linux?
How would i know? The write up only touched a handful of things.
This particular vulnerability is precisely about logins with key. Using password login renders this vulnerability useless.
Using password login renders this vulnerability useless.
This isn’t the case, though I can see how the article could lead you to believe that.
It’s directly based on password rather than key input. The entire vulnerability is that typing a password longer than 256 characters automatically defaults to
crypt()outputting the*character instead of a normal password hash, which is coincidentally the same default character used as the “hash” (not actually a value you could get from a normal hash, intended to be impossible to get from any normal password for that reason) for the built-indaemonaccount.When the Ars Technica article says “Second, the target system account must be configured to be passwordless”, what they mean is the default
daemonaccount, which has a hash set to*by default, and thus wouldn’t normally be accessible via any hashed password… unless you happen to find out that the system chooses to not make a hash, and just send*if you type more than 256 characters.Look at https://sshhole.com/ if you want the details.
Oh, thanks for details! Missed that.
Technically they’re correct though, in a way. If every account has a password, the system isn’t vulnerable.
You’re also not vulnerable if you use two factor auth, because all 2FA modules require PAM, whereas this vulnerability only exists when PAM is disabled or not installed.
Other way around. This requires a passwordless account configured with sha256 or sha512 password hash, musl system with openssh installed, no PAM, and password authentication allowed:
When logging into a passwordless account, the attacker can gain access by inputting more than 256 characters in the password field.
Given this is a vuln when entering the password, just turn off password auth and use ssh keys only. It’s what you should be doing anyway.
The vuln is specifically for key-based accounts:
Second, the target system account must be configured to be passwordless. Passwordless accounts are commonly used for accounts that use certificate-based authentication.
not true. regardless if all your user accounts are using key based auth or not, some system accounts will always be passwordless, because they are not supposed to be logged into.
“passwordless” here means that the account doesn’t have a password. What the person you’re replying to means is that password authentication should be completely disabled in the SSH config, as that’s the only way to exploit this issue.
But passwords need to be enabled for SSH. You must submit a faulty password of a specific kind, and the SSH server misinterprets it for passwordless accounts, which means the server must be configured to allow you to attempt a password.
When logging into a passwordless account, the attacker can gain access by inputting more than 256 characters in the password field.
If you disable passwords in SSH, there is no password field. There is no way for you to input anything. The connection will just be refused as the only auth method is an ssh key. If there is no matching key, SSH will refuse the connection.
If you don’t do that, the daemon will fall back to a password, which might then be vulnerable. However, no sane administrator would leave password auth enabled. Usually, you use the SSH password once - to copy an SSH key on the system. Then you disable it.
Easier said than done.
If you have to support the sort of user that barely managed to download a copy of putty a few years ago and kept using this, then walking them through key generation and use is a nightmare.
One could fairly argue that maybe such people shouldn’t have access to a system, but it is a reality.
Of course the likelihood that a system using musl is something that class of user would be accessing is lower.
Maybe now that ssh is generally “just there” for Windows users, you can more easily work with them to generate a key, or give them a zip file to make them have a key in a standard way. But the interaction of ssh and windows file security can be… Interesting, though mostly an issue on the ash server side.
For added security, there is a pam module for 6-digit OTP
Fuck. OpenWRT might fit that bill.
Edit: it does, kind of:
https://forum.gl-inet.com/t/sshole-exploit-does-not-impact-gl-oem-firmware/72657
The person who discovered *sshole said OpenWRT is affected, but by default, that OS uses the Dropbear package, not OpenSSH, for SSH connections (though maintainers provide instructions for installing the latter).
What an aSSHole!
Further, OpenSSH using musl (pronounced muscle) must not use PAM (pluggable authentication modules),
which Alpine doesn’t ship by default. Maybe one or the other remote administration suite depends on it, but those are often suspectible to exploits anyway. But you don’t expose either to WAN, right?
Most ssh ports should be accessed through a VPN anyway, not exposed directly to the internet
Are you sure the VPN server has the smaller attack surface than the SSH server?
Wireguard and apps based on it (like Tailscale) don’t respond at all unless the data is signed with the key of a configured peer (or with Tailscale, a system on your tailnet that passes the ACL). It’s literally impossible for an outsider/attacker to detect with a port scan since it just ignores unauthenticated packets.
The Wireguard codebase is also extremely small. The core Wireguard implementation (the version in the Linux kernel) is around 5,000 lines long, so an experienced developer can read through all the code if they dedicate a day or two to it, and understand how it works.
For comparison, OpenVPN is around 70,000 lines of code and OpenSSH is around 150,000 lines of code. There’s a huge number of features in SSH!
That is a convincing argument. I wasn’t thinking of wireguard, but it makes sense that it’s quite small and focused, being in the kernel.
I’m dealing more with OpenVPN and that can be very annoying, that had me lean in the opposite direction earlier (and lets not even speak of the Cisco crap).
Why aren’t there more minimal implementations like tinyssh?
There’s Dropbear SSH, which is the most common choice for a minimal SSH server and client, but even that is around 50,000 lines of code.
Because all those extra features are really useful. You can run all sorts of things over ssh, from sophisticated file transfer to remote development and tunneling
Ssh is remote code execution. Under any implementation, any authentication methods should be the absolute last line of defense after you’ve properly locked down access
Well, yeah but more features in remote code excution = more vulnerabilities.
For example sftp: you could also just run ftp over vpn for the same effect. Or just cat the file over ssh.Instead, we have one big monolith.imo it does not really matter how many lines is the openssh codebase, but how many lines of it deals with authentication/authorization and decryption of the incoming traffic.
then it can have all the features, but any bugs in them can only be abused by clients that were allowed access.
still, the related part of Wireguard’s codebase is likely much shorter, because it does not have multiple authentication methods and key algorithms, and whatever else. but then it still makes sense to use sftp over wireguard, because of its authentication inside the network and relative ease of use.
SFTP is just FTP over SSH… Also, FTP is essentially one small step away from code execution, even on a secure network it’s better to use SFTP
SSH isn’t many things, it’s one connection type through which you can do many things. You can’t simply cat a file over without being able to handle the throughput gracefully, otherwise you’d have to design a new layer to do exactly the same thing. The same for reading files, if it can’t gracefully spit out 10k lines in a second you couldn’t handle output
And if you can do all that, why not allow VIM over the same connection? You’re already going to have to manage the terminal session, why not fully reflect a terminal?
FTP is absolutely not the answer, but a server that will conditionally serve files as well as conditionally accept uploads is far more secure. The same for everything… If you want to manage users, instead of allowing remote code execution you should use an API to perform predefined tasks
SSH isn’t meant for security or production use, it’s meant for power and flexibility. It’s generally deactivated by default for a reason, but it’s included for a reason too. It’s just really damn useful
Ok, maybe i misunderstood ssh.
OpenVPN, no. Wireguard, yes.
Well that sucks. IIRC a lot of my docker images are alpine based, but they’d be safe due to never having their SSH ports exposed I assume.
Why would you even have an SSH server running in a Docker container (unless the purpose of the container is being a jumpbox or similar)?
I can’t emphasise it enough how much more secure you get when you use rootless, read-only containers that are limited to truly just the software you want to run and its dependencies.
Alpine is great for dev containers, but the final container should be:
- immutable on the app layer (any executable path needs to be read only)
- have only the absolutely necessary GNU tooling, service management, etc. included (for most software you don’t even need these tools, and why would you give an attacker who gain access to your container, an easy way to e.g. scan your network?)
- only the data mounted, rw only if absolutely necessary (e.g. if you host a Jellyfin server, just an example, you should have the mounted media be RO, and only the config and caches be writable, mounted noexec of course)
Not just rootless and read-only, but distroless/chiseled too. There’s no reason to have unneeded parts of an OS in your Docker containers. It should just be one app along with any libraries it needs (or just statically compile the app)
While I have not done it, I could see it being appealing as it’s quicker than SSHing into the parent box first for a commonly used tool. I had a direct SSH into a systemd container for a while before deciding the extra config wasn’t worth it.
Could that not be configured with a proxyjump through the host machine still, if one needed to have this?
unless the purpose of the container is being a jumpbox or similar
What’s wrong with a rootless jumpbox?
Nothing, my point was that SSH in a container is unnecessary unless it’s for the specific purpose of the container.
deleted by creator
