Skip to content

Lions and Tigers and h4x0rZ OH MY!

Posted by mikeg on
UNIX What do Romania, France, North Korea, South Korea, China, Czechoslovakia and Bulgaria have in common? Simple, they're all countries that show up in my SSH Access Blacklist Table. Why am I talking about this? My place of work got h4x0r3d yesterday (our Nessus security scanning server -- how's that for irony?). I will preface this by saying that **I** did not set up the aforementioned server (and thus my "Incredibly Unhacked" record is untarnished). I'm not particularly upset by this. Irritated yes (especially with myself -- I never read those stupid "You last logged in from [x] at [y] O'clock" messages (nobody does)), but upset no. The poor guy who set up the Nessus box however, looked like he wanted to cry. And the look on his face when he told me was kinda sad - he's probably not reading this but if you are - don't worry about it. Machines get hacked, we post-mortem them and move on. Even I'm not perfect (my record notwithstanding) and I've found some HUGE security flaws on my systems over the years - I've just had good luck at finding them before other people do. We got hacked because the machine has SSH open to the world. That's not a bad thing in and of itself -- flexo.bsd-box.net (the server this blog lives on) has SSH open to the world. In hindsight, since there are a LOT of machines and nobody to check up on each one every day SSH should probably have been restricted to our "Border SSH Machine" (the one host that is SUPPOSED to have SSH wide open, and is carefully monitored), but this is only a symptom, not the Real Problem. By the By, our head CISCO guy (and head of network security) ran scans from this server, logging in from random places, AS ROOT. He should have known better and locked down the access (instead of using it), but that's neither here nor there... The Real problem is SSH was configured to allow root logins in the first place. Not JUST root logins, Root logins using a regular old UNIX Password. And SSH being less-than-smart will happily let you keep reconnecting and trying passwords until you get it right! (It took our friend over 20,000 password attempts to get root on the Nessus box) I have no problem with allowing root logins -- actually I have a HUGE problem with it but for practical reasons you can't just say "No remote Root logins" -- BUT the system should only allow Root to log in with an "appropriately huge" (2048 bit) Public Key authentication, and it HELPS if you can secure the access list for SSH to something more reasonable than "Everyone in the universe". Now of course using keys for root doesn't solve the problem of J. Random Hacker breaking in as a regular user -- Even if you disable Root logins they can still hack user accounts, and insisting all your users use 2048 Bit SSH Keys is somewhat unreasonable (it's hard enough getting them to use SSH and SFTP), so how do you deal with the jackasses trying to Brute Force their way in? My SSH Log rolls a few times every day because it gets over 100KB in size (that's a LOT of failed auth attempts!), surely someone will eventually persist at this game long enough to brute force even the most lengthy and random passwords, right? Well, I deal with them by using a little script, customized for FreeBSD and Security-Nazi-ism. Basically what it does is watch the SSH Authentication log, and after 4 failed password attempts (or 4 invalid usernames) it assumes that you are trying to hack my machine and adds you to a blacklist for a year (or until I manually remove you after you call me and explain how you mistyped your username four times in 10 minutes, which will of course require me to make merciless fun of you and shame you in to remembering your login ID). For everyone's enlightenment, the gory details are below
First, the firewall rules; and I will give you all my WHOLE pf.conf file: 20051028-pf.conf [Disk Crashes Suck - The file's gone. MG20070718] There's nothing special about it really, mostly basic stuff right out of the pf howto. The Important Parts are the "table ssh-blacklist persist" line (line 3), which means "Create an empty list of addresses, and keep it around (even if it's empty, or no rules refer to it), and the first "pass in on fxp0" line (line 14), which means "As long as I haven't blacklisted the fucker, let him connect to SSH" Next, the blacklister code: 20051028-sshblack.pl.src [Disk crashes still suck.] This is actually based on Someone Else's Code, modified slightly (use pf, poll /var/log/auth.log instead of security.log, the blacklisting lasts for a year, PID is stored in /var/run/sshblack.pid, Blacklist lives in /var/db (instead of /var/tmp), and the blacklist is reloaded when you first run the program). All of this is configurable, so feel free to tweak as needed. This program should be run (and backgrounded) when you boot the machine. By default it runs in the foreground and logs to STDOUT (you DaemonTools afficianados should love it!). It will monitor your SSH Daemon's log file and blacklist anyone who seems to be trying to hack you. My changes ensure that the blacklist will persist across reboots (a fatal flaw of the original script in my opinion -- what good is a security nazi shitlist if it goes away after I reboot and they can try to hack me again?) Running this will keep you from getting brute-forced by most crackers -- they'll hit the blacklist long before they can guess your password. If anyone with enough IP Addresses to brute force your usernames and passwords after this is attacking you, you have FAR more serious concerns than any mere perl script can help you with. Note that if you have other "ways in" (FTP, Telnet (satan forbid), IMAP, Web Access, etc.) you should modify the script to look at their log messages as well (this script JUST does SSH -- the one on flexo also considers login failures from the IMAP Daemon). Remember that if they cant brute force one login point they may just try another. Feel free to send me improvements to this script -- like a fine wine, it has gotten better with age (and development iterations :)) [Note: I've since found better, more reliable ways to do this. specifically, the pam_af (anti brute-force) module. I suggest using that rather than this firewall hackery]

Trackbacks

No Trackbacks

Comments

Display comments as Linear | Threaded

Jackson Tilley on :

*You should probably be reloading your entire firewall config everytime you reboot anyway. I mean MORE than just the blocked addresses. It would be easier to just do a daily or hourly dump of your pfctl config and make sure that is pulled in when you reboot. The ssh blockings would be loaded with everything else. You also could be loading duplicate blockings if you don't check them first. Like when you restart the code but havent actually rebooted. The blockings would still be in there and when the code restarts it would pull in the cache blocks and be duplicates.

mikeg on :

*Dumping the pf configuration makes sense in a lot of cases, particularly when a lot of content is being dynamically generated. In my case though, I only dynamically update the blacklist table, and the minute or so during reboots where the table isn't loaded yet doesn't bother me too much. If I had dynamically generated tables that could not be easily regenerated by the program that created them, I would certainly cron a dump of the pf configuration to make sure I could always recover the configuration in its latest state. Among the tweaks I've made since posting this, I updated the code to flush the blacklist table before reloading the blocked addresses (fixing the duplicate address issue). I'm not sure if pf does the Right Thing when it sees duplicate addresses (i.e. ignores the second one) or if it creates a second entry in the table, but either way this is a cleaner way to go. More importantly, I changed the base pf rules from "No SSH for you!" to "Fuck off and die, BITCH!" -- drop any traffic from addresses in the blacklist, on the grounds that if I'm not letting you SSH to the machine anymore because you tried to hack me, I probably don't want you abusing my web server or other services either (major senior moment that I didn't do this in the first place). At some point I intend to revisit this script in greater detail to improve the attack-detection logic (which is pretty damn good, but I'm sure I can find cases I missed if I look hard enough). I'm particularly interested in capturing "401" errors from my web server (which obviously don't go to auth.log, but there are some password-protected areas that use passwords culled from the UNIX password file, and I'd like to know if people are trying to break in to those. This script is all the more important since I just enabled password authentication (previously I only allowed publickey and keyboard-interactive -- and the latter reluctantly) to support my new toy (entries about THAT coming soon). :-)

Add Comment

E-Mail addresses will not be displayed and will only be used for E-Mail notifications.
To leave a comment you must approve it via e-mail, which will be sent to your address after submission.

To prevent automated Bots from commentspamming, please enter the string you see in the image below in the appropriate input box. Your comment will only be submitted if the strings match. Please ensure that your browser supports and accepts cookies, or your comment cannot be verified correctly.
CAPTCHA

Form options