mikeg's blog

This is part Zero of an N-part series (I'm thinking 4 parts) discussing the investigation of radmin as a patch/deployment tool for FreeBSD. It will be filled in over the course of Q2/2009 as we test (and possibly deploy) radmin at Premier Heart. As many (if not all) of you know, I'm a BSD Bigot - I firmly believe that the BSD projects have a huge edge over Linux on servers in all the areas that matter to me most: Performance, Security & Stability. That being said, there's one area where they suck big floppy donkey dick: Patch Management. Over the years I've had several solutions:

1. Ignore the Problem
   Don't patch and hope you never get compromised. This has never been an acceptable solution to me.
2. Patch Manually
   Log in to every machine in your organization once a quarter, do a Make World / Portupgrade and deal with the fallout.
   This is great if all you have are 1-2 machines, and it's how I patch bsd-box.net.
3. Deploy a Build Server
   A central machine builds the world and ports, then you log in to each machine to install them. It's manual patching, but you don't wait for Make World to run a bunch of times.
   This is what most BSD admins do, but I'm lazier than the average admin.
4. Deploy a Commercial Solution
   This is OK as far as it goes, but using commercial solutions puts you on someone else's patch schedule.
   Also, I'm not aware of any commercial solutions that Don't Suck™.
5. Use freebsd-update
   While I greatly admire what the FreeBSD Security Team has done with freebsd-update, it's just not for me -- Patching my custom software and ports with freebsd-update is a pain in the ass (I would have to essentially roll a custom release - More work than I want to do.) Also, freebsd-update is geared toward applying the security team's patches, not managing system deployments - that's only half of what I want.

None of these solutions work for me, so for the last 3 years or so I've been trying to find one that does. I've examined a few options, from the traditional "let a junior admin do the patching" through writing a remote execution program to handle it. None of these give me what I need: A simple, straightforward way to go from configuration X to configuration Y without breaking anything. Continue reading "Managing Multiple FreeBSD Machines with radmind -- Part Zero"

ZOMG! Something good came of Linux? I'll eat a Yankees hat... Yes folks it's true, something that Finnish Fucktard created has actually impressed me. The lead developer at the new job was shopping around for a new version control system (because CVS Sucks) and he wanted to try GIT because of its offline capabilities (commit locally when you're not connected to das interweb, then push it up when you are). For the record, that's something SVN doesn't do either. So I set out on a mini crusade to learn about and deploy GIT for him - Stumbling along the way into a bit of Python called gitosis which allows you to manage GIT repositories & access using SSH keys (w00t?). I played around, deployed it, ran a wonderful CVS-to-GIT script, and found GIT + gitosis to be quite good. I'm even considering moving some of my stuff into GIT. GIT also has one major advantage over CVS/SVN in my mind -- The distributed local copy is a full repository - therefore each workstation with a working directory is a de-facto backup of the master repository (as of whenever a pull/update was last done). Should a meteor strike the GIT master server all one would need to do is put one of these repositories on another box somewhere and development can continue relatively uninterrupted (minus the gitosis configuration -- I haven't quite worked that out yet). Continue reading "You little git!"

It's a disaster. A bloody pig's breakfast it is! Apparently my dspam maintenance scripts have been "not running" for about... 9 months. I have 9 months worth of stale data in the system. I just purged 13742670 hysterical tokens from the system (tokens with < 5 hits that are older than 2 months). Let's look at that number again: 13,742,670. 13 MILLION, 742 Thousand, 670. That's just ONE table. The system is purging 4 more tables. Then it has to vacuum (with this much shit, I may have to offline postgres to vacuum). I've turned off sendmail. There's NO WAY it's going to be able to deliver anything while this is running (in fact sendmail was shitting itself because it couldn't get answers from dspam within 5 minutes and shit was stacking up in the mail queue). Better to have the secondary MX and remote sites queue this shit than beat up an already upset server... On the bright side, I know why dspam has been so goddamn slow - apparently the purge script doesn't work and you HAVE to use the SQL purge. Fixing this is priority #2 (right after getting the database cleaned up so performance stops sucking)).

Im not dead. Flying update Sunday/Monday after my next lesson. In the interim, MARVEL BITCHES. Continue reading "Not Dead"

What do Romania, France, North Korea, South Korea, China, Czechoslovakia and Bulgaria have in common? Simple, they're all countries that show up in my SSH Access Blacklist Table. Why am I talking about this? My place of work got h4x0r3d yesterday (our Nessus security scanning server -- how's that for irony?). I will preface this by saying that **I** did not set up the aforementioned server (and thus my "Incredibly Unhacked" record is untarnished). I'm not particularly upset by this. Irritated yes (especially with myself -- I never read those stupid "You last logged in from [x] at [y] O'clock" messages (nobody does)), but upset no. The poor guy who set up the Nessus box however, looked like he wanted to cry. And the look on his face when he told me was kinda sad - he's probably not reading this but if you are - don't worry about it. Machines get hacked, we post-mortem them and move on. Even I'm not perfect (my record notwithstanding) and I've found some HUGE security flaws on my systems over the years - I've just had good luck at finding them before other people do. We got hacked because the machine has SSH open to the world. That's not a bad thing in and of itself -- flexo.bsd-box.net (the server this blog lives on) has SSH open to the world. In hindsight, since there are a LOT of machines and nobody to check up on each one every day SSH should probably have been restricted to our "Border SSH Machine" (the one host that is SUPPOSED to have SSH wide open, and is carefully monitored), but this is only a symptom, not the Real Problem. By the By, our head CISCO guy (and head of network security) ran scans from this server, logging in from random places, AS ROOT. He should have known better and locked down the access (instead of using it), but that's neither here nor there... The Real problem is SSH was configured to allow root logins in the first place. Not JUST root logins, Root logins using a regular old UNIX Password. And SSH being less-than-smart will happily let you keep reconnecting and trying passwords until you get it right! (It took our friend over 20,000 password attempts to get root on the Nessus box) I have no problem with allowing root logins -- actually I have a HUGE problem with it but for practical reasons you can't just say "No remote Root logins" -- BUT the system should only allow Root to log in with an "appropriately huge" (2048 bit) Public Key authentication, and it HELPS if you can secure the access list for SSH to something more reasonable than "Everyone in the universe". Now of course using keys for root doesn't solve the problem of J. Random Hacker breaking in as a regular user -- Even if you disable Root logins they can still hack user accounts, and insisting all your users use 2048 Bit SSH Keys is somewhat unreasonable (it's hard enough getting them to use SSH and SFTP), so how do you deal with the jackasses trying to Brute Force their way in? My SSH Log rolls a few times every day because it gets over 100KB in size (that's a LOT of failed auth attempts!), surely someone will eventually persist at this game long enough to brute force even the most lengthy and random passwords, right? Well, I deal with them by using a little script, customized for FreeBSD and Security-Nazi-ism. Basically what it does is watch the SSH Authentication log, and after 4 failed password attempts (or 4 invalid usernames) it assumes that you are trying to hack my machine and adds you to a blacklist for a year (or until I manually remove you after you call me and explain how you mistyped your username four times in 10 minutes, which will of course require me to make merciless fun of you and shame you in to remembering your login ID). For everyone's enlightenment, the gory details are below Continue reading "Lions and Tigers and h4x0rZ OH MY!"