mikeg's blog - Computers

InterMapper & RT: You two play nice now…

mikeg@bsd-box.net (mikeg) — Thu, 15 Apr 2010 15:30:00 -0400

This is going to be one of my rambling sysadmin-y entries talking about stuff that's probably of little general interest. Fair warning given.

There are two components that I would consider essential to proper system administration: GOOD Monitoring and Issue/Incident Tracking. It is imperative that you know when a problem arises (preferably before anyone else notices) and that you keep track of the problems you have encountered in order to spot troublesome systems and redesign them to stop bugging you.
Those of you who have worked with me know I have my prejudices in both of these areas, and that for the last few years I've settled on two pieces of software to fill these roles: InterMapper for monitoring and RT for issue tracking.

The major caveat of this pairing is that the two have no formal integration: InterMapper will happily send emails, and RT will happily accept emails and turn them into tickets, but RT doesn't know when InterMapper is telling it about the same problem twice, or that a previous issue has been cleared. The end result of this lack of integration is that you have a bunch of RT tickets for the same issue which need to be manually merged and resolved, and this manual bit bugged me enough that I actually took the time to fix it!

Continue reading "InterMapper & RT: You two play nice now…"

Interesting (impractial) OpenSSL attack

mikeg@bsd-box.net (mikeg) — Fri, 05 Mar 2010 13:31:23 -0500

While I usually read El Reg for teh lulz occasionally they come up with an interesting nugget, like their article on a "severe" OpenSSL vulnerability (quotes theirs). This vulnerability comes to us courtesy of the University of Michigan, and is tied to a simple oversight in the OpenSSL code: It doesn't perform a verification pass on signatures before sending them to a client.
So all you crypto-nerds, how many of you do a verify on your signatures before you send them? Hands please? Nobody? Not even me? -- Yeah, we all tend to trust that our math Just Works. I mean it's a computer, you put in fixed input through a deterministic algorithm and you get the same output ever time. Right? Nope.

in their paper

That is all well and good, except it relies on errors in the system generating the signatures. These errors are rare (and pretty random) in the real world, so no problem, right? Wrong again Sparky! These clever folks came up with something I wouldn't ever have thought of.

Those of you who have worked on old Commodore computers probably know one of the first symptoms of a power supply going wonky is that the system starts acting up: Lots of random crashes, video corruption, etc. Why does that happen? Because the voltage being supplied to the system is fluctuating. Fluctuating voltages (specifically down-swings) cause all the magical voltage regulation hardware to go out of regulation, which feeds out-of-spec voltages to the chips, which in turn start making mistakes in their math.
The bottom line for this attack is an extension of that well-known Commodore problem: By putting controlled voltage fluctuations into the power supply of real computers (SPARC boxen running Linux) the UMichigan folks were able to induce errors in the math used by the real-world OpenSSL code and recover an actual key.

Continue reading "Interesting (impractial) OpenSSL attack"

The Wheel: Let me reinvent it for you! (update-notifier redux)

mikeg@bsd-box.net (mikeg) — Fri, 26 Feb 2010 15:57:51 -0500

OK, for the record: I DESPISE Linux. I Hate, Loathe, Abhor and Revile it. I am a BSD-Bigot and proud of it, and if all the BSDs suddenly evaporated I would eschew Linux in favor of a commercial Unix (probably AIX).

Why do I hate Linux so much? Simply put, it's shoddy code written by shoddy coders. In my experience shit mysteriously breaks for no reason, standards and conventions are arbitrarily ignored, critical components of the system are perpetually at version zero-dot-something, regression testing seems to be a myth, and the average Linux developer seems to make no effort to ensure their code will work on anything except their particular favorite distribution (to say nothing about porting it to <GASP> a BSD system, or <HORRORS> Commercial Unix).

This particular tirade was kicked off by the Ubuntu update-notifier program suddenly and mysteriously no longer popping up update notification balloons. As some of you know my company ships a Linux-based appliance (built around a very stripped-down Ubuntu plus our commercial packages), and one of the things that made me go with Ubuntu was that they had gotten the update-notifier thing working beautifully and it had been stable for several versions.
Lo and Behold about a month ago our support guys came to visit me and asked "Hey, is the update notifier bubble broken?" I looked upon it and saw that it indeed appeared to be non-functional, but as all good (lazy) admins are wont to do I demanded they test and verify the breakage.

The breakage came back to me verified earlier this week, and as I really couldn't be assed to figure out why the update-notifier is happy to display the "you must reboot!" dialog box but refuses to display the "Yo, bitch! You have updates!" notifier icon & bubble I took the easy way out and re-implemented update-notifier in Python.

Continue reading "The Wheel: Let me reinvent it for you! (update-notifier redux)"

Trigonometric Programming: The tangent function & Software Development

mikeg@bsd-box.net (mikeg) — Tue, 23 Feb 2010 12:17:45 -0500

Blocked waiting for my either the software development group to give me new code to test or my FreeBSD build VM to give me a new OS build to test, so how about some random thoughts on programming?

I've reached the inescapable conclusion that all software development cycles can be expressed as a single equation: y=tan(x+(π/2))

X represents time (on a totally non-linear scale) with each interval of width π being the development of a software release and the zero point where the function crosses the X axis representing the release of the X/π^th version. The Y axis represents the state of the code - how "bug free" it is. I'll call the Y value within a version the "completeness" of the code for simplicity.

This functional model actually works surprisingly well:

During pre-0.0 release (X|0...π/2) the software is riddled with bugs and brokenness ("completeness" is negative - that shit don't work!).
- At some point (X=(π+ε)/2) the software becomes at least functional (miniscule positive completeness), and is released to the unsuspecting public.
- The initial release is buggy as shit, and massive patching and bug fixing happens
  (This is roughly from X|(π+ε)/2...3π/2 -- For the sake of argument let's call the 3π/2 mark the .1 release, or in MS parlance, "Service Pack 1")
- The software continues asymptotically approaching infinite completeness -- that nirvana state of having no bugs...
...At which point Marketing comes along and says the users want new features -- On our graph this corresponds to one of the vertical asymptote at multiples of π.
- Development begins on the next (N/π)-dot-zero release, starting al over again from negative completeness.

In practical terms software development is not a true function: Each development window is independent and shifted toward X=0, with some overlap between the currently released version and the version under development.

Lions and Tigers and Unscheduled Outages, Oh My!

mikeg@bsd-box.net (mikeg) — Mon, 08 Feb 2010 14:38:46 -0500

For those of you wondering where bsd-box.net went this weekend, here's the scoop:

A while back I stopped being a data protection douche and implemented backups. I didn't blog about the implementation, but it was bacula writing to rsync.net over sshfs (fuse for the win).

These backups worked fine for a good while, but on Friday (my first full backup since upgrading to 8.0) the damn thing blew up and took the server down with it. Near as I can tell without being on the console something caused the FUSE kernel module to go insane - The system goes unresponsive, starts taking forever to respond to pings and eventually falls over entirely.

Troubleshooting that mess is on my list as soon as I can reproduce the problem in a test lab, but until then bsd-box.net is running a ghetto-ass tar-then-SCP backup routine (so I'm not a data protection douche since I've still got backups, but my backups are decidedly more ghetto).

Those of you who sent me emails, panic not: I have a very fine secondary MX which will eventually despool whatever you sent me. If it's urgent go ahead and resend.

The Open-Source Environment (List)

mikeg@bsd-box.net (mikeg) — Tue, 13 Oct 2009 11:37:19 -0400

It occurs to me that aside from a few exceptions I've managed to get Premier Heart to a nearly 100% open-source / free(-as-in-beer) footing.

As 2009 is pretty much over I think it only makes sense to take stock of the software we're using for posterity - 5 years from now we can look back at this list and laugh the same way people laugh now when they remember FoxPro or COBOL...

Continue reading "The Open-Source Environment (List)"

Broken... and now, UnBroken

mikeg@bsd-box.net (mikeg) — Mon, 14 Sep 2009 11:55:07 -0400

So yeah, I decided to upgrade the server to FreeBSD 7.2 in preparation for the 7->8 transition.

SOMEHOW (IT speak for "I'm sure it's my fault, but I don't know what the fuck I did to cause it") flexo wound up with the GENERIC kernel instead of his nice customized one. Customized with all sorts of essentials, like GEOM_MIRROR which provides my RAID.

So obviously the system wasn't going to boot.

I did this on 9/10

CoLo closed(ish) on 9/11

Server fixed 9:30 on 9/14

Spam flow resumed 9:31 on 9/14

Kernel rebuilt. Much other tuning done.

Postgres and Apache upgrades next week. Hard to really fuck those up.

(Oh, and maybe I'll implement off-site backups now and stop being a douche)

Managing Multiple FreeBSD Machines with radmind -- Part Four

mikeg@bsd-box.net (mikeg) — Thu, 05 Mar 2009 14:40:00 -0500

This is part Four of an N-part series (Definitely 4 parts, plus an epilogue) discussing the investigation of radmin as a patch/deployment tool for FreeBSD. This part deals with creating and deploying patch sets using radmind. I'm assuming you already built an radmind server, created a base load and deployed it to some machine(s). Now you just found out about a "ZOMG SUPER CRITICAL MAY CAUSE YOUR SYSTEM TO SPONTANEOUSLY COMBUST!" vulnerability in the base load, and you need to patch your systems. This is why I started looking at radmind in the first place, and it's pretty well suited to the task.
Continue reading "Managing Multiple FreeBSD Machines with radmind -- Part Four"

Managing Multiple FreeBSD Machines with radmind -- Part Three

mikeg@bsd-box.net (mikeg) — Tue, 03 Mar 2009 11:50:00 -0500

This is part Three of an N-part series (I'm thinking 4 parts, plus an epilogue) discussing the investigation of radmin as a patch/deployment tool for FreeBSD. This part deals with initial deployment of machines using radmind, and it has some prerequisites: - You need to have a target machine (a naked FreeBSD install w/ the radmind package installed) - You need to have an radmind server set up - You need to have a base load built If this is the first time you're doing an radmind deployment there is an additional requirement: You need to be willing to break the target machine (It might happen -- If there is a problem with your base load you might wind up with an unusable or even unbootable system). As with Part Two, I'm not getting into the details of how to meet the prerequisites. If you can't get FreeBSD installed and add a package you really don't belong here.
Continue reading "Managing Multiple FreeBSD Machines with radmind -- Part Three"

Managing Multiple FreeBSD Machines with radmind -- Part Two

mikeg@bsd-box.net (mikeg) — Fri, 27 Feb 2009 12:18:00 -0500

This is part Two of an N-part series (I'm thinking 4 parts) discussing the investigation of radmin as a patch/deployment tool for FreeBSD. It will be filled in over the course of Q2/2009 as we test (and possibly deploy) radmin at Premier Heart. This part deals with some more basics: Creating an radmind "Base Load" to distribute to your machines. The radmind Base Load is the template that will be deployed to all servers and consists of Three major parts:

A "Positive Transcript" (often just called a transcript), which lists files and directories to be included in radmind deployments

A "Negative Transcript", which lists files and directories to be created or adjusted, but not replaced by radmind deployments

A "Defaults Transcript", which is a special case of a negative transcript.

A "Command File", which lists the transcripts to be applied, and the order in which to apply them. These transcripts are used to create "Load Sets", which are what radmind actually distributes to the clients.
Continue reading "Managing Multiple FreeBSD Machines with radmind -- Part Two"

Managing Multiple FreeBSD Machines with radmind -- Part Zero

mikeg@bsd-box.net (mikeg) — Wed, 25 Feb 2009 11:22:19 -0500

This is part Zero of an N-part series (I'm thinking 4 parts) discussing the investigation of radmin as a patch/deployment tool for FreeBSD. It will be filled in over the course of Q2/2009 as we test (and possibly deploy) radmin at Premier Heart. As many (if not all) of you know, I'm a BSD Bigot - I firmly believe that the BSD projects have a huge edge over Linux on servers in all the areas that matter to me most: Performance, Security & Stability. That being said, there's one area where they suck big floppy donkey dick: Patch Management. Over the years I've had several solutions:

Ignore the Problem
Don't patch and hope you never get compromised. This has never been an acceptable solution to me.
Patch Manually
Log in to every machine in your organization once a quarter, do a Make World / Portupgrade and deal with the fallout.
This is great if all you have are 1-2 machines, and it's how I patch bsd-box.net.
Deploy a Build Server
A central machine builds the world and ports, then you log in to each machine to install them. It's manual patching, but you don't wait for Make World to run a bunch of times.
This is what most BSD admins do, but I'm lazier than the average admin.
Deploy a Commercial Solution
This is OK as far as it goes, but using commercial solutions puts you on someone else's patch schedule.
Also, I'm not aware of any commercial solutions that Don't Suck™.
Use freebsd-update
While I greatly admire what the FreeBSD Security Team has done with freebsd-update, it's just not for me -- Patching my custom software and ports with freebsd-update is a pain in the ass (I would have to essentially roll a custom release - More work than I want to do.) Also, freebsd-update is geared toward applying the security team's patches, not managing system deployments - that's only half of what I want.

None of these solutions work for me, so for the last 3 years or so I've been trying to find one that does. I've examined a few options, from the traditional "let a junior admin do the patching" through writing a remote execution program to handle it. None of these give me what I need: A simple, straightforward way to go from configuration X to configuration Y without breaking anything.
Continue reading "Managing Multiple FreeBSD Machines with radmind -- Part Zero"

Woot?

mikeg@bsd-box.net (mikeg) — Wed, 12 Nov 2008 11:18:04 -0500

Spammers begone! I'm quite pleased - It doesn't solve the problem of the infected computers that comprise the botnets controlled by the folks hosted at McColo (so they could just start up again at another ISP), but at least it quiets the spam wars down for a few weeks. I'll take good news wherever I can get it :) Now, all of you motherfuckers with unpatched virus laden computers listen up: You're just as bad... no, you're WORSE than the McColo's of the world -- Clean your shit up so they can't keep using your machines for cannon fodder. Anyone with half a brain knows that this is just a blip and they'll be moving their machines to a new home (probably offshore with more unscrupulous upstream providers - the kind that don't care enough to cut off spammers and child pornographers but are too big for our Tier 1s to just kick off the internet), so take this reprieve for what it is and patch the holes in your roof! KTHXBAI.

I Can Has Cheezburger? No...

mikeg@bsd-box.net (mikeg) — Mon, 13 Oct 2008 14:40:35 -0400

... but Cheezburger can has ICANN. I bet next time Ceiling Cat wants a seat on the ICANN board they'll take him more seriously, huh?

ICANN't believe it...

mikeg@bsd-box.net (mikeg) — Sat, 28 Jun 2008 23:16:43 -0400

ICANN has officially lost their limited minds. Open Top-Level Domain registration. It's usenet all over again. Someone please stop teh intertubes, I wanna get off.... This has been a public service crying by a crusty old fart who remember when 14.4kbps modems were blazingly fast, CompuServe didn't suck (sucked less than AOSchnell!), and you had to manually start up PPP on your shell account if you wanted that fancy crap, otherwise you could use Lynx and Pine and be DAMN FUCKING GREATFUL FOR THE OPPORTUNITY! These kids today with their high speed multimeg pipes to the home... when I was your age we shared one T1 line for an entire class C of dial-up users! What? What do you mean "What's a 'class C'?" DAMNIT! DIE! (:-P)

You little git!

mikeg@bsd-box.net (mikeg) — Thu, 01 May 2008 10:39:51 -0400

ZOMG! Something good came of Linux? I'll eat a Yankees hat... Yes folks it's true, something that Finnish Fucktard created has actually impressed me. The lead developer at the new job was shopping around for a new version control system (because CVS Sucks) and he wanted to try GIT because of its offline capabilities (commit locally when you're not connected to das interweb, then push it up when you are). For the record, that's something SVN doesn't do either. So I set out on a mini crusade to learn about and deploy GIT for him - Stumbling along the way into a bit of Python called gitosis which allows you to manage GIT repositories & access using SSH keys (w00t?). I played around, deployed it, ran a wonderful CVS-to-GIT script, and found GIT + gitosis to be quite good. I'm even considering moving some of my stuff into GIT. GIT also has one major advantage over CVS/SVN in my mind -- The distributed local copy is a full repository - therefore each workstation with a working directory is a de-facto backup of the master repository (as of whenever a pull/update was last done). Should a meteor strike the GIT master server all one would need to do is put one of these repositories on another box somewhere and development can continue relatively uninterrupted (minus the gitosis configuration -- I haven't quite worked that out yet).
Continue reading "You little git!"