This is going to be one of my rambling sysadmin-y entries talking about stuff that's probably of little general interest. Fair warning given.
There are two components that I would consider essential to proper system administration: GOOD Monitoring and Issue/Incident Tracking. It is imperative that you know when a problem arises (preferably before anyone else notices) and that you keep track of the problems you have encountered in order to spot troublesome systems and redesign them to stop bugging you.
Those of you who have worked with me know I have my prejudices in both of these areas, and that for the last few years I've settled on two pieces of software to fill these roles: InterMapper for monitoring and RT for issue tracking.
The major caveat of this pairing is that the two have no formal integration: InterMapper will happily send emails, and RT will happily accept emails and turn them into tickets, but RT doesn't know when InterMapper is telling it about the same problem twice, or that a previous issue has been cleared. The end result of this lack of integration is that you have a bunch of RT tickets for the same issue which need to be manually merged and resolved, and this manual bit bugged me enough that I actually took the time to fix it!
While I usually read El Reg for teh lulz occasionally they come up with an interesting nugget, like their article on a "severe" OpenSSL vulnerability (quotes theirs). This vulnerability comes to us courtesy of the University of Michigan, and is tied to a simple oversight in the OpenSSL code: It doesn't perform a verification pass on signatures before sending them to a client.
So all you crypto-nerds, how many of you do a verify on your signatures before you send them? Hands please? Nobody? Not even me? -- Yeah, we all tend to trust that our math Just Works. I mean it's a computer, you put in fixed input through a deterministic algorithm and you get the same output ever time. Right? Nope.
Computers, as we all hopefully know, aren't perfect. Trillions of minor errors happen every day, and they're usually compensated for: ECC RAM, cross-checks of math, etc. But because OpenSSL doesn't do a cross-check it's possible for those minor errors to creep into a transmitted signature. As the good folks from UMichigan explain in their paper if you can induce some minor errors in the OpenSSL signature math and collect a good signature and a large enough set of flawed signatures you can eventually derive the private key used for the signing.
That is all well and good, except it relies on errors in the system generating the signatures. These errors are rare (and pretty random) in the real world, so no problem, right? Wrong again Sparky! These clever folks came up with something I wouldn't ever have thought of.
Those of you who have worked on old Commodore computers probably know one of the first symptoms of a power supply going wonky is that the system starts acting up: Lots of random crashes, video corruption, etc. Why does that happen? Because the voltage being supplied to the system is fluctuating. Fluctuating voltages (specifically down-swings) cause all the magical voltage regulation hardware to go out of regulation, which feeds out-of-spec voltages to the chips, which in turn start making mistakes in their math.
The bottom line for this attack is an extension of that well-known Commodore problem: By putting controlled voltage fluctuations into the power supply of real computers (SPARC boxen running Linux) the UMichigan folks were able to induce errors in the math used by the real-world OpenSSL code and recover an actual key.
Continue reading "Interesting (impractial) OpenSSL attack"
OK, for the record: I DESPISE Linux. I Hate, Loathe, Abhor and Revile it. I am a BSD-Bigot and proud of it, and if all the BSDs suddenly evaporated I would eschew Linux in favor of a commercial Unix (probably AIX).
Why do I hate Linux so much? Simply put, it's shoddy code written by shoddy coders. In my experience shit mysteriously breaks for no reason, standards and conventions are arbitrarily ignored, critical components of the system are perpetually at version zero-dot-something, regression testing seems to be a myth, and the average Linux developer seems to make no effort to ensure their code will work on anything except their particular favorite distribution (to say nothing about porting it to <GASP> a BSD system, or <HORRORS> Commercial Unix).
This particular tirade was kicked off by the Ubuntu update-notifier program suddenly and mysteriously no longer popping up update notification balloons. As some of you know my company ships a Linux-based appliance (built around a very stripped-down Ubuntu plus our commercial packages), and one of the things that made me go with Ubuntu was that they had gotten the update-notifier thing working beautifully and it had been stable for several versions.
Lo and Behold about a month ago our support guys came to visit me and asked "Hey, is the update notifier bubble broken?" I looked upon it and saw that it indeed appeared to be non-functional, but as all good (lazy) admins are wont to do I demanded they test and verify the breakage.
The breakage came back to me verified earlier this week, and as I really couldn't be assed to figure out why the update-notifier is happy to display the "you must reboot!" dialog box but refuses to display the "Yo, bitch! You have updates!" notifier icon & bubble I took the easy way out and re-implemented update-notifier in Python.
Blocked waiting for my either the software development group to give me new code to test or my FreeBSD build VM to give me a new OS build to test, so how about some random thoughts on programming?
I've reached the inescapable conclusion that all software development cycles can be expressed as a single equation: y=tan(x+(π/2))
X represents time (on a totally non-linear scale) with each interval of width π being the development of a software release and the zero point where the function crosses the X axis representing the release of the X/πth version. The Y axis represents the state of the code - how "bug free" it is. I'll call the Y value within a version the "completeness" of the code for simplicity.
This functional model actually works surprisingly well:
During pre-0.0 release (X|0...π/2) the software is riddled with bugs and brokenness ("completeness" is negative - that shit don't work!).
At some point (X=(π+ε)/2) the software becomes at least functional (miniscule positive completeness), and is released to the unsuspecting public.
The initial release is buggy as shit, and massive patching and bug fixing happens
(This is roughly from X|(π+ε)/2...3π/2 -- For the sake of argument let's call the 3π/2 mark the .1 release, or in MS parlance, "Service Pack 1")
The software continues asymptotically approaching infinite completeness -- that nirvana state of having no bugs...
...At which point Marketing comes along and says the users want new features -- On our graph this corresponds to one of the vertical asymptote at multiples of π.
Development begins on the next (N/π)-dot-zero release, starting al over again from negative completeness.
In practical terms software development is not a true function: Each development window is independent and shifted toward X=0, with some overlap between the currently released version and the version under development.
These backups worked fine for a good while, but on Friday (my first full backup since upgrading to 8.0) the damn thing blew up and took the server down with it. Near as I can tell without being on the console something caused the FUSE kernel module to go insane - The system goes unresponsive, starts taking forever to respond to pings and eventually falls over entirely.
Troubleshooting that mess is on my list as soon as I can reproduce the problem in a test lab, but until then bsd-box.net is running a ghetto-ass tar-then-SCP backup routine (so I'm not a data protection douche since I've still got backups, but my backups are decidedly more ghetto).
Those of you who sent me emails, panic not: I have a very fine secondary MX which will eventually despool whatever you sent me. If it's urgent go ahead and resend.