| « | Savages Redux |
»
|
|
Friday, June 17, 2005
From The Bit Bucket: Adventures In FibreChannel-Land
Enough with the heavy sociopolitical crap for today. It's time for some good old-fashioned kvetching.
The great majority of you know your Curmudgeon solely as an overflowing fount of opinions on politics, culture, and society. That's only because you've never seen him with his shirt off. For beneath the mild-mannered sociopolitical commentator's exterior is another layer of reality: the khaki shirt, pressed jeans, and pocket protector of:
...sworn defender of Truth, Justice, and Really Weird Linked Lists. He who sallies forth daily to beat back the digital demons ever straining to encroach upon our excessively computerized lives. Who laughs in the face of defective device drivers, routinely patches object code in hexadecimal and edits registry entries by hand, sneers at Blue Screens Of Death, and debugs even the most complex applications with no more than a front panel, a telephone, and a really big hammer.
Truth be told, it can get wearying at times being a hero to screaming millions. Altogether too many people know that if they need something done quickly, they should take it to the busiest man around. Guess who that is in your Curmudgeon's shop? Well, he knew the job was dangerous when he took it.
Anyway, today's tale from the digital domain has actually been accumulating over the past seven weeks. It concerns a not-particularly-popular communications interface called FibreChannel. Your Curmudgeon's employer is involved in a collaborative effort with another defense contractor, henceforth to be called The Other Company, to develop a weapons system that imbeds this interface. Indeed, it's the most important information conduit in the system.
However, the way in which FibreChannel was designed into this system suggests that some system architect at The Other Company needs psychiatric help, and pronto.
FibreChannel is capable of many different "styles" of communication, the great majority of which look and feel pretty much like any other scheme for passing messages among a bunch of processors. But there's one, called Remote Direct Memory Access (RDMA), which attempts to simulate "tight coupling" among the communicating processors; in other words, it purports to give each processor direct access to a segment of every other processor's memory. The segment of memory a processor exposes to others on an RDMA FibreChannel network is called its "target buffer."
We who understand the principles of message-based application architectures would never dream of using RDMA for such a thing:
- Each processor has to know the semantic significance of every address in every other processor's target buffer.
- Each processor has to watch its own target buffer for changes, which could occur anywhere in it, and need not be coherent with anything else in it.
- Any data written into a processor's target buffer could be overwritten by subsequent data without the processor ever becoming aware of the event; a transfer initiator is under no obligation to allow its target to become aware of any change to its target buffer.
- All provisions for clocking, time-stamping, verification of sound transfers, and acknowledgement of transfers are application responsibilities.
So which of the multitudinous FibreChannel protocols did The Other Company insist upon for this weapon system?
You've got it. RDMA.
Their reason? RDMA has "the lowest latency" of all the FibreChannel protocols. Oh, sure, as long as you're willing to endure all the faults and vulnerabilities cataloged above. But The Other Company wanted reliability, too. So they wrapped an ornate set of proprietary protocols, heavy with reliability-enhancing measures, around RDMA, to make it look more like a conventional, and conventionally reliable, message-passing scheme. In doing so, they destroyed all the response-time and low-latency virtues they'd chosen RDMA specifically to get, but hey, all things have their price.
There's a saying about the folly of putting lipstick on a pig.
Well, one must adapt or die, mustn't one? So when your Curmudgeon was told that his group was to develop a set of laboratory simulators in which this weapon system would be tested, he gave the application layers to his subordinates and took the worst of the responsibilities -- developing the FibreChannel RDMA interfaces -- onto his own shoulders. That's what Alpha Geeks do. It's a point of pride. Besides, it keeps us off the Net at night.
But there was more in store for your Curmudgeon than just a few days' work wrapping unnecessary protocols around a badly chosen interface technique. My word, was there ever more!
FibreChannel was originally intended as a support for clustered file servers and Storage Area Networks, the better to allow them to share one another's disk storage. The usual implementation simulates a Small Computer Systems Interface (SCSI) multi-master bus, and is packaged with an extensive, expensive Storage Area Network management system. FibreChannel implementations for general-purpose, off-the-shelf systems are quite rare. Implementations for Windows that support RDMA are even rarer: at this time, there are only two commercially available, and one of them flatly doesn't work.
Well, that did seem to eliminate your Curmudgeon's buying decisions. But it's one thing to know that one must use a particular vendor's product, and quite another to actually use it. More, the vendor of that one-and-only-working-RDMA-implementation has a case of corporate paranoia that boggles the mind; they require non-disclosure agreements before they'll even admit that they sell FibreChannel technology. So it was apparent from the first that your Curmudgeon would be "working without a net." What wasn't apparent was how far it was to the ground.
It took your Curmudgeon seven weeks to puzzle out the application interface to the only adequate RDMA FibreChannel driver:
- The manual that comes with the driver is more than four years out of date, and is demonstrably wrong on many specific points.
- The driver offers several points from which one may seemingly change its operational parameters, but fully half of them don't have any effect -- and the documentation says nothing about this.
- Though a test program written in C is provided, it's horribly bad, written in "spaghetti code" style, in which all the variables and data structures are global, and thus pass retained values all over the program. Moreover, it's not documented nor commented; the reader is expected to understand the code without any extrinsic assistance.
- The test program quietly omits to address the very scenario your Curmudgeon had to implement: RDMA protocol over a switched network.
- The sole support engineer for the driver and test program is a very nice young man, a brand new graduate in engineering whose Indian accent is thick enough to stop up the Grand Canyon. No other support is available; the original driver programmer is long gone.
But your Curmudgeon was assured that The Other Company was using this very product, and had gotten it to work more than adequately well. Of course, The Other Company's code was proprietary, and therefore unavailable to your Curmudgeon. So it became a point of pride, to say nothing of the neck-on-the-chopping block effect, for your Curmudgeon to break the monster's back and produce a working system by the sheer power of intellect and the force of his indomitable will:
- He wrote multiple test programs for each and every driver operation.
- He rewrote the vendor's own test program.
- Several times, in fact.
- He lived on the phone for hours at a time, torturing all the information he could get out of that poor Indian support guy, who probably never guessed at your Curmudgeon's severe problems with tinnitus and general hearing loss.
- He pleaded with The Other Company for source code that would illustrate the very simple operations he couldn't get to work...in vain.
- He prayed. Perhaps that was what made the difference. One simply cannot know.
Well, today your Curmudgeon finally broke through, not merely in his mastery of the RDMA FibreChannel driver, but also in implementing the reliability protocols The Other Company wrapped around its bad design decision. He ought to feel a massive sense of accomplishment and relief...but he doesn't, because a rational design decision -- to use Gigabit Ethernet over fiber optics, for example -- would have made all his travails unnecessary. So would a properly implemented and documented driver and test program.
Why did The Other Company elect to bathe in this technological backwater? Why, having made that choice, did it choose to clutch the details of its solution to its chest? And why, once he'd grasped the full and pervasive horror of the thing, didn't your Curmudgeon do the sensible thing and circulate his resume?
Some mysteries are beyond the penetration of even the Alpha Geek. Well, let it pass. But your Curmudgeon has been told, just today, that the vendor is about to release a new version of the driver and test program, which:
- Will be fully and properly documented;
- Will contain a heavily revised, properly structured, documented and commented test program;
- Will address all the cases omitted by the previous test program, with particular emphasis on the ones your Curmudgeon was interested in.
Oh, and the young support engineer, who's also doing the re-release, gently asked whether your Curmudgeon would be kind enough to send his own special interface routines, test programs and personal documentation along...just to assist him with his coverage analysis, of course.
How could even a case-hardened Curmudgeon refuse so polite and humble a request?
Comments
Um, you could tell him to kiss your bleeding Irish-American ass?
Posted by CGHill on 06/17/2005 at 08:25 PMLet’s see......
I’ll guess you spent at least 500 hours fighting your way thru the vendor’s maze (yes, I know you work more than 40 hrs/week), and the consulting rate for a true Alpha Geek is a nice, round number like $500/hr (since Alpha Geeks are *much* *much* rarer than the run-of the ambulance chasing law weasel)…
Sounds easy. They pay your invoice for $250,000 in development time and they can have your code (to repackage and ship with their <sarcasm> fine </sarcasm> piece of hardware.......
Posted by on 06/18/2005 at 12:48 AMGive them the code. Not because they deserve it (of COURSE they don’t!!!) but because it may save your counterpart at another company some fraction of the pain you’ve just suffered. This hypothetical counterpart does not deserve to be punished for your frustrations, even as a side effect of punishing the vendor (who does).
Just MHO, of course, after reading a post which made me either cringe or scream in agony with practically every sentence. (Never worked with RDMA. And I’m glad. Just getting FC itself to work is hard enough.)
It almost doesn’t matter what decision they made...any engineering team that doesn’t understand that in most applications a protocol like RDMA won’t so much _reduce_ latency as _push it up the stack_ and thereby make it _worse_...well, such a team is all but guaranteed to make some collossal blunders from sheer ignorance and thoughtlessness.
Posted by Matt on 06/18/2005 at 02:11 AMYour Curmudgeoness,
I would hope that your management have enough
esteem for your opinion to raise the issue with
the other company and-uh the customer.
No expert in this field I am but evidence of
impending trainwreck plain is.Posted by on 06/18/2005 at 10:38 AMWow. Reason #102 I don’t want to get in that field for a living. Granted, I say it like mathematics research is any different.
Posted by Jason on 06/18/2005 at 11:58 AMIf there is any chance at all of the powers that be at the top of your corporate food chain understanding/comprehending the sweat and toil you have been through, I would say go for the substantial billing for work above and beyond the call of duty. Then hand over your work to “Paranoid Products Inc” whose newest version is ,in fact, “powered by Porretto” ™ © . As a side note, has this caused you to break out in a cold sweat when ever someone mentions the governmental entity who would be buying said end product, has once again changed *their* parameters?
Posted by Guy S on 06/18/2005 at 12:34 PM
