More Bad Software

By James Kwak

Last week BATS admitted that its software suffered from systematic problems for four years, failing to obtain the best execution price for about 250 customers and costing them about $400,000. That should be a giveaway: no self-respecting company would break the law just to steal $400,000 from its customers. This was a programming error, pure and simple.

Also last week, a RAND study revealed that, despite billions of dollars of investment, electronic medical records have done little to reduce costs for healthcare providers. This is more complicated than a simple programming error. The issue here is that projected savings of this kind are typically based on some model of how operations will be done in the future, and that model depends on perfectly-designed software functioning perfectly. Medical records systems apparently fall far short of this ideal: as the Times summarized, “The recent analysis was sharply critical of the commercial systems now in place, many of which are hard to use and do not allow doctors and patients to share medical information across systems.”

The common feature to these stories, however, is that big, complex, business software is really, really important—and a lot of it is bad. In many niches, it’s bad because there aren’t that many companies that serve that niche, it’s hard for customers to evaluate software that hasn’t been delivered and installed yet, and there are all sorts of legacy problems, particularly with integration to decades-old back-end systems. And most of the incentives favor closing the sale first rather than making sure the software works the way it should.

I don’t have much to add that I didn’t put in my Atlantic column on a similar topic last summer. Nothing has changed since then. So I’ll stop there.

27 thoughts on “More Bad Software

  1. An important factor in the lack of improvement from medical record keeping is that there is an on-going battle between providers and insurance companies. The insurance companies try to preclude payment for certain services based on the diagnosis codes. This has led to an arms race where providers fudge the diagnosis so that they can maximize payments. Meanwhile, well meaning politicians think that if we just have electronic records, we’ll be able to gather the data we need to provide the best service for the lowest cost. But this never happens because the data is polluted by profit-maximizing services employed by providers. Too bad “do no harm” doesn’t apply to the whole health care system.

  2. A root cause is that electronic medical record (EMR) systems and their vendors evolved as business logic solutions and not as medical resources. They are primarily concerned with ordering and billing for procedures and with the flow of money within a healthcare provider. The results of exams and other procedures were added to the system initially as collateral for services rendered, and were only later treated as a valuable medical resource. That means that the loyalty and priority of EMR vendors is with the management of healthcare providers who are their customers. Keeping the health information inextricably bundled in the same database as the business logic gives the vendors great leverage and they’re in no hurry to separate it. EMRs are also not regulated by the FDA as medical devices despite the fact thay they store and convey life critical information.

    Ideally the industry would be reformed by splitting these two functions: The healthcare business logic function which is legitimately private to each provider, and the electronic health record which should be patient-focused, shared across the healthcare industry, and regulated. This is unlikely to happen because there’s no good customer for the health record in the US. Insurers would be the logical customer but moral hazard required patients to keep their medical record secret from insurers. This may change with universal coverage but attitudes are sticky. Government won’t set it up due to political opposition. A few years ago Google and Microsoft offered health records directly to patients but they floundered for a variety of reasons including disempowered patients, an uncooperative industry, and lack of monetization.

    Since the US isn’t going to a uniform state-managed health system, the best likely scenario is consolidation of the industry down to a handful of giant care providers (ACOs under Obamacare) that each function like a national health system but are in competition with each other for subscribers. Each ACO will be motivated to realize efficiencies internally but transferring between ACOs will be as difficult as ever.

  3. “no self-respecting company would break the law just to steal $400,000 from its customers”
    I’m not sure if that is sarcasm or not….

  4. I worked on a cytology medical record system in the late 1970’s. It worked for the pathologists , but had to be scapped because 70% of the lab’s clients were new patients, and we were in effect creating a record system for patients the bulk of whom we would never see again.

    Cost savings for medical record systems work work for universal , single payer type systems , but don’t work in a US environment , where we refuse to take advantage of economies of scale.

  5. Kwak wasn’t being sarcastic, there, Engineer, I don’t believe.

    Medical records? Been overnighted at a hospital within the past year or so? Your 15 minute interaction with your friendly nurse, devolves into 12-13 minutes of data entry, with the nurse bent over a computer mounted on a rolling stand, and you in bed like a big beached whale.

    So much for electronic medical records…lol.

  6. These failures of software systems do not surprise me. I see it every day in the corporate world. The only viable solution (that can be paid for) is a clean slate approach with ( in this case ) very clearly defined interaction protocols for records. But executives can’t give up on #1 so the software is a cludge and it makes #2 basically impossible. For #2 the protocol basically needs to be defined like the various IT solutions out there. RS232 or HTML or similar. Common list of commands and data structures, etc. to allow interchangeability. But that runs into corporate profit interests.

  7. Boeing Dreamliners were grounded. So much for building in unecessary complexity in critical systems and tossing the manufacturing of those parts around the planet.

    400K “skim” is a lot of Fiat USA $$$$ when you are earning pennies an hour.

    @Bond – and no one checks whether the data the nurse entered is accurate. The only check is one that is pre-programmed in the software and asks, “Is that really the blood pressure? It’s out of range.” And then you have to enter it two more times to confirm it is out of range. So how many times do you think a newbie is “led” by the software to throw in a bp number that the software won’t keep challenging because it is “out of range”?

  8. @Bond – a computer is the human equivalent of a retard – a serverely non-functioning retard. GIGO – garbage in, garbage out.

    Medical practionioners don’t have the time, nor is it their job or responsibility, to take the same amount of time to feed the retard info in hope of it someday being smart enough to assist in diagnosis (especially when it is time sensitive decisions that make the difference!) as they spend time interacting with the patient to have that complete and complex human conversation that is the ONLY thing that matters at first contact, so to speak.

    That’s why the job of medical transcriptionists EVOLVED. Someone else can “electronize” the records AFTER they are complete for other beings with a cerebral cortex to access for the MEDICAL data. In the case of medical records, the retard is never going to be smart enough to RULE the lives of the cerebral cortexes involved in mano et mano HEALTH CARE. He will only force everyone else to be a retard – “…are you sure that is the bp?…”.

    There is SUCH A HUGE LIMIT to the role the computer (retard) will play in the future of medical research, my unasked for recommendation about where the next “gold rush” might be is this – go figure out what happened to the Dreamliner….”inwest” in the retard’s education where it just needs to know when to turn on the light.

  9. The “monetization” going on within the predatory monkey brain *idea* of “macro-economics” is meant to replace PERSONALITY (traditionally recognized as “soul”) and the superior cerebral cortex (loosely comparable to the computer) of the human species with the retard technology of Lady GIGO – garbage in, garbage out. The perfecting of greed as the motive, no less.

    I can’t be the only one to ask, “gee, what could go wrong with that?”, can I?

  10. Larry Ellison bought himself an island based on his marketing prowess. Oracle software is crap.

  11. Somehow I accidentally put this in the wrong comment thread, so I’m copying and pasting my own comments agin, this time in the correct post thread (the blog hosts may delete the facsimile in the prior thread if they wish):

    Software companies are rather different on multiple levels. This site (as this is something James Kwak has high degree of knowledge on) has commented often on the likes of Groupon and Facebook, and it seems there is a certain level of dishonesty with shareholders that comes from these companies. I think similar to the banking industry there seems to be a certain shroud of mystery involved which lends itself to corruption, or at minimum, a degree of misrepresentation.

    I have invested in software companies more than once and done amazing well (probably more out of walking a$$ backwards into shear luck than any real competency on my part). Certain things again lend themselves to lies, for example there is more “leeway” given in accounting standards as to how software companies “capitalize” expenses as the software is being “developed”. This is one thing I know I keep a strong peripheral eye on when I invest in software companies. I “advise” (though this is not investment advice) others who may be thinking of investment in software industry to do the same. Also “Goodwill” (as an accounting term) is something very hard to measure quantitatively in software industry.

    Lastly, does the company put any thought into customer service AFTER the sale?? This is strongly true in software and technology fields in general that they pay “lip service” to customer service, but very rarely give a flying crap after they’ve gotten their money from the sale. If you find a software company that makes a tangible and concrete effort on customer service AFTER the sale more often times than not you’ve found a long-term winner.

  12. Well, folks, in case you’ve ever wondered whether the TBTF banks in a “free market” which is supposed to be COMPETITIVE between the participants (as “opposed to” say socialism, where a few bigwigs at the top get preferential treatment), ever get FED (no pun intended) preferential info which gives them a huge advantage over individual investors (you know those few who make their own decisions and aren’t led around by the nose in ETFs and mutual funds where they get sodomized for fees for “active” management)….your answer is here:

    What is “active” fund management?? That’s like when your 401kplanner—usually an insurance company (say for example Met Life), gets KICKBACKS for offering you 15 equally crappy choices in your 401k Plan. That’s “active” management. And, for doing you the “favor” of “active” management (getting KICKBACKS for 15–20 crappy fund/ETF choices from those same fund families), they then charge you 2% for the favor of sodomizing you when you could have done better if you did your own homework.

    Anyway……. , back to my original reason for commenting….if you’ve ever wondered whether TBTF banks get Fed preferential info, the answer is YES.

  13. You might also wonder the following question: Did Geithner’s FEEDING this info to TBTF banks give Geithner any LEG UP when the choice of Treasury Secretary was made???

    Read the following conversation taken verbatim from Nelson Schwartz’ NYT story, and decide for yourself. It’s important to remember this conversation happened IN THE SUMMER OF YEAR 2007:

    “Fed officials are not supposed to provide details of monetary policy decisions to market participants before they are made public. Bankers could profit from the information.

    After Mr. Geithner reported to the group that a range of ‘market participants’ had raised questions about the Fed’s practices on the discount window, he added that ‘they obviously don’t have any idea that we’re contemplating a change in policy.’

    Mr. Lacker asked the group if he could follow up, and said: ‘I spoke with Ken Lewis, president and C.E.O. of Bank of America, this afternoon, and he said that he appreciated what Tim Geithner was arranging by way of changes in the discount facility. So my information is different from that.’

    Mr. Geithner remained cool and reiterated that he had not shared private information of what the Fed had planned.

    ‘Well, I cannot speak for Ken Lewis, but I think they have sought to see whether they could understand a little more clearly the scope of their rights and our current policy with respect to the window,’ Mr. Geithner said.

    Asked to comment on Friday, Mr. Lacker, who continues to head the Richmond Fed, reaffirmed his account. ‘My understanding was that President Geithner had discussed a reduction in the discount rate with these banks in connection with these initiatives,’ he said.”

    Here is Nelson Schwartz’ original story:


  14. Not at all surprising, I recall Paulson holding the lame stream congress hostage until they relented, and then he walked out the door with a cool $300 million and turned the reigns over to the timster and the bernankster. Some people are not as self taught as they assume themselves to be, especially when the info is NOT as available to the public.

  15. OK, I’ve gone through all the comments. I’ve worked (and am working) in solution development in several vertical markets over 25 years. Here’s the beginning and end of all this analysis.

    1. Academics, purists, technological evangelicals (effectively lobbyists/marketeers for technology companies with funded product campaigns or Intellectual Property warehouses wanting to generate demand for and leverage their IP), and pundits/media outlets starving to fill out magazines and web sites produce materials claiming to be able to improve some general business vertical or horizontal solution. These folks produce papers, articles, interviews, and do the tech conferences, speaking and cheerleading.

    2. As the interest level builds up enough, other niche providers who feel they can cash in on the marketing jihad against the stuff that works today (or that has just been recently implemented and is operating to some degree for businesses) arrive on the scene, flush with venture capital or with IP/algorithms generated at Universities or think tanks. Since the groups in Step 1 are generally more interested in making money as pundits or maintaining/increasing credibility of their prognosticating shed (e.g. Gartner Group), this is the critical piece in the puzzle. It’s rare that the vendors (actually have something tangible for sale) have ALL the parts of the solution available. The key tipoff in their material is that they “enable” the jihad. They don’t do it all. So here, enter all the other ‘playas’…. lookin to get rich – fast.

    3. In general, CIOs and IT Directors are generally as clueless and naive as any other executives – they just happen to have been Peter Principled to higher places. Exceptions abound, but they are not the rule. So those not realizing how this cycle works, move toward the shining city on the hill to get more information on these amazing solutions. They are told in elaborate briefings at conferences, business events, or at vendor locations about how the solutions coming forward will save x-fold dollars based on “Total Cost of Ownership” studies done by entities paid by the very same folks who will profit by the sale of the solutions (but the studies are in color – credible and trustworthy!!).

    4. Underlings are then commissioned to see how much they could save once the execs get back to home office. Sensing that the exec wants to see the answer, “we can save BUNCHES!,” and hoping to be able to further/maintain their careers by leading project, the underlings generate the expected answer.

    5. CIOs sell the plan to executives. Everyone excitedly signs on because slashing costs is always good and always reliable. CIO is eventually tasked to get it done. They go full bore. Quotes come in and jaws hit the floor. Turns out that “value pricing” has come in to crush the preposterous savings estimates. How? The vendors who fill in the niche holes (e.g. the UDDI vendors making databases for SOA service registration and discovery slap together some tools, patent concepts, and now a relatively simple database that took $250K to initially build costs $450k just for one instantiation – plus Oracle license fees, plus data replication features, plus support, plus installation at $250/hr, etc… and you “need” to have one in each of your office locations…) had been spending as much, if not more, researching how much customers will save over 5-10 years. In general, they semi-collusively (no purposeful, direct contact is needed; if you’ve been in the industry you know it happens innocently, but it happens) price their solutions in such a way that all those long term customer savings are what customers will pay up front for their products.

    6. Now the CIO is between rock and hard place. It’s very hard to go back and say, “Well, turns out it costs more than we thought,” without looking a fool. So, a large number hunker down and go for it. They’ll ask for pieces to be left out. They’ll see what pieces they can implement on their own. They’ll find out quickly that the tools for sale are shoddy, full of absurd dependencies, or perform below expectations – due to outsourcing, bad architectures, bad programming tools, and other reasons, but rationalized by vendors as “innovation is hard! don’t worry, we’ll get it working!” In short, they’ll try to find ways to get the project down to their initially projected cost estimates – and do so at the cost of capability. Or they blow their budget to hell and get no actual payback – ever. So no matter which way it goes, projects don’t produce results.

    At the end of the day, it’s a massive loss for the customer. Anyone who’s been on the front line for something like an SAP deployment knows. I remember a few years ago that the running joke was to take an SAP quote, double the price and quadruple the schedule time in order to accurately predict the project. Not sure what the multipliers are today – and it’s not just SAP… they’re just one of many, many who can do whatever they want and companies use them anyway. I GUARANTEE that this is the same effect you’re seeing with these health vendors. Guaran-freaking-tee it.

  16. @timmo, but, but…according to Obama, IT purchases will “….remake our government, revamp the tax code, reform our schools, and empower our citizens….”

    Prez said, in part, “…We understand that outworn programs are inadequate to the needs of our time. We must harness new ideas and technology to remake our government, revamp our tax code, reform our schools, and empower our citizens with the skills they need to work harder, learn more, and reach higher….”.


  17. Can we save money on this? Well that’s hard to say because frankly the military already has electronic medical records so some of the government already is using it. However, the issue at hand is how do we get ALL of these companies to a agree on the same system.

    Hardly ANY company has the same standards in IT. C is the backbone of much of computer science. C++ came from it, Java came from it, python is kinda related, java script maybe.. ok. So microsoft made C#. C# looks 95% like C++. So why they’d make C#? Well because…it’s theirs. Meanwhile IBM deals with Linux and Apple deals with iOS and OSX.

    Ok that’s for the consumer…what about medical.

    Well there’s Meditech out of Dedham Mass. It’s a private company so your guess is as good as mine as to how well they do. They have a counterpart out of the midwest called Epic..yup same thing.

    Ok so how many medical software companies can there *REALLY* be?

    Try over 100!

    How can anyone make 100 companies agree to the same standards? We’re not talking about a style of a form or a cover sheet on a TPS report. We’re talking about connectivity of software platforms here. NONE of these companies are open sourced for a good reason.

  18. @Annie…. Not sure I see anything inconsistent with what I said there. I see you’ve got tongue in cheek there, but the general impact of the process I outlined is virtually universal – even someone who’s generally considered to be of high intellect reflexively bows at the same altar. Seems to me that the only people who aren’t prey to the pervasiveness of the “applying more technology makes _______ better” are some of us inside the beast.

  19. @timmo – my post was just an “amen” to yours since I agreed with everything and thank you for posting “just the facts” from an insider.

    Every once in a while, the comments section here on BS from “insiders” are PSAs – Public Service Announcements. Remember those days?

    Even the wikipedia write up about what a PSA *IS*, is still in dispute among the contributers. Ironic that the internet did not carry on with the tradition of providing information for the health and safety of all its citizens. I guess The Weather Channel still does the most PSAs :-)

  20. @Matt – In other “developed” countries, the individual human being is the holder of their “electronic records” on a flash drive that they carry with them.

    Too simple?

  21. I impressed, I must say. In fact hardly ever do I encounter a blog that each educative and entertaining, and let me let you know, you’ve got hit the nail on the head. Your concept is outstanding; the difficulty is something that not enough people are talking intelligently about. I am very glad that I stumbled across this in my search for something regarding this

Comments are closed.