No You Can’t

By James Kwak

Yesterday the Obama administration announced that “will work smoothly for the vast majority of users.” Presumably they intended this as some sort of victory announcement after their self-imposed deadline of December 1 to fix the many problems uncovered when the site went live two months ago. But anyone who knows anything about software knows that it’s not enough to “work smoothly” for the “vast majority” of users.

Apparently pages are now loading incorrectly less than 1 percent of the time. Well, how much less? Pages failing 1 percent of the time make for a terrible web experience, especially for a web site where you have to travel through a long sequence of pages. There is evident fear that the current site will not be able to handle any type of significant load, like it will get around the deadline to sign up for policies beginning on January 1. And we know that “the back office systems, the accounting systems, [and] the payment systems”—in other words, the hard stuff—are still a work in progress.

None of this should come as any surprise—except to the politicians, bureaucrats, and campaign officials who run The single biggest mistake in the software business is thinking that if you throw resources at a problem and work really, really hard and put lots of pressure on people, you can complete a project by some arbitrary date (like December 1). It’s not like staying up all night to write a paper in college. This isn’t just a mistake made by people like the president of the United States. It’s made routinely by people in the software business, whether CEOs of software companies who made their way up through the sales ranks, or CIOs of big companies who made their way up as middle managers. You can’t double the number of people and cut the time in half. And just saying something is really, really important won’t make it go any faster or better.

Clearly all sorts of things were wrong before October 1 (and not just because they were relying on Oracle to do something other than supply a database). According to the Times, the website “had barely been tested before it went live,” which is a sure recipe for disaster. Back in my day, every feature was supposed to be finished three months before release. I know web companies do things differently today, but when it comes to performance they already know they can handle the load, and I doubt they cut corners when it comes to software that handles financial transactions. If you don’t have time to test, you shouldn’t ship. It’s that simple. Anything else is just wishful thinking.

It seems like had at least two huge problems at launch. The first was performance—the ability of the system to deliver pages quickly when under load. I don’t have any insider information, but from the outside it sounds like a lot of what they are doing is switching hardware around, increasing the bandwidth at certain key chokepoints, and firing their hosting company. That’s all good, but performance is only secondarily a hardware issue. The software has to be designed properly to be scalable—so that adding twice the hardware will allow it to support twice as many users. If not, you need to scrap it and start from scratch. I can’t tell from the outside (and I couldn’t even tell from the inside) if it’s designed properly in this case, but I sure hope so, because otherwise no amount of hardware shuffling will do the trick.

The other problem was data integrity. When you’re dealing with financial transactions, it’s really, really important that the data don’t get messed up between the two counterparties. But it seems like, at a minimum, customer records weren’t making it through to the insurers. It sounds like fixing that mess has been deferred until later. It could be as simple a problem as bad data mapping between one data model and another. But fixing these problems involves another software quality issue. With high-quality code, it’s relatively easy to find and fix these errors. With bad code, it’s hard to find bugs and it’s harder to fix them without destabilizing the rest of the system. Again, let’s hope for the former.

I’m not technically skilled enough to be the type of person you would want making decisions about this mess, and I don’t know anything more than you can read in the newspaper. But when custom software projects go this badly, I think that in general (meaning more than half the time) you are better off cutting your losses and starting over. Obviously there are administrative and political reasons why the Obama administration can’t do that. We know that this project has to succeed like few other projects in history, and it will get there one way or another. But there’s no magic bullet, and neither hope nor trying harder is a viable strategy.

63 thoughts on “No You Can’t

  1. ‘It seems like had at least two huge problems at launch.’

    It had ONE insurmountable problem; it was attempting to provide a product without being subject to market discipline. I.e. it wasn’t a profit seeking enterprise.

    Governments that fail don’t go out of business and make way for the successful in their wake. Instead, they double down on their numbskullery (and tell the customers to use the Thanksgiving dinner table to hold criticism sessions of the saboteurs and wreckers who stand din the way of the brave new world).

    Markets handle problems like health care, if they’re allowed to. Even those scary derivatives can be taken care of;

    ‘Figure 1 shows that, although aggregate daily volume in the UK CDS market dropped during our sample period from about €2 billion in 2007–08 to about €1 billion after 2009, the market never really shut down. Strikingly, volumes shot up in the wake of Lehman’s default; this is surpising given that Lehman Brothers was a major dealer in the CDS market. It thus appears that the market held up well against concerns about counterparty risk. Figure 2 plots the dealer buy, sell, and net volume in relation to all other market participants. Consistent with the pattern of aggregate volumes, Figure 2 shows that dealers continued to intermediate between ultimate CDS buyers and sellers throughout and at the peak of the financial crisis.’

  2. Politically, Obama didn’t have the choice of delaying the roll out of, unless you think that it was a coincidence that the government shutdown was happening at the same time. Any delay would have fed the Republican frenzy and made governing more difficult. Taking a few lumps for a poorly designed website was the more feasible option. And now, 2 months later he has 100,000+ facts on the ground that make any repeal much, much harder.

  3. Really, the correct solution would be to throw out their site, and instead install one of the several working state sites. In fact, writing anything in the first place was a mistake; they should have just gotten Massachusetts’ code, which had already been stable and reliable for years when the ACA passed. Step 1 in any software project: ask yourself, “Does the software I’m about to write already exist?” If so, why aren’t you just using it? If there’s a good reason not to just use it, is there a good reason not to join the existing project, or at least fork the existing code?

  4. “We know that this project has to succeed like few other projects in history, and it will get there one way or another.”

    I think that is liberal speak for, ” I know better than you do what to do with your money.”

    The results really speak for themselves. Wouldn’t all of this money have been better spent by the folks that earned it instead of wasting it on software that doesn’t provide anything useful itself?

  5. On the news today, I heard a reporter recount his experience of trying to log on to the site. After leaving a bunch of personal data information (created by the network for use as a “trial prospective enrollee”), he got a message telling him the site was not accessible at this time, and asking for his email address so that the site administrator could send him an email notification when he could try to log on again. After four hours, he still had not received the notification.

    Sounds like the site was hacked. If so, a hacker could intrude and do damage not only to the enrollee’s email account, but also use the personal data for many variations of identity theft. If the promised notification was not received, the enrollee would not have a clue as to who or what received or used his personal data.

  6. Why did they start from scratch in the first place? I’ve been in the software business for more than 30 years, and I never start from scratch unless I absolutely have to. Why so many different and costly implementations? Why didn’t they take the Massachusetts system and modify it?

  7. Throw it out and start from scratch is pretty much the worst possible thing you can ever do in any programming enterprise ever.

  8. Nov 3, 2012 … The Obama administration is relying heavily on outside contractors to … UnitedHealth Group’s acquisition has caught some select attention of the Senate:
    “If an insurance company had influence over the information technology architecture used to run the exchange, it could interpret federal standards in a way to exclude competitors or make it more difficult for them to win approval, say some insurance experts. Or it could have an inside track on knowing how to design plans that meet the standards.

    QSSI, contractor chosen to fix, faced questions from lawmakers last year
    By Jia Lynn Yang, Published: October 25 E-mail the writer
    The company chosen by the Obama administration to oversee the repair of the new federal health insurance Web site faced questions from lawmakers a year ago about whether it was an appropriate choice for earlier work on the site, given that it is owned by the country’s largest health insurance company.
    Columbia-based Quality Software Services Inc. — known as QSSI — was purchased by ­United­Health Group in September 2012, months after it was picked by the Department of Health and Human Services to help set up the Affordable Care Act Web site. That called for work on three areas: build a data hub for the site so that information could be transferred between different groups, deliver a tool to help users register, and do some testing of the technology.
    UnitedHealth Spurns Obama Exchanges as Rules Stall Profit
    By Alex Nussbaum – May 31, 2013 9:48 AM ET
    “UnitedHealth Group Inc. has trimmed its plans for selling to the uninsured under President Barack Obama’s health-care overhaul, in the latest sign large insurers see little gain from quickly plunging into the new market. Photographer: Michael Nagle/Bloomberg”

  9. “In Blow To Religious Conservatives, SCOTUS Refuses To Hear Obamacare Lawsuit” – December 2, 2013

    “Single Payer” aka “Medicare for All” – The president cares more about what the Republicans have to say than his politcal base that had voted for him (twice), go figure, and then report back your findings.

  10. as for fixing the site. they haven’t replaced the network support contractor just yet. unless they just pulled it out and changed that out but have to keep paying the old contractor till the contract ends. might be the best choice to at least get the web pages to be display consistently. as far as design, it depends (as it always does). did they set the servers so that each states has its own instance of the code that backs the site up? if not, why not? that would at least make it so that as states declined to participate, then you could add instances for those states, and that would provide scalability to the code. otherwise you have to have a really expandable environment. of which there aren’t many. and of course. the development stage of this code, was probably under the same constraints as any other large project, that is being run by many contractors. seems like no matter who is running it, they seem to think that adding staff late in the project will solve the deadline problem. it wont as it takes time for new staff to get up to speed, and guess who has to make that happen? the original team. so for some reason that never works out right. the back end problems (i.e. communicating with the insurance companies). sounds like a problem of not just the code but testing with them. it will be hard to tell if is the originator or receiver of the data as to whom didnt do their part. or both. and since the insurers probably didnt get paid to create their code, and highly unlikely they even tested it, it wouldnt surprise me if they have major issues with it. and just because you didnt get an email from the site in a few hours because its busy, doesnt mean its been hacked. unless you think that happens at every other time you interact with any company and that happens too? cause it does. a lot. and i dont really know why they created a new application, after all they had CMS to start with. all they were needing was more interactions with others (the IRS, insurance companies).

  11. “…Government = TBTF….”

    Ah, Paddy, me laddy, ease up on the Egg Nog…

    The human species IS the “government” – “We the People”…the debate is always about what “We The People” decide based on our ideals, ethics, and experience with what does NOT work.

    No one is going to follow a Nihilist into a cockamani utopia that, basically, is hell on earth….

    These ad hoc skirmishes launched with psychobabble and the shifting sands definition of 50 lalaland political words is getting ridiculous – we need a “JUST WAR”…

    See you on the other side of the line up :-)

    I’ve knocked so many drunks on their arse, mano et mano, that it will be fun using a “drone” made in the garage out of spare parts for a change – LOL…

  12. I picked one out the garbage that did some damage a while back, Booker T Word was his name, slight of hand was his game.

  13. The Affordable Care Act …The Affordable Care Act…The Affordable Care Act…The Affordable Care Act….The Affordable Care Act ….
    The Affordable Care Act …The Affordable Care Act…The Affordable Care Act…The Affordable Care Act….The Affordable Care Act ….
    The Affordable Care Act …The Affordable Care Act…The Affordable Care Act…The Affordable Care Act….The Affordable Care Act ….The Affordable Care Act … he Affordable Care Act …he Affordable Care Act …e Affordable Care Ac…e Affordable Care Ac… Affordable Care A… Affordable Care A… Affordable Care A…fordable Care …fordable Care …fordable Care …ordable Care …ordable Care …ordable care …

  14. Does it really matter how long it takes for the site to get operational? In the short-term I suppose and for the news cycle, and some “politics” if you still think there is a difference. I’m more interested in the premiums that are going to be charged and the deductibles that are going to apply to the insurance policies. There wasn’t very much focus on the cost that’s going to be charged to the end user of these products.

  15. @Paddy – One more time – the banksters are GUILTY of targeted economic genocide – openly admitted to as “sanctions” on Sunday talk shows – “…the sanctions are finally hurting the upper middle class in Iran…”.

    Deregulated “financialization” is a “modern means of destruction” – point 4 of Just War criteria:

    The just war doctrine of the Catholic Church – sometimes mistaken as a “just war theory”[15][16] – found in the Catechism of the Catholic Church, in paragraph 2309, lists four strict conditions for “legitimate defense by military force”:[17]
    1. the damage inflicted by the aggressor on the nation or community of nations must be lasting, grave, and certain;
    2. all other means of putting an end to it must have been shown to be impractical or ineffective;
    3. there must be serious prospects of success;
    4. the use of arms must not produce evils and disorders graver than the evil to be eliminated (the power of modem means of destruction weighs very heavily in evaluating this condition).

    Funny how two of Detroit’s most infamous “sons” – Prince of Blackwater and the Romneys blame the “Unions” for the bankruptcy of Detroit. Seriously, talk to the hand….

    Well, here’s how the rest of the world acts upon information about “genocide”, might help, Paddy, if your brown nose got some fresh air:

  16. O Christmas Tree,
    How steadfast are
    your branches!
    Your boughs are green
    in summer’s clime
    And through the snows
    of wintertime.
    O Christmas Tree,
    O Christmas Tree,
    How steadfast are
    your branches!

    O Christmas Tree,
    O Christmas Tree,
    What happiness befalls me
    When oft at
    joyous Christmas-time
    Your form inspires
    my song and rhyme.
    O Christmas Tree,
    O Christmas Tree,
    What happiness befalls me

    O Christmas Tree,
    O Christmas Tree,
    Your boughs can
    teach a lesson
    That constant faith
    and hope sublime
    Lend strength and
    comfort through all time.
    O Christmas Tree,
    O Christmas Tree,
    Your boughs can
    teach a lesson

  17. Unspoken in all the chari vari of the Affordable Healthcare Acts systemic and continuing failures – is the more terrible fact that a law written by the insurance, pharmaceutical, and healthcare provider industrial complexes was never intended to succeed in the sense of providing better cheaper healthcare. However unforgivable the failings of the IT application process, – the real unspoken horrorshow reality of this fascist law are is a greater evil. How can the government legally impose – force such an enormous cost burden on citizens already battered with rampant unemployment, stagnant or declining wages, and the widest divide between thehaves and thehavenots in our history? Even if we could afford healthcare (and many of us cannot) – we don’t even get decent healthcare no matter what the cost – since Amerika has the most costly and most dysfunctional healthcare in the socalled first world. Cuban’s are better off. So are the populations of 34 other nations. Naysayers and deniers and wingnut patriots can do your own googles and get back to me.

    I have not been to a doctor in twenty five years. The system gives me nothing, nor has anything (not one penny) extracted from the system by mein more than two decades! I owe and expect nothing from the mcmedicine joke that is the Amerikan socalled healthcare system. No one can or will ever force me into a system I cannot dream of affording, and that worse – provides shoddy low level unhealthy healthcare that only benefits the insurance, pharmaceutical, and healthcare provider industrial complexes.

    Obama ran on a singlepayer system not the fascist giveaway and wanton profiteering joke that is socalled Obamacare!

    The MSM and everyone here is focused on the wrong evil and irrelevant failures!

    Burn it all down! Reset!! It’s the only option for the 99%.

  18. Oh high priest of the catholic church lady, I pray and bow wow to thee like the dog you and your lost war on drugs is.
    Don’t click that link folks, she’s a hack, wanting to infect your computer and cost you money, so she can find some more maintenance powder to pick up her lame game.

  19. tonyforesta: I agree with you but must remain focused upon the infrastructure of what this holds for the future. In ten (short) years [just look back to years to see how quick) consider what this (false flag?) orchestration will entail under the heading of cost necessity. COST, of course, means medical/expense ratios in the process of supporting this privateer medical-insurance industrial complex. In the future, sustaining the “system” will have total blinding precedent over sustaining survival demands of public health demographics. It is true that many people are being sold the idea that they now have coverage plans…but they are not saying they are directly getting medical care itself in any quality assurance manner. The reality is that they have “coverage” and coverage only creates a new reservoir or money pool for a revenue streaming / scheming future. An artificial pool of money that will be always in debt and always overdrawn and perennially in need of re-finance strategies…and always in crisis mode…BECAUSE so many deadbeats are TAXING IT (will be the battle cry). When in fact, the taxing will be on the backs of poverty stricken “austerity” populations under capture economies of scale.

    And I ask you what is the difference between tax mandated state captured finance markets to support socially government supported and institutionalized class structures…, and command economy?

    Universal Health care is the only remedy from this stealth for wealth scheme! America can not only adopt one from models in Canada or elsewhere that already exist, or it could devise the best system in the world and lead the way for global health as humane necessity.
    That is…unless we get hooked into this trap!

  20. while i see that we are no longer talking about what this post was about. but i too think that universal health care is the real solution, that ACA is just a small step in that direction. when the majority of companies (and they are all considering it) drop health insurance as a benefit is when the real movement to fix this will happen, not before

  21. YES YOU CAN !!
    “No politician wants to mention “redistribution” because it conjures up images of worthy “makers” forced to hand over hard-earned income to undeserving “takers.”
    Article by Robert Reich Posted: 12/05/2013 2:00 pm
    One Answer to Low-Wage Work: Redistributing the Gains


  22. The first commenter, above, says “Markets handle problems like health care”. That is the funniest thing I’ve ever read on this site. Thank you!

  23. The solution should have been to purchase working, production-grade software from an experienced vendor. There are several in healthcare that develop software for payers (HMOs chiefly) that assist with the enrollment, and others that (in a less mature state) have been written for consumer-driven healtchcare which shares quite a bit of functional commonality with Obamacare. As another poster correctly pointed out, there’s the Massachusetts code that could have been licensed or purchased, and I’m pretty sure is was fairly scalable.

    Most sane managers in the private sector wouldn’t have attempted to re-invent the wheel, which is something that always raises my ire when I’ve interviewed (or worked with) defense contractor or former government computer programmers in the past. From afar, that looks exactly what the government was attempting, after it staffed the design and development leaders with politicos and insular career bureaucrats.

    Oh well. Writing the software from scratch is pure insanity. Releasing such software into production without stress tests was pure public sector stupidity.

  24. really? you really dont think the private sector wouldnt have done that? of course they would. never seemed to be a whole lot different than what the private sector would do and has. same bureaucrats and politicos. same deal,just different names for the same thing. but i could see using the MASS code, but then just like the private sector, not invented here, we wont use it (then again no telling (If mass was willing to sell or license). might or might not be scalable. but you might get around that by having separate instances for each state, and it does look like they did little stress testing, then again, they did hire an ‘experienced’ contractor to do the work. and its not like the private to not have done that too. and get just as bad if not worse results. seems like a wall street company installed a change to their mainline software. and ended up almost taking down the company (if it didnt). but then management is usually more engrossed with creating the software on time. whether it works seems to always be an after thought. saw that with many private companies. they just dont usually get the PR that this web site did

  25. Well food stamps work out pretty good, perhaps health stamps would be even better. Health stamps would be accountable and could only be utilized in one direction. Unlike food stamps, they would not have much cash-in value since they can only ultimately be used for medical expenses. Since we are daydreaming…why hasn’t the idea ever been considered?

  26. It might have been included in the original republican single payer plan, but the democratic “majority” voted it out lock, stock, and barrel. A fine idea no doubt, if not used toward health issues after 30 years, get a discount on your good driving insurance plan and save even more.

  27. my bittys are in quarks…..

    The ‘official’ page of Bill Still shared a link.
    December 8
    Still Report # 141

  28. What is “minimum wage” in Bitcoin? Right, “labor” will never be paid in Bitcoins…

    Yup, the inalienable right of every Human Being to make their lives LESS MISERABLE through honest work has no software, algorithm, technobabble equivalent, so “solly charlie”

  29. Yes You Can !!

    Colin Powell calls for universal health care in U.S. by ericlewis0
    …………………………………….say again?
    Fri Dec 06, 2013 at 02:37 PM PST
    Colin Powell Calls for Universal Health Care in U.S.!!!
    by ericlewis0

    “I am not an expert in health care, or Obamacare, or the Affordable Care Act, or however you choose to describe it, but I do know this: I have benefited from that kind of universal health care in my 55 years of public life,” Powell said. “And I don’t see why we can’t do what Europe is doing, what Canada is doing, what Korea is doing, what all these other places are doing.”

  30. “Bitcoin is not an official currency. That said, most jurisdictions still require you to pay income, sales, payroll, and capital gains taxes on anything that has value, including Bitcoin.”

  31. dw:

    I’ve worked in both sectors, and have sold to both sectors. If you firmly believe that the two sectors optimize along the priorities, then there is nothing to discuss. One of my professors at Yale (who also even more extensive experience in the public sector) used to maintain that the private sector optimized along quantitative metrics like ROI or EVA, while the public sector optimized along subjective qualitative factors (making internal factions happy). Now you might be more experienced or clever than my old Yale prof, but I’ll just side with my experience and his wisdom.

  32. hm/. not really sure what you meant. all i really pointed out is that both have politics them, neither isnt subject to the same failures. and private sector doesnt always use ROI or any other metrics. unless you haven’t noticed that they can do some really stupid projects, that will never ever create a ROI. but they have the correct buzzwords, or are the latest greatest what ever. never mind that they make no financial sense. or end up failing in a big way. examples are Apples map app, that wall street broker that almost killed their company because of a mistake they made in their software. and when did the private sector not want to keep their internal fractions happy. they just aren’t the same sort of fractions. try changing an application that serves 2 or more departments. you will have to get buy in from them all. and some that dont even use the application will need to be convinced that the change is ok. have i seen that in the private sector? yes. in the public sector? yes. but then again i was working in the companies and seeing from an internal point of view. not from the outside

  33. >>anyone who knows anything about software knows…
    I’ve been writing software for decades. Sometimes a 99% success rate is fine. It depends, in part, on your dependencies knowing you have no control over them. Sometimes that’s hardware or bandwidth but sometimes it’s waiting on other sites, like for example HI company servers in the exchange to respond to queries. If the solution to a failed page load is to “refresh”, so refresh and move on.

    More interesting excerpts fro this article…
    >> I can’t tell from the outside…
    >> It sounds like..
    Not exactly confidence building admissions when it comes to having faith in your critiques

    And the most revealing…
    >> I’m not technically skilled enough to be the type of person you would want making decisions about this mess, and I don’t know anything more than you can read in the newspaper.
    That pretty much says it all.

    Just throwing stones at this thing from the outside may be fun, but it really doesn’t help. The rarely spoken fact is that the default position, if the ACA was never passed, is “do nothing, leave it as it was before”. That wasn’t exactly a well functioning system if you recall. At the very least, I salute the admin for pushing this as a means to at least try something. “Repeal and Replace” is fine as long as you have something better to replace. I’ve yet to see anything like that from a unified opposition.

    >> Governments that fail don’t go out of business and make way for the successful in their wake.
    Sure they do. Every 2-4-6 years we can, and often do, vote them out and vote in new governments. Happens all the time.

    >> It had ONE insurmountable problem; it was attempting to provide a product without being subject to market discipline.
    You mean market discipline like that which was controlling the financial sector in the times leading up tp 08? Unbridled, “Market discipline” is a bull in a china store.

  34. Data: EXography
    EXography: States say most uninsured still won’t be covered in 2014
    Richard Pollock
    Senior Watchdog Reporter
    The Washington Examiner

  35. “….If they are breathing, they are lying….”

    Anyone think that the websites handling job applications is operating any differently than the “healthcare” sites….?

  36. “Pre-meditated” should weigh in on the scales of JUSTICE when it comes to THEFT:


    They used “modern” technology – which makes it even WORSE. They embezzled MY TAX DOLLARS to rob me via a “spy apparatus”.

    No one is going to roll over and get over THAT….that would be insane behaviour – people are either duress or drugged to the point of life-threatening to put up with such flagrant disrespect for thousands of years of experiential LAW and ORDER!

    You know where you can shove your gizmos….

    Oh, yeah, Merry Christmas Global War Drug and Slave Lords :-))

    Who’s ur Daddy….?

  37. Mapping between formats isn’t the only data issue. The quality of the data will vary from database to database. Some records will contain errors that are of no consequence to the host database, but that can generate false negatives (mostly) when attempts are made to match them with records from another database. At, users enter their data. The website then attempt to verify those data by querying other databases, and to calculate subsidies, etc. When that process is completed, the user’s record must be translated to a format understood by the insurance company’s software. There are a lot of places were data anomalies can be reconciled only through human intervention.

    We should have had a debate over the administrative practicality of the ACA, and would have had that debate had single-payer been a viable alternative. But single-payer was not seriously considered, so the only debate was over the Rube Goldberg economics of the ACA. Obama’s people simply assumed that whatever his economists conjured up, no matter how complex, his in-house IT operation could translate into a seamless website in the snap of a finger.

    Most of this will get fixed, but is never going to run all that smoothly. And given the love policy wonks have for gratuitous complexity, we’re never going to have a useful debate over administrative practicalities in a situation like this.

  38. “….We should have had a debate over the administrative practicality of the ACA, and would have had that debate had single-payer been a viable alternative…..”

    If double entry bookkeeping were still employed, the increase in “worker productivity” would have easily covered the costs of a “single-payer” administration of “health care”.

Comments are closed.