> In Slavitt's defense, data security may not have been an explicit feature of QSSI's federal contract
Coming from the construction industry, there isn't a contractor in the world that cares about things which aren't in the contract. Implementing features which the client has not asked for, in the design-bid-build world of public-sector contracting, just means you will lose the bid to a lower bidder who won't do those extras.
"Coming from the construction industry, there isn't a contractor in the world that cares about things which aren't in the contract."
In the construction industry, a contract might not explicitly say that the building has to comply with local building, fire and electrical codes - that's taken as a given. To construct a web site that deals with sensitive information without taking security into account is like building a skyscraper with no fire exits - not something that any honest or competent builder would do.
Yes, but those are actual codes. In fact, that's why they're codes. Contractors won't do more than they're legally required to (nor should they be expected to), so safety features were made legal requirements.
Relying on the honesty or competency of your contractors go above & beyond their requirements is an inefficient and unsustainable solution in a free market. This is a clear case of Moral Hazard: In the same way that bad money drives out good, corner-cutting organizations out-compete honest ones.
What about reputation? I regularly do more than the legal minimum for my clients and there are companies who do the same for me. Meeting the legal minimums only makes sense for commodities, e.g. this salt is 99.5% pure. This software is 99.5% "secure"? This software is 99.5% "complete"?
It seems to me that reputation works better in the free market than the government "market". I don't know why that is, but I suspect it has to do with the same rules for the bidding process that are meant to decrease corruption, but also tend to entrench contractors.
True, but quantifiable aspects of reputation (like having successfully completed similar projects in the past) can be part of a pre-bid qualification process. This is common for large or complex projects.
> In the construction industry, a contract might not explicitly say that the building has to comply with local building, fire and electrical codes
In the public construction industry, every contract includes that language. It's superfluous, because no contractor that wants to keep his license violates those codes (all construction is inspected by local authorities).
Furthermore, in construction, those codes exist and are easily incorporated into contracts by reference, even non specifically ("all work will comply with local building codes and statutes"). Also, the liability for non-compliance, even if undetected by inspectors, is massive.
In the tech industry, I've seen "work will be performed to professional standards and best practices" and other such meaningless clauses, intended to give the contractee some standing for complaint if they aren't happy later, but it's hard or impossible to measure compliance to generic standards that mostly don't exist.
This is one reason why Engineers say that dorking around on computers isn't Engineering. They do have a point.
> In the construction industry, a contract might not explicitly say that the building has to comply with local building, fire and electrical codes - that's taken as a given.
That's because you have to have a license to perform that work. If you don't meet the code, you don't keep your license.
This is one of the reasons why McConnell et al have argued that we need professional licensure. Licensed Professional Engineers can lose their licenses if they sign off on half-assed plans.
Not "would do" but "could do;" you can't build, honest and competent or not, without local code-compliance. Perhaps the same should be true for the web; perhaps there should be damages for poor security; government or not.
Which is also true in government contracting. What's odd is that security requirements are mandated by law, FISMA, so the government is supposed to require that systems meet the FISMA requirements. You shouldn't be able to get an ATO without it, unless the management of that agency "accepts the risks" (SES levels only, gS personnel can't do this) and waives the vulnerabilities.
So if I had to guess, if they didn't protect the data then it's likely those controls were notin place everyone knew they were not and the agency heads said it was ok. All fed systems get tested, and the system certifier is supposed to be an impartial reporter of the facts (these are the vulns, missing controls, risks, etc. and they are almost always impartial). They report those to the DAA along with a recommendation to approve or not approve the system to operate. Given how important this system was, I highly doubt anyone would not have issued the ATO no matter what vulns it had. And that's not to pick on this system, sadly slap dash rush ATOs happen far too often in the USG.
If you spend a ton of time and money on a system, execs in the USG have their careers on the line to show that wasn't wasted and since they don't understand security it all seems like science fiction to them. They almost always ATO the system. And why not, no USG exec has ever been fired for having a poorly protected system broken into.
eigenvector: Implementing features which the client has not asked for, in the design-bid-build world of public-sector contracting, just means you will lose the bid to a lower bidder who won't do those extras.
growupkids: Which is also true in government contracting.
actually security is not a feature, but a constraint. It doesn't bring the user anything, but it absence would be a problem for secondary reasons (that might be important in the grand scheme of things).
Imagine a bank (well just an account), on the functional level, you just need to deposit and withdraw money. The fact that they have to secure it is a huge constraint, but at the functional level we're not far from a trash can, a parking lot or a gas tank.
Right. A solution is for the government to develop a series of security standards that cover a wide range of IT projects, and each project should declare that "implementation must conform to Security Level X, Category Y"
> A solution is for the government to develop a series of security standards that cover a wide range of IT projects, and each project should declare that "implementation must conform to Security Level X, Category Y"
Well, you'd think so, which is why that's essentially already mandated:
I work in the information security industry--primarily focused on healthcare security--and you'd be amazed and appalled at the lengths that CIOs will go to say that they "don't need to be HIPAA compliant."
More and more, though--especially as high-profile incidents become more widely reported--security is starting to become an accepted norm. There's still not much budget for it, and it's still considered an inconvenience, but at least people recognize that spending some preventative dollars can save a significant amount in reactive responses (DFIR, legal fees, etc.).
The HN crowd might be interested to know that the worst security offenders I've seen (even worse than old legacy systems) are software startups. Why? Slogans like "Ship First," or agile development cycles that don't include QA/security testing as part of a meaningful SDLC. What good is shipping something every two weeks, if one of those releases loses your customer database?
My latest drive is to help organizations realize that security is a feature, not just an inconvenience. Once that idea is more commonly accepted, we'll all be better off.
I might be horribly misreading this, but it reads like they're talking about no specific threat or problem but rather arguing if that was part of the contract or not?
If that's the case then shouldn't lawmakers, being lawmakers, pass laws so that future government projects include a clause in the contractual arrangement? Isn't that just common sense?
Seems like this is a trend with the US government, they try to find others to blame instead of just fixing the damn issue. They had this massive set of hearings about people abusing military disabilities (e.g. claiming for disabilities obtained before joining the US military) but didn't actually, you know, fix the freaking issue -- or even try to. They just berated the people being advantaged from it.
> If that's the case then shouldn't lawmakers, being lawmakers, pass laws so that future government projects include a clause in the contractual arrangement?
They already passed a law creating mandatory security requirements for Federal IT systems, whether contracted out or built in house or any combination. [1]
> they try to find others to blame instead of just fixing the damn issue.
This is for two reasons:
1) Fixing something makes it appear, politically, like it's your fault, or at least savvy opposing politicians can spin it that way.
2) The incentive to fix the issue <= incentive to shift the blame. If one party points their fingers, the other must point back, lest the public think the first party is right. Shifting blame is easy, one can do it without working too hard, and it can fuel public debate which forgets about the original issue.
Many politicians care less about fixing problems and more about getting/staying elected.
So Rep. Mike Rogers is now suddenly concerned with protecting the security of data? The same guy that's been defending everything the NSA has been exposed to have done?
So, I know we are looking for hypocrisy here because gasp hypocrisy, but there really isn't any. You were providing your information to the government, making sure nobody other than the government sees it isn't at conflict with defending the NSA's actions if you see this kind of activity as legitimate.
On one hand he's alarmed that enrollment data may not be properly protected. On the other hand, he's protecting an agency that's requiring technology companies to create back-doors and weaken encryption algorithms for the NSA's convenience. How are those two objectives not in conflict?
A. The government wants all your data available to them.
B. The government does not want all your data available to the public.
A != B
Edit: Not to come off as rude, I just tend to agree.
If you want to say that in that sense, the government is being hypocritical, perhaps that's true. At the same time, though, their goal here is (hopefully) to keep your data safe from outsiders, whereas we can "trust" them to be good with the data they backdoor from companies ;)
Basically, the gov't is allowed to have all your information and see it, and no one else. Enrollment data needs to be protected from everyone but the US gov't is the stance. From this perspective, there is not conflict of interest.
I hope (naively) that one day, software developer will collectively tell the governments that they work for "Sorry, no amount of money will deliver what you're asking for".
Pretty much every government IT projects I've seen fail, that's most of them here in Denmark, has done so because of politics, laws and regulations. The thing is: the big contractors like CGI, aren't going to turn down the money. They are only to happy to deliver a poor product because they are paid millions and never asked to account for the failures of these projects.
It would be better for the industry as a whole to say NO to projects we know will fail. I don't for a second believe that CGI software developers thought that they could make healthcare.gov a massive success, or even just plain decent. If they couldn't see that, then they shouldn't be doing software development in the first place.
Doing a postmortem this way is refreshing, except you'll never get the right answer: "With the time available (not the money) this was never going to work. The integration with legacy system and the shear amount of these integration points makes this an almost impossible project". In reality will get a lot of blame allocation and CGI won't be made accountable for accepting a project that should have be rejected by anyone with half a brain.
Do you think that when CGI signed the contract they realized the integrator, the government's CMS, plus those above, HHS and the White House, would be constantly demanding major changes, right into the week before launch??? Also check the date of the "you must register first, no window shopping" decision, it was quite late and had a very major impact.
That CMS, responsible for integration testing, would delay that until 1-2 weeks before launch (see above change orders problem), see the tests failed, and launched anyway.
Well, I'm actually sure CGI wasn't surprised by the latter, but, still, I can't blame them for any of the above. The White House -> HHS -> CMS, all insufficiently experienced in running a major development project, turned out to be a customer from hell. Heck, given how important this site is, a key part of Obama's legacy, I and many others assumed the Administration would treat it with appropriate seriousness. We were wrong.
>Do you think that when CGI signed the contract they realized the integrator, the government's CMS, plus those
above, HHS and the White House, would be constantly demanding major changes, right into the week before launch???
When the second or third of the changes happens, CGI should have told them "NO, this cannot be done professionally" and hand them their money back.
Well, that's also a decision to get out of the business of Federal contracting, and to lose a lot more of their contracting business.
Did you notice how obsequious CGI's VP Cheryl Campbell was when asked the usual leading questions about the bureaucrats and political appointees running the show? This bit from the hearings is really telling (http://www.washingtonpost.com/politics/house-panel-grills-co...):
"Later, Rep. Leonard Lance (R-N.J.) asked the contractors whether they could conceive of “a more incompetent administrator” than CMS.
“I have no opinion on that,” Campbell replied."
Might be a lie, might be she's been jaded by so many customers from hell she really can't precisely rate CMS....
I'd also note that it wasn't necessairly the 2nd or 3rd change that were fatal, while the late ones clearly were. I'd further note that a lot of us are optimists, believing we can perserve in the face of tough situations including clients from hell, and we can take that too far when things become manifestly impossible.
It's also possible that CGI et. al. believed, certainly hoped, that when the first integration tests in the last 1-2 weeks before the scheduled launch resulted in total failures the administration would admit reality and delay opening a site that they knew was going to fail hard. I mean, this steady dribble of bad news is a zillion times worse than a one time hit in a program that's had a zillion delays and waivers already.
I guess I'm saying I'm not sure CGI could accurately gauge the depths of stupidity of this client prior to the most insane decision to turn on the known failing system.
If anyone saw this briefing, it was mostly a lot of really technologically illiterate people asking pointed questions about a gigantic set of systems that can't easily be answered to technologically illiterate people. At one point, a congressman, with the tone of a disappointed parent, said (paraphrased) "I am very disappointed in you all. have yet to hear any one of you say sorry to the American people."
Someone in the room retorted with "we didn't hear an apology when the government shut down either."
(You'd expect someone to get timeout for this kind of pettiness.)
With that said, one of the obvious problems was blame shift, as this article points out. The guy was (I don't remember this particular interchange) most likely outlining the fact that the question being asked of him wasn't relevant to "his responsibilities" in the contract. However, when responsibilities of a project are distributed, things fall through the gap.
This article is a bit too short to give enough space for the subject, but I think it does point out the most important issue - no one really had full control of the project.
I watched most of it yesterday. It was a fascinating post-mortem of a software project.
What I saw was a small minority of politicians that seemed to have a handle on some of the technological challenges at-hand. Some politicians used their time to score political points completely unrelated to the technology issues - basically "our party is good / your party is bad" idiocy. Some politicians read prepared smart-sounding questions which they wouldn't be able to rephrase in another-way if they were asked to. One politician asked for a copy of the server logs so that Congress could get to the bottom of what was going wrong. And none of the company representatives were willing to accept that they'd done much wrong - instead taking every opportunity to blame people or entities not in the room.
> If anyone saw this briefing, it was mostly a lot of really technologically illiterate people asking pointed questions about a gigantic set of systems that can't easily be answered to technologically illiterate people.
This is, unfortunately, what every discussion among politicians regarding any sort of technology is.
I believe the part about end-to-end testing and HIPA were something that needed to be asked and was not answered in an acceptable manner. There are some technical folks on that committee and they even showed source code.
Totally agree. One of the thoughts that came to my mind is that a public project like this one sh(c)ould be open source right from the beginning. That would solve a lot of quality issues on one hand, and probably the pricing will also be better for the Government/taxpayers.
While I am pretty sure that most congressmen do not get it, but there are some very qualified and awesome technical people who work in the Government. Probably, someone has to have the balls and call it open source one fine day.
My opinion is that anything that the government funds should be open source. Patents that come from government funded research, software projects, electronic voting, should all be open source. If we, the people, paid for it, we should be able to have access to it.
> anything that the government funds should be open source.
Generalization to open source everything that the Government does, or taxpayers fund, might not be such a great idea. There are some projects which are supposed to be secret (Like a NSA that's focused only on anti-socials) because they give our Government an edge over others.
Probably a competent agency with an oversight to decide on open/closed aspect can help, but the subject needs much longer and detailed debate. IMO, healthcare, education, civil investigations are perfect examples where we can go completely open source. Or get disrupted the valley way.
Open source would certainly help in terms of transparency and hopefully quality but bottom line is that it would not have saved this project. This project failed because of incompetent management.
Requirements came in so late that development didn't even start until March of this year, and requirements were changing right up until the week before launch. Only one week of full integration testing was performed.
While open source should be used in government IT, that's not the main problem in government IT. The main problems are poor project management, a lack of accountability, a broken contract award process, and decisions makers who don't understand IT well enough to guide these projects to fruition. Open source would not solve any of those problems.
Big military projects have the additional problem of congressional interference. Congressmen regularly force military acquisitions that the military may not even want or need in order to funnel dollars to their home districts. So it might make more sense to build all of the components of something like the Joint Strike Fighter in the same state, but instead it's spread over something like 40 states to ensure the project's survival.
And for whatever reason, the military seems to have a very short memory when it comes to major weapon system procurement failures. "Hmm, the V-22 went over budget, killed a good number of people, has higher maintenance needs and operating costs than we anticipated because of the USMC's requirements. I know, why don't we tie the entire future of our combat airpower to the JSF and hamstring that project with the needs of the USMC..."
Long story short, the space constraints of operating from an amphibious carrier forced design compromises on the V-22 that significantly worsened that program. Now with the JSF, the same thing is happening with the VSTOL variant that the USMC wants.
Read up on the '60s Robert Strange McNamara TFX debacle if you want to understand a lot about the F-35/JSF now and likely in the future, minus it also trying to do air superiority (we learned that lesson, although we didn't procure enough F-22s). The Strategy of Technology by Pournelle et. al. is a great source: http://jerrypournelle.com/slowchange/Strat.html
In this latest witch hunt I am sure there is plenty of blame to go around. This is not the first 1.0 system to have serious problems. It will get better. You would think they crashed the Space Shuttle and killed some people given the level of grilling they have received.
Ummm.... many people are now legally required to buy insurance, and this marketplace is the only option for many of those people. Saying "use the system or pay a fine, oh and by the way the system doesn't work" is not the same comparison to many other "1.0" systems which are optional to use.
My understanding is that it IS possible to register rather quickly during non-peak times. You are also not required to go through the marketplace to acquire insurance. And finally, open enrollment ends in March 2014, so there is plenty of time to get this sorted.
The media coverage and congressional circus is blowing this way out of proportion.
Took me 2 weeks to get in, finally got in. Application created for my wife and I. System broke. Logged in a couple days later. My wife's data is gone. No ability to add it back in. No ability to delete current application. No ability to start a new application. It's a mess.
Registering and using vastly different. Given that it took them years to get this far, I don't have a whole lot of hope that this will be usable for everyone, with data problems sorted, by March. Possible, but not likely. Especially because the attitude of "hey, just go somewhere else to buy" is so quick to be used. If that's really the case, why did we bother to even put this together, if it's not at all necessary or required?
There's going to be massive loads of hidden bugs that will plague this for months, if not years. Having such a tight deadline on it is crazy because it's not at all how any large insurance agency works, nor is it how the federal government works.
I think it's pretty obvious that if the bugs aren't sufficiently fixed within the next few weeks, the individual mandate penalty will be waived or at least delayed.
I don't give a flying rip about the individual mandate penalty. But..
If you need insurance and you just received a cancel notice from BCBS that your policy is gone as of Jan 1st it becomes imperative that this exchange work within the next few weeks.
This isn't a political game real people are going to be royally hozed and soon if this doesn't get fixed.
No, 9 weeks would require everyone who must get a new policy by Jan 1 to get all the way through all the systems, exchanges -> insurance companies, in the last three days of December.
Even if several paths are kludged for these most desperate millions of people losing their current coverage Jan 1, I can't see how any more time than 7 weeks is feasible, leaving 2 and a fraction to process them. And to know the system is really working, large numbers, 10s of thousands, had better be getting successfully through the systems before then.
As someone very interested in signing up for ObamaCare this isn't just media+congress circus.
And March '14 isn't the date that individual plans go kaput. Most of us in FL and I assume other states got notices from BCBS that our plans end on Jan 1st. That means we must signup before Dec. 15th on healthcare.gov.
My experience with healthcare.gov...
I think I must've been the first person in the US to successfully create and account and apply. I received my authentication confirmation email at 4am on Oct. 1st.
However... I still haven't been able to view any plans. When I click the "View eligible plans" button it goes to a 404. Sometimes it's a 50x, sometimes it just times-out. I've tried nearly every day since Oct. 1st. I've tried early, I've tried late, I've tried on the weekdays and weekends... I've tried re-applying, I've tried applying in another state we are thinking of moving to. From my experience healthcare.gov is radically busted.
So yes, you can create an account, enter all your personal information and "apply" but you sure can't see any plans that you are qualified for or actually you know signup.
However, healthcare.gov generally seems very quick, very responsive, I thought the UI for signup an authentication was good.
Extending your remarks, it is estimated that about 16 million people and families on the individual? market are losing their existing plans come Jan 1st because they aren't grandfathered and of course don't meet all the gold-plating in the floor plan (e.g. coverage of your children till age 26). There are of course replacement plans, but they're a lot more expensive because they by law must cover a lot more stuff.
As I understand it, If you are due a subsidy, as of now the only way you can get it is through the Federal exchange. State exchanges must use its subsidy calculator, ditto insurance companies (the latter is reported to have been a very low priority item in the system development).
As things look now (putting on my student of failed projects cap), a lot of people are going to be without insurance of any sort come Jan 1, especially because they can't afford whatever policies they're de jure or de facto allowed to buy. Which could easily kill as many people as lost in the Space Shuttle program ... except they didn't volunteer for something so obviously dangerous.
The most useful part about this article for me was to discover that Northrop Grumman's CISO is phishing its own employees. Brilliant. Wish I had thought of it. Great way to keep people on their toes and aware of what's going on in the space.
Same for backups. If you have six months of backups but you haven't tested any of them, and they broke four months ago, then it's going to be very unpleasant.
I guarantee that the contractors involved were loaded with a significant number of Java & Microsoft certified developers. Furthermore, I'd guarantee that the Big Dig, et al, were loaded with certified "real" engineers. Certification (even some fancy "project management" certification) doesn't get you any guarantees of project quality as a whole.
If anything, I'd expect a focus on certification to decrease quality. Those things are expensive & time-consuming and the only ones who invest in them are the people who expect to be in that consulting ecosystem for a long time.
There are processes, not certifications, that result in generally high-quality projects (see NASA's processes) at the cost of being incredibly slow and expensive.
I am 99% certain that the software engineers involved--that is, the ones who did the actual development of the software--had almost no say in the process that created this debacle, and were almost certainly powerless in steering the project's direction.
To take something from the article as an example, the QSSI executive's hedge about security not being part of his contract illustrates one of the real problems with our government contracting system. The sad fact is that it is more likely to have technologically illiterate "Business Analyst" and (cost-conscious) "Project Management" types making critical technology-related decisions about the scope of a project and contents of contracts than people who are technology-savvy, such as software or systems engineers, even if said people have some input.
A companion problem to this is the nature of government contracting. The QSSI guy is just doing what contractors do: they fulfill the letter of the contract (sometimes completely, sometimes by handwaving) and then when the government complains that the product doesn't work the way they wanted the contractor wins an extension or a follow-on (paid) contract to do the "additional" work.
It works like this for a reason. The "Analyst" ass-clowns in the government get better job security, because they're overseeing a project that can have its duration extended almost indefinitely. The contractor owners get to reap huge contracting windfalls. The lobbyists for the contractors who continue to bilk the government continue to get paid, and the Congressmen who the contractors buy, er, provide donations and campaign support for continue to hold their seats of power.
Look it like this: from their perspective, badly run projects that don't actually accomplish what they need to are a feature of government contracting, not a bug or problem to be solved.
Certification would be, at best, a third of the equation. It can demonstrate that you know how to do something correctly, not that you will do it correctly all the time or that you will be working in an environment that permits you to do it correctly.
No, we're talking about "affix a stamp and make yourself liable" engineering certs, not just competency testing. You know, PEng engineers, not just inflated job titles.
I'm sure your software engineer certification program will be way more successful and effective than government procurement certification program that certified these clowns in the first place.
I keep thinking whether it would be a good idea or not to hold companies responsible for security issues affecting their users. If you fail to secure passwords, why shouldn't there be a fast-track way to sue? Almost any physical service provider I can think of already is responsible for damages. Your plumber fails - in most cases the company is insured and will fix at no cost and will be responsible for damages. Your mechanic fails - there's likely a clause in their agreement that allows you claim for the damage caused. Even if your payment card is misused, the bank is expected to reverse the charges. Why are the internet service providers excluded from this treatment?
Sure, it's possible that this kind of danger would be then aimed at separate developers working in those companies too... but I'm not sure it's a bad thing. It would be easier to resist stupid requirements (I won't do it, because it would bring legal issues on me) and even the comments on SO would be different too ("if you do this, you will be sued", rather than "this is not a secure way to do it").
It was a 3 year project with requirements changing up until the last 4 months where end-to-end was managed under waterfall (some pieces/contractors used agile like methodology)...and there was 2 weeks of final end-to-end manual QA...what the hell else did they expect to happen?
It was worse, the "no window shopping" decision was made less than 4 months before launch (August to September as I recall), and changes were made through the week before launch. The NYT reports 7 major changes in less than 10 months.
And it didn't matter what the QA reported, the site was hell or high water launch Oct 1st whatever they revealed.
As always, those with political power (reported to be the White House and CMS, maybe some of HHS) trump those who have to make it work in the real world.
I don't think the one line quote from the hearing necessarily supports the headline, but I certainly don't have high hopes that the site is secure; the number of bugs and inconsistencies I found just trying to create an account dissolved any hope of that.
I'm seeing enough of this trope, not to include the BBC initially, to wonder if this didn't hit the current incarnation of the Journolist.
Unfortunately this disaster is so huge that's not adequate to tamp it down. But it sure looks like some are trying ... hmmm, what is the broadcast part of the MSM aside from Fox saying?
The overall project needs an independent security evaluation by a separate organization. Without an audit and tests before launch (and ongoing after), any problems discovered in production are the primary contractor/procurors's responsibility.
Coming from the construction industry, there isn't a contractor in the world that cares about things which aren't in the contract. Implementing features which the client has not asked for, in the design-bid-build world of public-sector contracting, just means you will lose the bid to a lower bidder who won't do those extras.