Category Archives: Software Quality

Will information systems be able to collect and track the vaccination records for hundreds of millions?

Doubt it, based on past experience:

But Hannan says the US immunization data collection systems at the state level are prepared to manage the expected crush of information from an unprecedented mass vaccination campaign like the one the country is about to start. Most tools were in place even before the pandemic hit.

Source: Multidose COVID-19 vaccines will test state tracking systems – The Verge

My state is famous for its Cover Oregon ACA health exchange albeit for the wrong reasons. After spending $450 million dollars on the agency, it never enrolled a single person and was shut down as a failed information system.

There is a long history of failed information systems. Will they be able to successfully track millions of vaccinations? We should all be worried about this as it could create another vaccine distribution bottleneck. The organization responsible for the Cover Oregon mess is also now responsible for the vaccination tracking. What could go wrong?

Continue reading Will information systems be able to collect and track the vaccination records for hundreds of millions?

Software: The greatest software error in history?

In The Price of Panic, a detailed, albeit dated, overview of the world’s Covid-19 response, the authors note the use of Neil Ferguson’s ICL Covid simulator software – and its rather outrageous errors – could be classified as the most expensive software error in history.

Ferguson’s model predicted massive numbers of deaths from Covid-19 including 2.2 million in the U.S. through spring of 2020. In his 20 years as a disease modeler, each of his predictions has been off by orders of magnitude.

In the case of his model output last March, his model was the basis for adopting lock down measures in the UK, the US and other countries. Yet his model was completely wrong.

Continue reading Software: The greatest software error in history?

Public Health England lost about 16,000 Covid-19 positive tests due to not understanding Excel spreadsheet limitations

The BBC has confirmed the missing Covid-19 test data was caused by the ill-thought-out use of Microsoft’s Excel software. Furthermore, PHE was to blame, rather than a third-party contractor.

Source: Covid: Test error ‘should never have happened’ – Hancock – BBC News

Could have happened to anyone. What’s 16,000 lost positive test results among friends, anyway?

Public health says it has no data on what works and what does not work

Oregon has released its periodic modeling update report.

Shockingly, they say they have no data on what measures work or do not work, or whether or not anyone is adhering to them. They have no information on whether some measures work better or worse than others. They have no data on any measures at all.

Continue reading Public health says it has no data on what works and what does not work

It’s worse than we thought: “Second Analysis of Ferguson’s Model”

In the past I had some comments on Neil Ferguson’s disease model and have repeatedly noted its poor quality. This model was used, last spring, as the basis for setting government policies to respond to Covid-19. Like many disease models, its output was garbage, unfit for any purpose.

The following item noted that the revision history, since last spring, is available and shows that ICL has not been truthful about the changes made to the original model code.

Source: Second Analysis of Ferguson’s Model – Lockdown Sceptics

THIS! Many academic models including disease models and climate models, average the outputs from multiple runs, some how imaginatively thinking that this produces a reliable projection – uh, no, it does not work that way.

An average of wrong is wrong.  There appears to be a seriously concerning issue with how British universities are teaching programming to scientists. Some of them seem to think hardware-triggered variations don’t matter if you average the outputs (they apparently call this an “ensemble model”).

Averaging samples to eliminate random noise works only if the noise is actually random. The mishmash of iteratively accumulated floating point uncertainty, uninitialised reads, broken shuffles, broken random number generators and other issues in this model may yield unexpected output changes but they are not truly random deviations, so they can’t just be averaged out.

Software quality assurance is often missing in academic projects that are used for public policy:

For standards to improve academics must lose the mentality that the rules don’t apply to them. In a formal petition to ICL to retract papers based on the model you can see comments “explaining” that scientists don’t need to unit test their code, that criticising them will just cause them to avoid peer review in future, and other entirely unacceptable positions. Eventually a modeller from the private sector gives them a reality check. In particular academics shouldn’t have to be convinced to open their code to scrutiny; it should be a mandatory part of grant funding.

The deeper question here is whether Imperial College administrators have any institutional awareness of how out of control this department has become, and whether they care. If not, why not? Does the title “Professor at Imperial” mean anything at all, or is the respect it currently garners just groupthink?

When a software model – such as a disease model – is used to set public policies that impact people’s lives – literally life or death – these models should adhere to standards for life-safety critical software systems. There are standards for, say, medical equipment, or nuclear power plant monitoring systems, or avionics – because they may put people’s lives at risk. A disease model has similar effects – and hacked models that adhere to no standards have no business being used to establish life safety critical policies!

I and another software engineer had an interaction with Gavin Schmidt of NASA regarding software quality assurance of their climate model or paleoclimate histories[1]. He noted they only had funding for 1/4 of a full time equivalent person to work on SQA – in other words, they had no SQA. Instead, their position was that the model’s output should be compared to others. This would be like – instead of testing, Microsoft would judge its software quality by comparing the output of MS Word to the output of another word processor. In other words, sort of a quailty-via-proxy analogy. Needless to say, this is not how SQA works.

Similarly, the climate model community always averages multiple runs from multiple models to create projections. They do this even when some of the model projections are clearly off the rails. Averaging many wrongs does not make a right.

[1] Note that NASA does open source their software which enables more eyes to see the code, and I do not mean to pick on NASA or Schmidt here. They are doing what they can within their funding limitations. The point, however is that SQA is frequently given short shrift in academic-like settings.

What if you could be convicted with secret evidence you cannot see nor contest?

All defendants have a right to review the evidence before them. When software applications produce a conclusion, then the software source code must be re-viewable by the defense.

The government argues it can use secret software against a defendant – software that may very well be defective (think Neil Ferguson’s Imperial College London’s secret disease modeling code that ignores all modern software engineering practices).

Can secret software be used to generate key evidence against a criminal defendant?

Source: EFF and ACLU Tell Federal Court that Forensic Software Source Code Must Be Disclosed | Electronic Frontier Foundation

Twitter still a mess in the aftermath of the take over of their systems

Read the whole thing  –  Twitter blog post update.

It’s pretty clear they still do not have a full handle on the situation.

Twitter acknowledges that the hackers downloaded the Twitter Data for some accounts,  which may include private Direct Messages.

I no longer regard Twitter as safe. I deactivated 2 of my 4 accounts, and had already deleted all content of my main account – except DMs. I’m in process of clearing out all the DM’s now. I intend to keep one or two of the accounts alive but will probably no longer use them.

This incident was a total and complete failure of Twitter security and their ability to be trusted with holding information. At this time, no one should have any trust in Twitter – and I mean no one. Clear your data as soon as possible. What just happened could have created one or more international incidents as hackers seized control of prominent political accounts.

Japan pulls its coronavirus tracking smartphone app due to software design errors

The Japanese government has pledged to fix within a week bugs that have caused its coronavirus contact-tracing smartphone app to be shut down, the health minister said Tuesday.

The free app, which was launched Friday and downloaded around 3.71 million times as of Tuesday morning, erroneously accepts ID numbers not issued by the Health, Labor and Welfare Ministry, Katsunobu Kato, the minister responsible for the system, said at a press conference.

Source: Bugs force Japan gov’t to temporarily shut down virus contact-tracing app

Experts criticize ICL’s Ferguson’s Covid SIM model as garbage

Those of us who have seen Neil Ferguson’s ICL Covid sim model have the same views as this computational epidemiologist:

As Ferguson himself admits, the code was written 13 years ago, to model an influenza pandemic. This raises multiple questions: other than Ferguson’s reputation, what did the British government have at its disposal to assess the model and its implementation? How was the model validated, and what safeguards were implemented to ensure that it was correctly applied? The recent release of an improved version of the source code does not paint a favorable picture. The code is a tangled mess of undocumented steps, with no discernible overall structure. Even experienced developers would have to make a serious effort to understand it.

I’m a virologist, and modelling complex processes is part of my day-to-day work. It’s not uncommon to see long and complex code for predicting the movement of an infection in a population, but tools exist to structure and document code properly. The Imperial College effort suggests an incumbency effect: with their outstanding reputations, the college and Ferguson possessed an authority based solely on their own authority. The code on which they based their predictions would not pass a cursory review by a Ph.D. committee in computational epidemiology.

Source: Britain’s Hard Lesson About Blind Trust in Scientific Authorities

Continue reading Experts criticize ICL’s Ferguson’s Covid SIM model as garbage