Those of us who have seen Neil Ferguson’s ICL Covid sim model have the same views as this computational epidemiologist:
As Ferguson himself admits, the code was written 13 years ago, to model an influenza pandemic. This raises multiple questions: other than Ferguson’s reputation, what did the British government have at its disposal to assess the model and its implementation? How was the model validated, and what safeguards were implemented to ensure that it was correctly applied? The recent release of an improved version of the source code does not paint a favorable picture. The code is a tangled mess of undocumented steps, with no discernible overall structure. Even experienced developers would have to make a serious effort to understand it.
I’m a virologist, and modelling complex processes is part of my day-to-day work. It’s not uncommon to see long and complex code for predicting the movement of an infection in a population, but tools exist to structure and document code properly. The Imperial College effort suggests an incumbency effect: with their outstanding reputations, the college and Ferguson possessed an authority based solely on their own authority. The code on which they based their predictions would not pass a cursory review by a Ph.D. committee in computational epidemiology.
Source: Britain’s Hard Lesson About Blind Trust in Scientific Authorities
A model is an hypothesis about how the world works. The output of a model is itself an hypothesis. Unfortunately, media and politicians view model output as realistic predictions of the future – which is not how these models should be used.
The author of the above, a computational epidemiologist, notes that models are never “true” or “false” – they can only be “good” or “bad”.
Ferguson’s code is not just lousy – from a software engineering perspective in terms of quality assurance, maintainability, upgradability, and repeatability of results, the code is awful, really awful. The code has never undergone independent review. Many “experts” are saying the same thing. His model produces wildly different results each time it is run with the same input parameters yet this model was the basis for pubic policies globally costing an estimated $9 trillion.
I did my own review of Ferguson’s software – a version of the softwre was released by the University of Edinburg on Github. I posted my analysis on this blog but removed nearly all of my Covid-related comments from public view because of the perception that only “experts” are permitted to comment on anything having to do with a pandemic, and especially the work of an academic. My review found the same fundamental software problems identified by “experts”, plus a few more.
I have a degree in information and computer science and an M.S. in software engineering (and an M.B.A.), have worked in Silicon Valley and for Microsoft and others, have written a dozen technical books and have two U.S. patents. I have also taught university courses in decision science and business modeling (use of computational models to find optimal solutions to business operations and manufacturing scenarios). And I managed that in spite of six traumatic brain injuries… heh … Even though I am presumably qualified to review software, I lack the necessary title to be an “expert” and for my comments to be taken seriously. My review found issues that violate the “guidance of experts” requirement of social media platforms.
Unfortunately, in our world today only identified experts may comment on serious subjects so I made my review private and am merely linking to the comments of “experts”.
