Scientific involution?
Scientists, perhaps specially the younger ones, are under constant pressure to get new funding until they can settle their position. If they want to survive in this competitive environment, they have to adapt their strategies in order to be successful. Only the fittest to this environment can survive and teach their methods and quality standards to their scientific descendants, i.e. students. If you think my wording reminds biology, you are fully right.
It is clear that the budget for scientific research is limited. Thus, it becomes necessary to use some sort of methodology to discern the best scientist and their projects, so we can keep funding them against other… let’s say not so good. We can regard it as a genuine market of scientific funding. And I don’t think there is anything wrong with this. A healthy amount of competence encourages scientist to work hard so they become as good scientist as possible. So far so good. The tricky part starts when we have to discern what it means to be a good scientist. This is, we need an objective metric of scientific performance.
Since Science is reported in form of papers, the number of papers published is a natural measure of performance. But the quality of the science is also important, and good papers, those that redefine the branches of Science, tend to be cited more often, so the number of citations is also a natural unit to have into account. Finally, it is customary to cite the original authors of an idea, so papers developing novel ideas usually get many citations. These simple arguments, hard to debate, are currently implemented in most scientific programs around the world. Thereby, the current system is based on the idea that scientists that publish very often, and with projects dealing with novel topics, are synonymous of good Science that deserves funding.
The problem is that this setup encourages scientists to focus on the elements of this metric if they want to get funded in the long term and acquire a more stable position at some point it their career. It is the Goodhart’s law applied to scientific funding: the metric becomes the target, so it ceases to be a good metric. The system encourages novelty, which is good, but also discourages studies seeking for the reproducibility of former results. It is a rational choice for scientists to try to publish as soon and as much as possible, not paying too much attention to details or weakness of their own and others findings. Such an approach would be counterproductive for their scientific record, which may risk the long-term ability of the researcher to keep working in Science. All in all, the system encourages that scientists publish false positives.
How bad can this be for Science? Well, A new study published in Arxiv has implemented an evolution-like approach to try to address this issue. Their findings suggests (and actually finds support for) a remarkable hypothesis: evolutionary forces are promoting a new era of bad Science. The study is based on a computational model similar to those aimed to study interactions between species in Ethology, and nicely described by Dawkins in The Selfish Gene. In this case, the individuals modeled are researchers that belong to various laboratories. The aim of each lab is to get funding and hire new scientists, and for this they need to keep a constant rate of publications in competition with other labs. The in silico researchers follow a set of simple rules that allow them to develop different strategies when they make genuine discoveries, such as keep searching to discard false positives or disregard them and try to publish as is without double checking. The rules are simple and try to mimic current competence and metric of scientific success. Different strategies can render them more or less successful with respect funding, with plays a role in their ability to survive and leave scientific descendants (note how this is nothing but memetic evolution). It is important to remark that in this world there are no cheaters. This is, all researchers were adapting their strategies, but following a set of rules that preclude them from doing bad Science on purpose. They could be bad scientists, but they endorse a fully ethical behaviour and act rationally according to the system rules. The result is that after few generations, these in silico researchers end up focusing on novelty above all, ignoring the search of reproducibility, one of the most important aspects of the Scientific method. In this model, evolutionary forces are the only drivers of bad science by accident!
So what does this mean? Well, I think we should not over-interpret these results. This is just a model, and it can only be as good as the hypothesis it relies on. Although they were carefully set to mimic current system, it’s a crude simplification that neglects many factors and personal motivations that play a role in the choices of real scientists. Still, I believe these results deserve a bit of a reflexion for the scientific community, and specially those responsible for the distribution of funding. Maybe focusing on novelty and number of papers above all is not a good metric after all. But changing the metric could not be enough! Evolutionary forces are powerful and we are all subject to them. A change in the metric will not fix the problem, since researchers will always evolve to abuse it (again, following Goodhart’s law). I don’t know if there is a solution to this problem, and it is way beyond my qualifications to suggest one. But at least I think the scientific community should start to rethink seriously about the meaning of being a good scientist. Setting the right incentives might be the only way to put those evolutive forces to work in favour of good and solid Science.
You can keep reading about this interesting topic in this article in New Scientist.
Tags: simulation opinion
Categories: Funding
subscribe via atom