By 1913intel
Please take a look at some short clips (articles further below) in order to get a broad view of the problems in science:
- "relentless pursuit of taxpayer funding has eliminated curiosity, basic competence, and scientific integrity in many fields."
- "physics, economics, psychology, medicine, and geology are unable to explain over 90 percent of what we see"
- "climate, demography, asset prices, and natural disasters—are minimally predictable."
- "chronic inability to reproduce research findings."
- "progress in developing better theory and forecasting capability has stagnated since the 1960s."
- There is a "replication and reproducibility crisis."
- “’Nutrition’ is now a degenerating research paradigm in which scientifically illiterate methods, meaningless data, and consensus-driven censorship dominate the empirical landscape.”
- "These dubious practices may include misrepresentations, research bias, and inaccurate interpretations of data."
- "when a superstar scientist dies their field sees a small burst of activity in the form of fresh publications." These are highly cited. The superstar was hindering new ideas.
- "There’s an increasing concern among scholars that, in many areas of science, famous published results tend to be impossible to reproduce."
- "What is going on here? In two words: “communal reinforcement,” more commonly known as group-think."
- "Another big project has found that only half of [psychology] studies can be repeated."
- "But what about economics? Experimental econ is akin to psychology, and has similar issues."
- "the whole academic discipline of economics is being re-considered: the theory as well as the policy advice."
- "I’m suddenly concerned that all of published math is wrong because mathematicians are not checking the details, and I’ve seen them wrong before,". Because math builds on itself and is increasingly too difficult to check properly.
My experiences at four research universities and as a National Institutes of Health (NIH) research fellow taught me that the relentless pursuit of taxpayer funding has eliminated curiosity, basic competence, and scientific integrity in many fields. Yet, more importantly, training in “science” is now tantamount to grant-writing and learning how to obtain funding. Organized skepticism, critical thinking, and methodological rigor, if present at all, are afterthoughts. Thus, our nation’s institutions no longer perform their role as Eisenhower’s fountainhead of free ideas and discovery. Instead, American universities often produce corrupt, incompetent, or scientifically meaningless research that endangers the public, confounds public policy, and diminishes our nation’s preparedness to meet future challenges.
Mounting evidence suggests a lot of published research is false.
Scholarly research is in crisis, and four issues highlight its dimensions.
The first is that important disciplines such as physics, economics, psychology, medicine, and geology are unable to explain over 90 percent of what we see: dark matter dominates their theoretical understanding. ...
The second dimension of the research crisis is that systems which are critical to humankind—especially climate, demography, asset prices, and natural disasters—are minimally predictable. ...
The third dimension is a chronic inability to reproduce research findings. ...
The final indicator of crisis in research is that progress in developing better theory and forecasting capability has stagnated since the 1960s. ...
A new report released this week by the National Academies of Sciences, Engineering, and Medicine is weighing in on a contentious debate within the science world: the idea that scientific research is fundamentally flawed, rife with published findings that often can’t be reproduced or replicated by other scientists, otherwise known as the replication and reproducibility crisis.
So which is it? Is the egg good or bad? And, while we are on the subject, when so much of what we are told about diet, health, and weight loss is inconsistent and contradictory, can we believe any of it? Quite frankly, probably not. Nutrition research tends to be unreliable because nearly all of it is based on observational studies, which are imprecise, have no controls, and don’t follow an experimental method. As nutrition-research critics Edward Archer and Carl Lavie have put it, “’Nutrition’ is now a degenerating research paradigm in which scientifically illiterate methods, meaningless data, and consensus-driven censorship dominate the empirical landscape.”
These dubious practices may include misrepresentations, research bias, and inaccurate interpretations of data. One common questionable research practice entails formulating a hypothesis after the research is done in order to claim a successful premise. Another highly questionable practice that can shape research is ghost-authoring by representatives of the pharmaceutical industry and other for-profit fields. Still another is gifting co-authorship to unqualified but powerful individuals who can advance one’s career. Such practices can unfairly bolster a scientist’s reputation and increase the likelihood of getting the work published.
Years later, at a national meeting of scientists to set future research policy, I got a horrible shock. There was the fraudulent academic. They hadn’t been sacked; they had been promoted and invited to give their views on the future of research on a level footing with myself and other colleagues who I greatly respect. I was flabbergasted.How dis this happen? My strong suspicion is that the academic’s university had no desire for an inquiry that might risk damaging their reputation, given their failure to prevent his wrongdoing. I now regret not making a formal complaint for malpractice at the time, which would have forced them to take action.
There’s a reason everyone’s confused about whether coffee causes cancer, or whether butter’s good for you or bad. Food research has some big problems, as we’ve discussed here and here: questionable data, untrustworthy results, and pervasive bias (and not just on the part of Big Food). There’s reason to hope that scientists and academic journals will clean up their acts, and that journalists will refine their bullshit detectors and stop writing breathlessly about new nutrition “discoveries” that are anything but. Until that happens, though, we all need to get better at filtering for ourselves.
In recent years, psychology has dealt with a legitimacy crisis. Many influential psychological studies could not be reproduced by other psychologists, discrediting some key insights and weakening academic faith in the entire field. Nutrition science has a similar problem. The loudest critics argue that the methodologies relied on by researchers give bad data that are meaningless at best. Others worry that funding gives undue influence to the federal government, big business, or influential nonprofit associations. And some critics think nutrition science focuses on the wrong questions entirely about nutrition.
Older scientists aren’t notably worse at accepting revolutionary ideas compared to younger colleagues, research has found. But a paper published in August in the American Economic Review suggests there may be subtler ways in which the top dogs have a discouraging effect on new entrants. According to the paper, which draws on decades of data on more than 12,000 elite biology researchers, when a superstar scientist dies their field sees a small burst of activity in the form of fresh publications. What’s more, the authors of the new papers, which are more likely than usual to be highly cited, are typically newcomers who have never published in this subfield before.
So, let’s imagine what might happen if the rules of professional science evolved such that scientists were incentivized to publish as many papers as they could and if those who published many papers of poor scientific rigor were rewarded over those who published fewer papers of higher rigor? What would happen if scientists weren’t rewarded for the long-term reproducibility and rigor of their findings, but rather became a factory that produced and published highly exciting and innovative new discoveries, and then other scientists and companies spent resources on the follow up studies and took all the risk?
The case against p-values and statistical significance is not a criticism of the concepts themselves but of their misuse.
Should we believe the headline, “Drinking four cups of coffee daily lowers risk of death”? How about, “Mouthwash May Trigger Diabetes. . .”? Should we really eat more, not less, fat? And what should we make of data that suggest people with spouses live longer?
These sorts of conclusions, from supposedly scientific studies, seem to vary from month to month, leading to ever-shifting “expert” recommendations. However, most of their admonitions are based on flawed research that produces results worthy of daytime TV.
Misleading research is costly to society directly because much of it is supported by the federal government, and indirectly, when it gives rise to unwise, harmful public policy.
Social science studies are notorious offenders. A landmark study in the journal Nature Human Behaviour in August reported the results of efforts to replicate 21 social science studies published in the prestigious journals Nature and Science between 2010 and 2015.
The multi-national team actually “conducted high-powered replications of the 21 experimental social science studies — using sample sizes around five times larger than the original sample sizes” and found that “62% of the replications show an effect in the same direction as the original studies.” One out of the four Nature papers and seven of the seventeen Science papers evaluated did not replicate, a shocking result for two prestigious scientific journals. The authors noted two kinds of flaws in the original studies: false positives and inflated effect sizes.
Science is supposed to be self-correcting. Smart editors. Peer review. Competition from other labs. But when we see that university research claims – published in the crème de la crème of scientific journals, no less -- are so often wrong, there must be systematic problems. One of them is outright fraud – “advocacy research” that has methodological flaws or intentionally misinterprets the results.
There’s an increasing concern among scholars that, in many areas of science, famous published results tend to be impossible to reproduce. This crisis can be severe. For example, in 2011, Bayer HealthCare reviewed 67 in-house projects and found that they could replicate less than 25 percent. Furthermore, over two-thirds of the projects had major inconsistencies. More recently, in November, an investigation of 28 major psychology papers found that only half could be replicated.Similar findings are reported across other fields, including medicine and economics. These striking results put the credibility of all scientists in deep trouble.
The latest in a raft of experiments suggests a predicted “train wreck” in social sciences is under way. Paul Biegler reports.“The findings reinforce the roles that two inherent intuitions play in scientific decision-making: our drive to create a coherent narrative from new data regardless of its quality or relevance, and our inclination to seek patterns in data whether they exist or not,” he says.Dingledine also says the results speak to a bigger problem, something Kahneman famously described in an open letter to colleagues in 2012 as a “train wreck looming”: the widespread failure to replicate the findings of many important studies in the social sciences.That wreck may well be upon us.A recent article in the journal Nature Human Behaviour reported an attempt to replicate 21 social science experiments published in the journals Nature and Science between 2010 and 2015. One study, for example, found that viewing images of Rodin’s sculpture The Thinker led people to think more analytically and discouraged a belief in God. It fell at the replication hurdle. In fact, the researchers only succeeded in 13 replications, and even then effect sizes were, on average, just half of those seen in the original experiments.
What is going on here? In two words: “communal reinforcement,” more commonly known as group-think. The headlines may say “research shows” but it doesn’t: researchers show. Scientists, like all of us, are affected by their peers’ opinions. If everyone does it, they think it’s probably ok. They also like to be liked, not to mention that they like having an income. This biases their judgement, but the current organization of the academic system does not offer protection. Instead, it makes the problem worse by rewarding those who work on popular topics.
There is increasing concern that most current published research findings are false. The probability that a research claim is true may depend on study power and bias, the number of other studies on the same question, and, importantly, the ratio of true to no relationships among the relationships probed in each scientific field. In this framework, a research finding is less likely to be true when the studies conducted in a field are smaller; when effect sizes are smaller; when there is a greater number and lesser preselection of tested relationships; where there is greater flexibility in designs, definitions, outcomes, and analytical modes; when there is greater financial and other interest and prejudice; and when more teams are involved in a scientific field in chase of statistical significance. Simulations show that for most study designs and settings, it is more likely for a research claim to be false than true. Moreover, for many current scientific fields, claimed research findings may often be simply accurate measures of the prevailing bias. In this essay, I discuss the implications of these problems for the conduct and interpretation of research.
Another big project has found that only half of studies can be repeated. And this time, the usual explanations fall flat. Over the past few years, an international team of almost 200 psychologists has been trying to repeat a set of previously published experiments from its field, to see if it can get the same results. Despite its best efforts, the project, called Many Labs 2, has only succeeded in 14 out of 28 cases. Six years ago, that might have been shocking. Now it comes as expected (if still somewhat disturbing) news.In recent years, it has become painfully clear that psychology is facing a “reproducibility crisis,” in which even famous, long-established phenomena—the stuff of textbooks and TED Talks—might not be real. There’s social priming, where subliminal exposures can influence our behavior. And ego depletion, the idea that we have a limited supply of willpower that can be exhausted. And the facial-feedback hypothesis, which simply says that smiling makes us feel happier.
By now, most people have heard of the replication crisis in psychology. When researchers try to recreate the experiments that led to published findings, only slightly more than half of the results tend to turn out the same as before. Biology and medicine are probably riddled with similar issues. ...
But what about economics? Experimental econ is akin to psychology, and has similar issues....
This strongly suggests that there is a lot of mystery meat that goes into economists’ analysis. Scholars may add or remove different control variables, cut up their data into different slices, add or delete data, or modify their statistical analyses in a number of subtle ways. Any one of these can change the results.
But there is something else going on as well: the whole academic discipline of economics is being re-considered: the theory as well as the policy advice. Not all economists are on board with this project, and not all economics is being overthrown. And not all the rethinking is actually new. But it does seem that we are in the midst of a slow paradigm shift. Who knows where it will end?
"I think there is a non-zero chance that some of our great castles are built on sand," he said, arguing that we must begin to rely on AI to verify proofs.
Kevin Buzzard, a number theorist and professor of pure mathematics at Imperial College London, believes that it is time to create a new area of mathematics dedicated to the computerization of proofs. The greatest proofs have become so complex that practically no human on earth can understand all of their details, let alone verify them. He fears that many proofs widely considered to be true are wrong. Help is needed. ... New proofs by professional mathematicians tend to rely on a whole host of prior results that have already been published and understood. But Buzzard says there are many cases where the prior proofs used to build new proofs are clearly not understood. For example, there are notable papers that openly cite unpublished work. This worries Buzzard. “I’m suddenly concerned that all of published math is wrong because mathematicians are not checking the details, and I’ve seen them wrong before,” Buzzard told Motherboard while he was attending the 10th Interactive Theorem Proving conference in Portland, Oregon, where he gave the opening talk.
Academic journals in Russia are retracting more than 800 papers following a probe into unethical publication practices by a commission appointed by the Russian Academy of Sciences (RAS). The moves come in the wake of several other queries suggesting the vast Russian scientific literature is riddled with plagiarism, self-plagiarism, and so-called gift authorship, in which academics become a co-author without having contributed any work. The loudest critics argue that the methodologies relied on by researchers give bad data that are meaningless at best. Others worry that funding gives undue influence to the federal government, big business, or influential nonprofit associations. And some critics think nutrition science focuses on the wrong questions entirely about nutrition.
https://old.reddit.com/r/EmergingRisks/comments/ewt6i1/science_is_broken/
No comments:
Post a Comment