Delineamento Experimental - Parte 3: Vieses na Ciência
Overview
Bruno Martynhak explains how publication incentives and researcher choices create biases that weaken reproducibility and generalizability, including publication bias, HARKing, p-hacking, selective exclusions, and subjective interpretation. He illustrates the risks with omitted stroke-model mice, a Nature-reported survey in which 15% of respondents said they had discarded data based on subjective judgment, and preclinical ALS findings where non-industry-funded effects were about three times larger, then recommends defining criteria and experimental units in advance and designing studies to represent biological variability.
Key takeaways
- Positive findings can be disproportionately published because journals value citations and researchers depend on publications for grants, status, and sometimes salary.
- An unexpected result from a multi-variable experiment is exploratory evidence, not confirmation; repeat the experiment with the specific finding as a preplanned outcome.
- Removing animals or data points after seeing whether they support a hypothesis can inflate treatment effects, even when researchers believe they are correcting bad data.
- Predefined exclusion rules should be applied when an animal is judged unfit, rather than after its measurement is known; the transcript cites a survey in which 15% reported subjective data-point exclusions.
- Experimental units must match the design: counting several pups from one rat as independent observations creates pseudoreplication and overstates sample size.
- Running control and treatment groups concurrently, and including both sexes, helps reduce confounding and makes animal-study findings more generalizable.
Chapters
- Weak reproducibility wastes research funding, time, and animals, and can make promising animal treatments fail in human trials or other animal models.
- Researchers may rely on publications for status, grant funding, and, in some US institutions, part of their salary.
- Journals seeking citations and prestige can favor statistically significant positive findings, while negative results face higher publication barriers.
- HARKing means presenting a hypothesis created after seeing results as if it had been predicted beforehand; exploratory findings should be labeled exploratory.
- Measuring many variables in one experiment makes chance differences more likely, so unexpected findings should be tested in a separate confirmatory experiment.
- P-hacking includes switching statistical tests, removing an inconvenient animal, or choosing a more permissive post hoc test until a desired difference appears.
- Behavioral scoring can be influenced by expectations, while literature selection and scientific fads can encourage researchers to find effects reported by others.
- A stroke-treatment paper described 10 treated mice, but reported only 7 after three treated animals died; omitting those deaths made the treatment appear more effective.
- A laboratory review of a highly diluted immunoglobulin study found that control measurements were discarded when their values were unfavorable, alongside a financial conflict of interest.
- A Nature-reported survey found that 0.3% of respondents admitted falsifying data, while 15% reported setting aside data points based on subjective judgment.
- A study's prior plausibility affects false-positive risk: an implausible claim such as homeopathy is more likely to produce a false positive than a well-supported hypothesis.
- Preclinical ALS treatment studies showed larger effects in published academic research, while industry studies found no survival benefit; non-industry-funded effects were reported as roughly three times larger.
- Pseudoreplication can occur when pups from one pregnant rat are counted as independent experimental units instead of treating the litter or dam appropriately.
- Controls and experimental groups should run together, since testing controls on a different day can confound treatment effects with day-to-day conditions.
- Define exclusion criteria and experimental units a priori, investigate error sources before data collection, and include both male and female animals to improve generalizability.
Transcript excerpt
0:00 continuando com a parte 3 agora a parte mais subjetiva estão os viésis a partir de descrição dos resultados a gente já cobriu o Rive resolve uma boa parte disso se as pessoas seguirem essas recomendações ou o máximo possível mas ainda a gente precisa que os nossos resultados sejam robustos e generalizáveis quais são os problemas da e reprodutibilidade a gente tem um uso inapropriado de animais gasto de verba perda de tempo e frustrações em perspectivas de tratamento tem um tratamento para alguma doença que parece muito promissora em modelos animais e aí a gente vai testar em humanos não tem efeito a gente vai testar em outros
0:50 modelos animais não tem efeito então é importante que os nossos resultados sejam já de início bastante robustos e generalizáveis como resolver esse problema não temos uma solução infalível tem aspectos inerentes aos incentivos de publicação Então a gente tem coisas como status égo mesmo as pessoas querem publicar porque isso acaba sendo uma moeda social inclusive isso tem efeito no currículo e sim o currículo vai envolver o financiamento para o próximo projeto quanto mais você publica é mais provável que você consiga financiamento para publicar mais em muitos locais do mundo o pagamento do próprio salário vem do financiamento do projeto Como assim temos um professor nos Estados Unidos o
1:40 departamento banca 50% do salário o restante é bancado por Grant de pesquisa como que ele consegue doente de pesquisa tendo publicações então aí a gente tem um incentivo para publicação que pode atrapalhar a gente conseguir ter resultados acaba gerando vieses de publicação por exemplo esse mesmo tipo de incentivo ocorre nas revistas que publicam os artigos científicos as revistas querem que artigos de alta qualidade sejam publicados e artigos que acabem sendo muito citados porque isso vai aumentar o
This is the start of the transcript. Save this video free for the whole transcript, searchable and time-coded.
Related notes
Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, Bruno Martynhak.