Drug Safety, Probabilistic Causal Assessment, and Evidence Synthesis

Breadcrumb Navigation



27  January 2017

09:00 - 09:10 Welcome: Barbara Osimani
09:10 - 09:30 Rani Lill Anjum and Elena Rocca (Norwegian University of Life Sciences): Post Market Risk Assessment of Drugs as a Way to Uncover Causal Mechanisms Jürgen Landes (Munich)
09:30 - 09:50 Jon Williamson (University of Kent): EBM+: Improving the Way in Which Evidence-Based Medicine Handles Evidence of Mechanisms
09:50 - 10:10 Felipe Romero and Jan Sprenger (University of Tilburg): Making Scientific Inferences More Objective: Replication and Scientific Self-Correction
10:10 - 10:30 Barbara Osimani (MCMP/LMU Munich): A Multilayer Approach to Modeling Probabilistic Causal Inference through Evidence Synthesis
10:30 - 10:50 Q & A
10:50 - 11:15 Coffee Break
11:15 - 11:35 Jeff Aronson (Oxford University): Defining a Signal Barbara Osimani (Munich)
11:35 - 11:55 Ralph Edwards (WHO, Uppsala Monitoring Center): Causality in Pharmacovigilance: Small Chances – Big Problems
11:55 - 12:10 Q & A
12:10 - 13:15 Lunch
13:15 - 13:35 Mike Kelly (The National Institute for Health and Care Excellence/Cambridge University): EBM, Reductionism, and the Road to Medicine’s Adaptive Pathways Bennett Holman (Seoul)
13:35 - 13:55 Ulrich Mannsmann (IBE/LMU Munich): How to Quantify the Effect of and Correct Design-dependent Bias in RCTs
13:55 - 14:10 Q & A
14:10 - 14:35 Coffee Break
14:35 - 14:55 Brigitte Keller-Stanislawski (Paul Ehrlich Institut): Benefit Risk Evaluation of Medicinal Products Post-Authorisation and Evidence Based Decision Making in Pharmacovigilance David Teira (Madrid)
14:55 - 15:15 Beth Shaw (NICE): Assessing Drug Safety – What Can We Learn From Other Assessments of Harm
15:15 - 15:35 Q & A
15:35 - 16:00 Coffee Break
16:00 - 16:20 Stephen Senn (CCMS, Luxembourg Institute of Health): What Randomisation Can and Cannot Do for You Roland Poellinger (Munich)
16:20 - 16:40 Norbert Benda (BfArM): Challenges for a Decision-Theoretic Framework in Drug Safety and Benefit Risk Assessments
16:40 - 17:00 Q & A
17:00 - 17:30 General Discussion Barbara Osimani, Jürgen Landes, Roland Poellinger (Munich)
20:00 Dinner at Restaurant "Goldmarie" (Schmellerstraße 23, 80337 München)

28 January 2017

09:30 - 09:50 Stephen Mumford (University of Nottingham): Evidence Synthesis for Dispositionalists Rani Lill Anjum (Oslo)
09:50 - 10:10 Adam La Caze (University of Queensland): Evaluating Evidence in Drug Safety
10:10 - 10:30 Jacob Stegenga (Cambridge University): Bayesian Mechanista
10:30 - 10:50 Q & A
10:50 - 11:15 Coffee Break
11:15 - 11:35 Roland Poellinger (MCMP/LMU Munich): Shaping Causal Claims in Pharmacology: The Interplay between Population Characteristics and Causal Structure Adam La Caze (Queensland)
11:35 - 11:55 Jürgen Landes (MCMP/LMU Munich): Variety of Evidence
11:55 - 12:15 Martin Posch and Franz König (Center for Medical Statistics, Informatics and Intelligent Systems, Medical University of Vienna): Challenges of Optimizing Drug Development Programs for Regulatory Decision Making
12:15 - 12:35 Q & A
12:35 - 13:40 Coffee Break
13:40 - 14:00 David Teira (UNED Madrid): Evidential Pluralism and Regulatory Paternalism in Drug Testing Sara Ruggieri (AIFA)
14:00 - 14:20 Bennett Holman (Yonsei University) and Justin Bruner (Australian National University): Experimentation by Industrial Selection
14:20 - 14:35 Q & A
14:35 - 14:50 Coffee Break
14:50 - 17:00 Round Table Carlo Martini (Helsinki),
Christina Schneider (Munich)


Rani Lill Anjum and Elena Rocca (Norwegian University of Life Sciences): Post Market Risk Assessment of Drugs as a Way to Uncover Causal Mechanisms

Pharmacology is as much about establishing causation as it is about understanding it. To establish that a certain treatment causes a positive outcome, however, is only half the task. One needs to be able to make predictions for the single patient, in particular about potential harms and benefits, and ponder the risk for the first to outweigh the second. For this, a mechanistic understanding of causation is crucial. Here we argue that, while the repeated corroboration of the same causal hypotheses through experimentation is useful to establish causation, it is little valuable to advance its mechanistic understanding. Indeed, once we acknowledge that causation is a complex, context sensitive matter, we must accept that it is by digging into contextual influence, and not by eliminating it, that we can learn more about the how and why the intervention does its causal work. When such discoveries happen, indeed, they are referred to as “chancy” or “serendipitous”. When a treatment fails to give the expected outcome, therefore, it offers an opportunity to investigate the local context of failure and identify possible interferers. Potentially, this helps uncovering more of the causal nexus by which the outcome is produced. Both pre-clinical research and clinical experimentation alone are poorly fit for uncovering causal mechanisms through failure, since they are based on screening off, or disregard, of interferers. This leaves the post market drug monitoring as the best scenario for systematically generating mechanistic hypotheses about a treatment through studying instances of causal failure of treatment in individuals. We are going to suggest an integrated framework in which post market monitoring, through the study of treatment failure, feeds pre-clinical and clinical research with mechanistic hypothesis.top

Jeff Aronson (Oxford University): Defining a Signal

The World Health Organization (WHO) defines pharmacovigilance as “the science and activities relating to the detection, assessment, understanding and prevention of adverse effects or any drug-related problem” [1]. This definition includes both pharmacoepidemiology studies and the collection and evaluation of spontaneous case reports of suspected adverse drug reactions [2]. However, a large proportion of the information that leads to conclusions about adverse drug reactions comes from the latter and involves the analysis of what are commonly known as drug-event pairs.
An adverse event is defined as “any abnormal sign, symptom, laboratory test, syndromic combination of such abnormalities, untoward or unplanned occurrence (e.g. an accident or unplanned pregnancy), or any unexpected deterioration in a concurrent illness” [3]. If such an event occurs while a patient is taking a drug, or at some time afterwards, it may or may not be attributable to it. All adverse drug reactions are adverse events, but not all adverse events are adverse drug reactions.
The occurrence of an adverse event in a patient taking a medication does not establish that the medication caused the event, although, as psychological experiments have suggested [4], “[people’s] unwillingness to deduce the particular from the general [is] matched only by [their] willingness to infer the general from the particular." Nevertheless, it may sometimes be possible to infer causation from anecdotal evidence. There are two general problems: (a) to infer a causative link between a medication and an event in general (i.e. can the medication ever cause the event?) and (b) to infer a causative link between the medication and an event in an individual (i.e. did the medication, if it can cause the event, cause this one?). Here I explore the general problem.

There are three levels of causation:
1. Definitive (“between-the-eyes”) events In these [uncommon] cases there is an demonstrable relation between the medication and the event, and an association can be established from one or only a few events [5]. An example is thrombophlebitis that occurs immediately or soon after an intravenous injection and at the same site.
2. Designated medical events (DMEs) These are events that are very often, but not always, associated with medications. When a DME occurs in an individual taking a drug, causation is highly likely, although not definite. Examples include aplastic anaemia, Stevens–Johnson syndrome, anaphylactic reactions, and the unusual type of cardiac arrhythmia called torsade de pointes. In these cases the diagnosis of causation may be supported by other factors, such as timing.
3. Other drug-event pairs In cases other than 1 and 2, single reports or small numbers of reports do not give good evidence of causation. In such cases a large database of anecdotal reports allows computation of the disproportionality between the frequency with which an adverse event is associated with a medication and the frequency with which the same event is associated with all other medications (the proportional reporting ratio). If there is a significant disproportionality, the drug-event pair can be regarded as a pharmacovigilance signal.

A signal [of disproportionality] has been defined by CIOMS VIII [6], following Hauben & Aronson [7], as “information that arises from one or multiple sources (including observations and experiments), which suggests a new potentially causal association, or a new aspect of a known association, between an intervention and an event or set of related events, either adverse or beneficial, which is judged to be of sufficient likelihood to justify verificatory action”.
Thus, a signal, which can arise from more than source of information, does not prove causation, but merely provides statistical evidence of a possible association; when a signal is detected, further studies are needed to support causation.


1. World Health Organization. The Importance of Pharmacovigilance—Safety Monitoring of Medicinal Products. Geneva: WHO, 2002.
2. ICH E2E Harmonized Tripartite Guideline on Pharmacovigilance Planning. Step 4 version dated 18 November 2004. http://www.ich.org.
3. Aronson JK, Ferner RE. Clarification of terminology in drug safety. Drug Saf 2005; 28(10): 851-70.
4. Nisbett RE, Borgida E. Attribution and the psychology of prediction. J Pers Soc Psychol 1975; 32(5): 932-9.
5. Hauben M. Aronson JK. Gold standards in pharmacovigilance: the use of definitive anecdotal reports of adverse drug reactions as pure gold and high grade ore. Drug Saf 2007; 30(8): 645-55.
6. Council for International Organizations of Medical Sciences. CIOMS Working Group VIII on Application of Signal Detection in Pharmacovigilance (CIOMS VIII). http://www.cioms.ch/index.php/2012-06-10-08-47-53/working-groups/working-group-viii.
7. Hauben M, Aronson JK. Defining ‘signal’ and its subtypes in pharmacovigilance based on a systematic review of previous definitions. Drug Saf 2009; 32(2): 99-110.


Norbert Benda (BfArM): Challenges for a Decision-Theoretic Framework in Drug Safety and Benefit Risk Assessments

Regulatory decisions on drug approval or post-approval revisions as suspensions and variations are based on a number of different sources of evidence, as randomized clinical trials during drug development, randomized and non-randomized post-approval trials, observational studies or spontaneous reporting, aiming at a conclusion on a causal relation between drug and outcome. Whereas decisions based on selected outcome might be misleading and a non-standardized process obscures the properties of the decision procedure, standardisation and decision-theoretic considerations appear difficult due to the heterogeneity of the underlying evidence. Nevertheless, identifying the goal of the decision in relation to the expected outcome for the patient to be treated as well as a certain degree of consistency appears paramount while still considering the cautionary principle to be applied to safety signals.
The presentation will discuss principles for a decision-theoretic framework in drug safety and benefit risk assessments, the related challenges and consequences and oppose Bayesian and frequentist reasoning in risk assessment.


Maria Luisa Casini and Pasquale Marchione (AIFA): Spontaneous Reporting in Europe: The Point of View of Regulators

National competent authorities’ activities related to signal analysis are strictly regulated in Europe. Although the basis of the procedures in place relies on ICHs ones, Directive 2010/84/EU and Regulation (EU) No 1235/2010 have given further indications on how to perform it. An overview on the issue is given, with special focus on the practical problematic encountered by regulators in performing signal analysis.


Ralph Edwards (Uppsala Monitoring Center, WHO): Causality in Pharmacovigilance: Small Chances – Big Problems

Most individual drugs are a rare cause of clinically important harm, but the very many different products available, and their widespread use, results in a global public health concern.
Causal involvement by drugs in harm is multifaceted, ranging from causation by a single drug, through contingent and contributory causation in therapy, to problem of misuse and medication errors: the causal chain may be very complex and multiple methods are needed to determine causality using iterative Bayesian approaches.
Clinicians need useable information to help them prevent causing harm in risk situations; to enable early diagnosis; and to manage patients. All vectors impinging on individual causation need exploring.


Bennett Holman (Yonsei University) and Justin Bruner (Australian National University): Experimentation by Industrial Selection

Industry is a major source of funding for scientific research. There is also a growing concern for how it corrupts researchers faced with conflicts of interest. As such, the debate has focused on whether researchers have maintained their integrity. In this paper we draw on a case study of the estimation of harmful side-effects and formal modeling to argue that given methodological diversity and a merit-based system, industry funding can bias a community without corrupting any particular individual. We close by considering a policy solution (i.e., independent funding) that may seem to promote unbiased inquiry, but which actually exacerbates the problem without additional restrictions.


Brigitte Keller-Stanislawski (Paul Ehrlich Institut and PRAC): Benefit Risk Evaluation of Medicinal Products Post-Authorisation and Evidence Based Decision Making in Pharmacovigilance

A medicinal product is authorised based on the judgement of a positive benefit risk balance in the specified indication in the target population. Randomized controlled trials (RCTs) are considered to be the gold standard for assessing the efficacy of drugs. This is not necessarily the case for safety where inadequate power to detect either multiple or rare but serious adverse events is a major obstacle. Furthermore, conditions under which drugs are approved for market use are often different from the settings in actual use such as duration of exposure, co-morbidity of the target population and/or concomitant use of other medicinal products. Thus, not all risks will have been identified at the time of marketing authorisation and many of the risks associated with the use of a medicinal product will only be discovered and characterised post-authorisation. Usually a medicinal product is associated with multiple risks and individual risks will vary in terms of severity, incidence, effect on individual patients and public health impact. Hence post-marketing surveillance and risk management plan-based activities are crucial for proportionate decision making by regulatory authorities
Consequently, clinical trials prior to marketing authorization constitute only a part of the research that goes into assessing the safety of drugs and observational studies are increasing for safety evaluation of medicinal products, although such studies generally provide less compelling evidence than RCTs.
Balancing the benefits and risks of a drug post authorisation is a complex process based on best available evidence. By analysing data from various sources limitations such as heterogeneity of studies and potential confounders need to be considered.
The presentation will provide aspects of the European Union (EU) regulatory approach for benefit risk assessment post-authorization and an overview on the EU legal framework in the European Union for decision making.top

Mike Kelly (Cambridge University): EBM, Reductionism, and the Road to Medicine’s Adaptive Pathways

There has been some discussion and debate in the last several years about the development of Medicines Adaptive Pathways to Patients (MAPP). The development of MAPP has been driven in part by the desire to make new technologies available more quickly to patients especially in the case of rare diseases, but also by a recognition that the RCT as a basis for decision making in the era of genomics and personalised medicines may be nearing the end of its shelf life. The debate has focused on the role of the pharmaceutical industry especially its apparent desire to push back against the gold standard of the RCT and of the scientific limits of studies which are not RCTs. The dangers to patients of basing recommendations on anything other than RCTs has been highlighted. This paper will explore these developments and will offer an account of the general development of Evidence Based Medicines in which MAPP is viewed as a phase that draws upon methodological and scientific progress in the production of public health and clinical guidelines. It will be suggested that by exploring the philosophical and methodological provenance of MAPP we can describe a narrative of EBM and Health Technology Assessment in which “normal science” can be seen to be operating rather than as some have proposed, a paradigm shift.top

Adam La Caze (University of Queensland): Evaluating Evidence in Drug Safety

This talk will examine three approaches to evaluating drug safety evidence. The method-focused approach is the dominant approach to evidence evaluation in medicine. On this approach, a body of evidence is evaluated according to the method used to generate the evidence. This approach aims to reduce the risk of erroneously inferring a causal link between drug and effect by focusing on reliable methods. The second approach to evidence evaluation focuses on inferentialist accounts of causal assessment. Julian Reiss's inferentialist account and Russo and Williamson's account of epistemic causality will be considered. A key insight of Russo and Williamson's approach is that causal assessment consists in seeking evidence of difference-making and evidence of mechanisms. The third approach is the Bayesian epistemological account provided by Jürgen Landes, Barbara Osimani and Roland Poellinger. This approach provides a formal model of scientific inference for amalgamating multiple lines of drug safety evidence.
Each account will be assessed against contemporary drug safety cases and the following criteria: Does the account provide appropriate guidance in drug safety cases? Does the account accurately describe the way regulators and clinicians evaluate drug safety evidence? Can the account be employed by regulators and clinicians? This talk will:
(1) Identify the reasons why the method-focused approach frequently provides poor advice for evaluating drug safety evidence;
(2) Identify some challenges for the application of the Bayesian epistemological framework to specific drug safety cases; and
(3) Argue that the inferentialist causal approach provides an appropriate general account of the evaluation of evidence regarding drug effects. Indeed, the method-focused approach is best situated within this account and applied exclusively to the assessment of drug efficacy.top

Jürgen Landes (MCMP/LMU Munich): Variety of Evidence

The Variety of Evidence Thesis is taken to state that varied evidence speaking in favor of a hypothesis confirms it more strongly than less varied evidence, ceteris paribus. This epistemological thesis enjoys widespread intuitive support. Its evidential character makes it highly amenable to a Bayesian analysis. I here give such an analysis and thus put forward Bayesian models of inquiry in which I explicate variety of evidence and subsequently show that our explication of the Variety of Evidence Thesis holds in all models presented. These models also pronounce on disconfirmation and discordant evidence; I argue that they do rightly so. The case for the Variety of Evidence Thesis emerges strengthened.

Ulrich Mannsmann (IBE/LMU Munich): How to Quantify the Effect of and Correct Design-dependent Bias in RCTs

This talk will introduce in the concept of meta-epidemiological studies on RCTS as introduced by Lesley and al. (BMJ 2008;336;601-605).The Lesley paper provided for the first time a quantitative estimate for bias in efficacy estimate depending on inappropriate blinding, randomization or use of endpoints. Several years later there was a validation study by Savovic et al. (Ann Intern Med. 2012;157:429-438) which confirmed the findings of Lesley et al. on a broader data base. Besides presenting these ideas and discussing their consequences for added value discussions, the talk gives also insight in the methodology on how to derive these measures (ROR - relative odds ratios). Finally, strategies are presented on how to adjust results of meta-anayses with respect to hidden sources of bias. They rely on the concept of Welton et al (JRSS-A 2009 172:119–136) for bias-adjusted treatment effect estimates in a new meta-analysis.top

Stephen Mumford (University of Nottingham): Evidence Synthesis for Dispositionalists

Dispositionalists are in a position to challenge the meta-science that informs medical research including assumptions that inform the modelling of probability and weighting of evidence. There are reasons for dispositionalists to reject the classical mathematisation of probability, for example, which measures it on a bounded scale. The causal powers of nature can overdispose, and the relationship between extent of power and degree of probability is asymptotic. What applies to degree of power also applies to degree of belief, which is classically modelled as having a maximal value =1. In contrast, a dispositionalist need not accept that certainty is achieved only at that point; indeed, there are cases where there is more than enough reason to believe a theory or claim. This point transfers over to the synthesis of evidence. A dispositionalist is able to allow an evidence hierarchy, in that not all forms of evidence are equally good at indicating causal connections. The hierarchy is not strict, however, but dispositional: better evidence tends to get causation right more than weaker evidence. The weighting of evidence also cannot just be about slices of a pie, all of which add up to 1. Given the disruptive influence of dispositionalist thinking on these standard conceptions of evidence and probability, one might wonder whether it looks bad for dispositionalism. But there are good, independent motivations for the theory and its take on these issues.top

Barbara Osimani (MCMP/LMU Munich): A Multilayer Approach to Modeling Probabilistic Causal Inference through Evidence Synthesis

More than any other scientific domain, pharmacology is at the crossroads of heterogeneous aims and interests. Standard methodology and deontology have more or less obliquely taken into account such conflicts, by developing guidelines for drug development, approval and monitoring, which incorporate concerns about reliability, bias and cost-effectiveness. Both foundational questions as well as methodological analysis must therefore take these broader dimensions into account. In particular, current practices and decision-making models are rather narrow-focused and lack a comprehensive view on the complex network of interests (financial, reputational etc.), as well as legal rights and duties which frame the scientific and social ecosystem in which pharmacology is embedded.
I present an inferential framework for the purpose of probabilistic causal assessment developed together with Jürgen Landes and Roland Poellinger within the ERC project: “Philosophy of Pharmacology: Safety, Statistical Standards, and Evidence Amalgamation” (Landes et al. forthcoming; Osimani and Landes, forthcoming; Poellinger, forthcoming). This consists in a Bayesian network specifically adapted to model epistemic dynamics in probabilistic causal assessment in a three-layer perspective:
1. A basic level of evidential support to the hypothesis at hand (and various evidence synthesis techniques);
2. A higher order level of “meta-evidential” dimensions related to the body of evidence itself: coherence of reports, (in)dependence structure; reliability, relevance;
3. A further level related to the information/evidence concerning these meta-epistemic dimensions (e.g. financial interests, reputation concerns, legal constraints).
These levels have been working relatively independently so far, especially in the standard Evidence Based Medicine approach. Our project goes in the “meta-analytic” direction advocated by Gelman (2015) and offers at the same time a higher order perspective on various philosophical debates on methodological pluralism, reliability, replication, causal holism, relevance and external validity, by effectively embedding the related epistemic dimensions in a concrete topology.


[1] Gelman, Andrew. Working through some issues. Significance 12.3 (2015): 33-35.
[2] Jürgen Landes, Barbara Osimani, and Roland Poellinger. Epistemology of Causal Inference in Pharmacology. European Journal for Philosophy of Science, 2017. Forthcoming.
[3] Barbara Osimani and Jürgen Landes. Exact replication or varied evidence? The Varied of Evidence Thesis and its methodological implication in medical research. 2017. Submitted to Synthese.
[4] Poellinger, Roland. Analogy-Based Inference Patterns in Pharmacological Research. Submitted to Uncertainty in Pharmacology: Epistemology, Methods, and Decisions (Boston Studies in Philosophy of Science).


Roland Poellinger (MCMP/LMU Munich): Shaping Causal Claims in Pharmacology: The Interplay between Population Characteristics and Causal Structure

This talk builds on a Bayesian evidence-amalgamation framework (Landes et al., forthcoming) to formally explore the interplay between heterogeneous evidence and the different components of a causal hypothesis in pharmacological risk assessment. For this purpose, causal hypotheses are explicated as a four-place relation between cause, effect, target population, and causal structure in order to utilize formal explications of similarity and analogy for the evaluation of the relevance of given evidence for the investigated hypothesis. Relating the causal hypothesis and different sources of evidential support in an epistemologically interpreted Bayesian network allows for “zooming in” onto the hypothesis and formally explicate its components in a causally interpreted Bayesian network. In this talk I will discuss how embedding such a causal structure in the layered evidence-amalgamating network facilitates (i) locating causal co-factors and ceteris-paribus population characteristics, and (ii) explicating how cumulating relevant evidence may shape causal claims in pharmacology.top

Martin Posch and Franz König, Vienna (Section for Medical Statistics, Center for Medical Statistics, Informatics and Intelligent Systems, Medical University of Vienna): Challenges of Optimizing Drug Development Programs for Regulatory Decision Making

Decisions on the authorisation of medicines are based on context knowledge and information generated in (pre-) clinical trials and observational studies. Approaches for evidence synthesis are applied to structure, weigh and combine information from different sources to either make a decision on authorisation or withdrawal of medicines, or to request additional data. Decision makers are confronted with a range of challenges in data interpretation. While the efficacy of drugs can usually be measured with few pre-defined endpoints in confirmatory trials, the assessment of safety is based on high dimensional data sets from controlled trials as well as observational studies. The resulting multiplicity problem and potential estimation bias pose a major difficulty to reliably assess the characteristics of medicines for a sound benefit risk analysis to evaluate the “totality of the data”. Currently, benefit risk evaluation is mainly based on a qualitative assessments based on information from clinical trials that are summarized by quantitative methods [1]. While decision makers are aware of challenges as multiplicity (due to multiple studies, endpoints, subgroups and analysis time points), small samples, the variability of estimators and sources of bias, extrapolation between populations (e.g., from adults to children) or compounds, it is questionable if a qualitative approach is fit to appropriately take all these factors into account. The question remains if quantitative approaches as multicriteria decision analysis, Bayesian decision analysis optimizing utility functions or value of information analysis can be useful when designing, conducting and interpreting clinical trial programs and regulatory decision making. These methods require the definition of utility functions that assign quantitative values to different outcomes (as response, adverse events, etc.) and weigh the chance of false positive and false negative decisions. This poses a further challenge as the objectives of different groups of stakeholders as patients, regulators, payers, industry will differ, and will not be homogenous even within each group.


[1] Committee for Medicinal Products for Human Use. "Reflection paper on benefit-risk assessment methods in the context of the evaluation of marketing authorisation applications of medicinal products for human use." (2009).top

Felipe Romero and Jan Sprenger (University of Tilburg): Making Scientific Inferences More Objective: Replication and Scientific Self-Correction

In the ERC project "Making Scientific Inferences More Objective," we work on calibrating recent philosophical advances in the analysis of scientific objectivity with scientific practice in statistical, causal, and explanatory reasoning. As part of the project, we are concerned with replicability failure. Replication is central to scientific self-correction, but many findings in the behavioral sciences and biomedical research do not replicate. We are currently evaluating two general approaches to increase replicability. Social reformists hypothesize that changes in inference methods alone do not make science more self-corrective unless we change the social structure of science. On the other hand, methodological reformists hypothesize that scientific self-correction would improve by changing the statistics, moving from significant tests to Bayesian inference. This talk discusses these two approaches and presents preliminary work in finding a middle ground between the social and methodological reforms.


Stephen Senn (CCMS, Luxembourg Institute of Health): What Randomisation Can and Cannot Do for You

In a paper of 1938 on agricultural research describing methods for what we would now call evidence synthesis or meta-analysis, Yates and Cochran (1) contrasted scientific and technical research, describing the former as being easy compared to the latter.
I shall explain the relevance of their distinction to clinical research, claiming that as regards the former the contribution of randomisation is regularly misunderstood. In the context of scientific research, or causal analysis, I shall cover some fallacious critical arguments, in particular that of there being indefinitely many confounders. As regards technical research, or practical prediction, I shall explain how randomisation is not enough but not therefore irrelevant and that a combination of randomised studies, good scales of analysis and auxiliary data can help improve predictions and hence contribute to an important task of technical research: making practical recommendations that take risks and benefits into account.


1 Yates, F. & Cochran, W. G. The analysis of groups of experiments. Journal of Agricultural Science 28, 556-580 (1938).top

Beth Shaw (The National Institute for Health and Care Excellence): Assessing Drug Safety – What Can We Learn From Other Assessments of Harm

The use of integrated and combined information from different sources (for example, spontaneous case reports, literature, data-mining, pharmaco-epidemiological studies, post-marketing trials, drug utilization studies, non-clinical studies, late-breaking information) is now being encouraged for assessing drug safety. There is potentially great value in integrating different types of knowledge; however, there are challenges and unanswered methodological questions.
This presentation will outline the use of a range of knowledge in guideline development and some challenges and potential solutions with this approach.
Good guidelines often need to draw upon a range of knowledge sources: RCTs may be unavailable or not even provide the most suitable knowledge. Although a range of useful tools has been developed for grading RCTs and standard evaluations of intervention effectiveness – including benefits and harms - (primarily based on frequency based reasoning), methods for appraising and including knowledge from other sources are in earlier stages of development. Exploring alternative types of reasoning and making valid inferences may offer a solution.

  • outline the range of knowledge that is used in guideline development, with a focus on assessment of harms in public health and social care
  • link the use of different knowledge with different reasoning processes
  • explore how the use of different knowledge contributes to decision making in guideline development
  • outline areas for methodological development.


Jacob Stegenga (Cambridge University): Bayesian Mechanista

There are two radical views regarding the role of mechanisms in causal inference. One holds that causal inference, at least in medicine and the social sciences, should be based only on data from population-level studies (statistical evidence). The other holds that causal inference must be based in part on mechanistic evidence. This paper appeals to Bayesian confirmation theory to defend a middle view, and explains why the arguments for both sides can seem compelling. The competing views are local principles of inference, the plausibility of which can be assessed by a general normative principle of inference. The Bayesian tells us to base inferences on both the likelihood and the prior. The likelihood represents statistical evidence. One influence on the prior probability of a hypothesis like “d does x” is knowledge of how d does x. Thus, reasoning about causal relations by appealing to both statistical and mechanistic evidence is vindicated by our best general theory of inference.


David Teira (UNED Madrid): Evidential Pluralism and Regulatory Paternalism in Drug Testing

The recent passage of the 21st Century Cures Act (21CCA) in the United States will force the Food and Drug Administration to adopt a pluralistic stance about proofs for safety and efficacy for new medical treatments. Since 1962, two positive randomized clinical trials (RCTs) were a pre-requisite for FDA market approval. The 21CCA pushes the FDA to use instead “complex adaptive and other novel trial designs”, plus sources of “real world evidence, including ongoing safety surveillance, observational studies, registries, claims, and patient-centered outcomes research activities”. In parallel, the 21CCA will encourage federal agencies and health providers to use electronic health records systems and to collect this sort of data. Philosophers of science have been arguing for this sort of evidential pluralism for over a decade. But critics of the 21CCA warn about the potential exploitation of this pluralism in the best interest of the pharmaceutical industry.
In this presentation, I will argue that evidential pluralism involves a de facto relaxation of regulatory paternalism. Under the 1962 FDA Act, patients could only access treatments that met the most demanding standard of safety (the RCT). Under the 21CCA, physicians and patients will have try for themselves the reliability of all these new sources of evidence about drug safety. I contend that, even if patients are willing to take their chances, this relaxation of paternalism is only acceptable if the risks of each treatment are genuine. I.e., all the stakeholders have the same uncertainty about the treatment effects. The evidential pluralism of the 21CAA may easily generate dangerous asymmetries of information: with the proliferation of clinical data and testing methods, how is a patient to find which evidence serves best her interest?top

Jon Williamson (University of Kent): EBM+: Improving the Way in Which Evidence-Based Medicine Handles Evidence of Mechanisms

This talk will introduce the EBM+ consortium, which seeks to makes the role of evidence of mechanisms in the evidence appraisal process more explicit. It will explain why evidence of mechanisms needs to be taken into account and will also introduce two related research projects: Evaluating evidence in medicine, funded by the UK Arts and Humanities Research Council, and Grading evidence of mechanisms in physics and biology, funded by the Leverhulme Trust.