Publish in this journal
Journal Information
Vol. 73. Issue 8.
Pages 689 (August 2020)
Vol. 73. Issue 8.
Pages 689 (August 2020)
Letter to the Editor
Full text access
Designs and methods for impact evaluation of interventions
Diseños y metodologías para evaluar el impacto de las intervenciones
Antonio Sarria-Santameraa,b
a Department of Medicine, Nazarbayev University School of Medicine, Nur-Sultan, Kazakhstan
b Red de Investigación en Servicios de Salud y Enfermedades Crónicas (REDISSEC), Bilbao, Biscay, Spain
Related content
Guillermo Aldama, Manuel López, Melisa Santás, Xacobe Flores, Pablo Piñón, Jorge Salgado, Ramón Calviño, Nicolás Vázquez, Alejandro Mesías, Carlos González-Juanatey, Javier Muñiz, José Manuel Vázquez
Rev Esp Cardiol. 2020;73:69010.1016/j.rec.2020.02.013
Guillermo Aldama, Javier Muñiz
Article information
Full Text
Download PDF
Full Text
To the Editor,

The article by Aldama et al.1 describes the main components and outcomes of PROGALIAM. The question we must ask ourselves is if the changes observed are due to the intervention. In this case, the authors concluded that PROGALIAM had “achieved its 2 main goals”: increased survival and better access to primary revascularization for patients with ST-elevation acute myocardial infarction. They also reported a significant link between reduced mortality and increase in primary percutaneous coronary interventions. These results are in line with those published from the RECALCAR project,2 which found lower mortality in regions that had comprehensive programs for the management of myocardial infarction.

We believe that the regression analysis in the RECALCAR study, and the hazard ratio and before-after design of the study that assessed PROGALIAM are not suitable to determine a causal relationship with certainty.

Hazard ratios have a limitation: it cannot be assumed that the ratio will remain constant for the whole follow-up period,3 especially when the confounding variables may be unbalanced between the intervention and control groups.4 In the case of PROGALIAM, differences were observed in the clinical characteristics of the patients and the presentation of ST-elevation acute myocardial infarction in the periods before and after its implementation. The use of historic controls5 in observational studies increases this problem, due to the possible secular changes in medical practice: bias may increase with longer study duration.

Randomized clinical trials are the reference standard for clinical research to compare the average outcome with one treatment (the factual) with the average outcome in patients who do not receive the treatment (the counterfactual). Assessing the impact of interventions (such as PROGALIAM) to determine if the changes observed are due to the intervention and its efficacy involves a counterfactual analysis: comparing what happened with what would have happened in the absence of the intervention when such an experimental design would be unfeasible or unethical. The main challenge in assessing this impact is finding a good counterfactual – a reasonable, convincing comparator group.

Possible methods that provide a more robust counterfactual causal inference6 than before-after study designs are propensity scoring,7 difference-in-difference,8 instrumental variables,9 and regression discontinuity.10 These designs are carried out in nonexperimental situations and with systematically-collected data.

Stepped wedge designs also provide a way around the practical or ethical limitations of experimental evaluation of interventions.11 In these studies, several clusters are successively included over time (this timing may even be randomized) to receive the intervention. Eventually, the whole population receives the intervention, but the integrated randomization over time provides intervention and control groups.

These methodological approaches can generate valid evidence for evaluation of the effect, overcoming the inherent limitations of clinical research designs in such situations and reducing the bias in estimations on the benefit of these interventions in real clinical practice.


Red Temática de Investigación en Servicios de Salud y Enfermedades Crónicas (REDISSEC RD16/0001/0019).


G. Aldama, M. López, M. Santás, et al.
Impacto en la mortalidad tras la implantación de una red de atención al infarto agudo de miocardio con elevación del segmento ST Estudio IPHENAMIC.
Rev Esp Cardiol., 73 (2020), pp. 632-642
Á. Cequier, A. Ariza-Solé, F.J. Elola, et al.
Impacto en la mortalidad de diferentes sistemas de asistencia en red para el tratamiento del infarto agudo de miocardio con elevación del segmento ST La experiencia de España.
Rev Esp Cardiol., 70 (2017), pp. 155-161
M. Hernan.
The hazards of hazard ratios.
Epidemiology., 21 (2010), pp. 13-15
M.J. Stensrud, J.M. Aalen, O.O. Aalen, M. Valberg.
Limitations of hazard ratios in clinical trials.
Eur Heart J., 40 (2019), pp. 1378-1383
S.J. Pocock.
The combination of randomized and historical controls in clinical trials.
J Chronic Dis., 29 (1976), pp. 175-188
R. Khera, Y. Wang, K. Nasir, Z. Lin, H.M. Krumholz.
Evaluation of 30-day hospital readmission and mortality rates using regression-discontinuity framework.
J Am Coll Cardiol., 74 (2019), pp. 219-234
V. Blaya-Nováková, F.J. Prado-Galbarro, A. Sarría-Santamera.
Effects of annual influenza vaccination on mortality in patients with heart failure.
Eur J Public Health., 26 (2016), pp. 890-892
J. Stokes, S.R. Kristensen, K. Checkland, P. Bower.
Effectiveness of multidisciplinary team case management: difference-in-differences analysis.
BMJ Open., 6 (2016), pp. e010468
M. McClellan, B.J. McNeil, J.P. Newhouse.
Does more intensive treatment of acute myocardial infarction in the elderly reduce mortality? Analysis using instrumental variables.
JAMA., 272 (1994), pp. 859-866
Directorate-General for Employment, Social Affairs and Inclusion (European Commission).
Design and Commissioning of Conterfactual Impact Evaluations. A Practical Guidance for ESF Managing Authorities.
Publications Office of the European Union, (2013),
A.K. Huynh, M.L. Lee, M.M. Farmer, L.V. Rubenstein.
Application of a nonrandomized stepped wedge design to evaluate an evidence-based quality improvement intervention: a proof of concept using simulated data on patient-centered medical homes.
BMC Med Res Methodol., 16 (2016), pp. 143
Copyright © 2020. Sociedad Española de Cardiología
Revista Española de Cardiología (English Edition)

Subscribe to our newsletter

View newsletter history
Article options
es en

¿Es usted profesional sanitario apto para prescribir o dispensar medicamentos?

Are you a health professional able to prescribe or dispense drugs?