The present study involved pooled analysis of individual data from 12 population-based Spanish cohort studies performed from 1991onward 10 and with follow-up until 2005 aggregated in the FRESCO study11 (Table 1 of the supplementary material). All cohort participants were randomly selected, were aged between 35 and 79 years, had no cardiovascular disease at the start of follow-up, and had signed an informed consent form. The FRESCO study was authorized by the Ethics Committee of the Parc de Salut Mar, Barcelona (2009/3391/I).

Age, sex, and the other risk factors considered in the SCORE functions (smoking, systolic blood pressure, total cholesterol, and high-density lipoprotein cholesterol [HDL-C]) were known for all cohorts and had been collected using the standardized methodology recommended by the World Health Organization.10 Participants were classified as smokers (if they were smoking at the time of examination or had quit smoking ≤ 1 year previously) or nonsmokers (exsmokers since > 1 year previously or never smoked). Systolic and diastolic blood pressure readings were determined via the mean of 2 determinations obtained at least 5minutes apart. Analytical determinations were performed after participants fasted for 10 to 14hours. Diabetes mellitus diagnosis was established from a previous diagnosis or a baseline blood glucose level > 125mg/dL. In 9 of the 12 cohorts, a concordance study of total cholesterol and HDL-C was performed, with satisfactory results.12 The present validation assessment specifically considered the population for whom the SCORE function was designed: age 40 to 64 years, systolic blood pressure ≥ 100 and ≤ 180mmHg, and total cholesterol ≥ 105mg/dL (3 mmol/L) or ≤ 305mg/dL (8 mmol/L) (Figure 1).

Figure 1.

Flow diagram of participants in the FRESCO study.

(0.21MB).

Participant follow-up was at least 10 years for the following fatal cardiovascular disease codes of the ICD-9 (International Classification of Diseases, Ninth Edition), or their equivalents in the 10th revision, considered in the SCORE function5: from 401 to 414 and from 426 to 443, excluding 426.7, 429.0, 430.0, 432.1, 437.3, 437.4, and 437.5. In addition, death from cardiovascular causes5 was classified according to the following ICD-9 codes: 798.1 (instantaneous death) and 798.2 (sudden death occurring less than 24hours after symptom onset). Cause of death was ascertained from medical records, hospital discharge sheets, and the national death registry and that of each autonomous community.

Analyses were stratified by sex. Continuous variables are reported as mean ± standard deviation and categorical variables as proportions. Overall mortality rates and those according to cause (cardiovascular, noncardiovascular) were estimated using Kaplan-Meier analysis.

The 10-year risk of cardiovascular death was estimated with the SCORE-LOW function with Weibull model coefficients.5 The risk was also estimated with the SCORE-LOW-HDL and SCORE-C functions8 using the HeartScore calculator of the European Society of Cardiology9 because their coefficients are no longer published.7 The free software AutoIt13 was used to automatically add to the corresponding field of the calculator the values of the variables–sex, age, smoking, systolic blood pressure, total cholesterol, and HDL-C (when necessary)–taken from the database. The same program captured the risk estimated by the calculator for the SCORE-LOW-HDL and SCORE-C functions and incorporated this value into the database.

To validate the functions, the observed-to-expected mortality ratio was first estimated with each of the 3 risk functions. To determine the reliability of the SCORE-LOW function,5 its discrimination capacity was analyzed using the area under the ROC (receiver operating characteristic) curve.14 Accuracy was evaluated via a comparison of the risks observed in the cohort and those estimated by the SCORE functions using the Hosmer-Lemeshow test modified for cohort studies with SCORE-LOW risk sextiles (calibration).15 No ROC curves could be evaluated for the SCORE-LOW-HDL and SCORE-C because the HeartScore calculator rounds risk figures to whole numbers.9 Nonetheless, the calibration was analyzed by comparing the observed events and those predicted with 3 whole cutoff points (1%, 2%, and 3%).

The risk estimated with the 3 functions was multiplied by 3 and 5 in men and women with diabetes, respectively, according to the indications of the most recent clinical practice guidelines of the European Society of Cardiology.3 In addition, in a sensitivity analysis, the cardiovascular risk estimated with the SCORE-LOW function was multiplied by 2 and 4 in men and women with diabetes, respectively, as indicated in the reference publication.5

The population distribution was calculated in 4 risk groups defined by 1%, 5%, and 10% cutoff points with the 3 versions of the SCORE function and the percentage of cardiovascular deaths observed in each group.

All analyses were performed using R statistical software version 2.10 (R Foundation for Statistical Computing, Vienna, Austria).

Most European population-based studies analyzing the validity of the high-risk SCORE function indicate that estimates of risk of cardiovascular death are significantly higher in both sexes than the observed rates in the Netherlands,18 Denmark,19 Norway,20 Germany (2 studies),21,22 Iceland,23 and the United Kingdom.24 In all of these studies, SCORE approximately doubled the true risk of cardiovascular death. In an Australian population, who are mainly of European origin, the risk was overestimated by 75% in women.25

The risk estimation with SCORE-LOW was 40% higher than that observed in the nondiabetic population in Extremadura, Spain.26 This function has also been evaluated in high-risk countries such as the Netherlands18 and Iceland,23 where the risk of cardiovascular death was correctly estimated. However, in Germany21 and the United Kingdom,24 this function overestimated the risk by 31% and 25%, respectively. In Norway,20 the results were acceptable in men, but the risk was overestimated in women (observed-to-expected mortality ratio: 0.69 in patients 50 to 59 years old and 0.56 in those 60 to 69 years old), with similar findings from Australia (16% overestimation).25

In agreement with our results, a cross-sectional study performed in the Catalan population showed that the SCORE-C function (particularly in men) classified a higher percentage of patients as high risk vs the SCORE-LOW and SCORE-LOW-HDL functions.27 In addition, a risk overestimation was seen with the SCORE function calibrated for the Netherlands18 (approximately double), whereas the estimate was adequate in Germany.22 In Denmark, the calibrated function failed to improve the validity of the SCORE-LOW function.19

Overall, the 3 versions of the SCORE function analyzed provided significantly different cardiovascular risk estimates from those actually observed in the Spanish population. In women, 80% of fatal cardiovascular events were concentrated in the low-risk groups of the SCORE-C and SCORE-LOW-HDL functions and in the intermediate- and low-risk groups of SCORE-LOW, whereas 80% of events in men were concentrated in low- and intermediate-risk groups with SCORE-C and SCORE-LOW-HDL and 65% with SCORE-LOW. This observation indicates that the cutoff points defining low, intermediate, and high risk should be revised downward. Definition of the cutoff points is limited by the absence of decimal places in the estimates of the European Society of Cardiology calculator for the SCORE-C and SCORE-LOW-HDL functions.9

The results of the present study suggest that all SCORE versions5,8,9 have significant limitations in the accuracy of the estimates obtained in the Spanish population. It was additionally confirmed that most cardiovascular events occurring in Spain are not fatal.11 The accuracy of tables based solely on cardiovascular mortality can vary over time, given that this type of mortality has greatly decreased in the 3 decades since the development of the SCORE cohorts.2,5 Similarly, the exclusion of individuals aged from 65 to 74 years is also clinically relevant because the rate of death from cardiovascular causes in this population is double that of the individuals included in our cohort (40-64 years). In a Danish study, the authors suggested that future guidelines should focus on individuals older than 65 years because most fatal cardiovascular events occur in this population.19 Recently, a version of SCORE Older People was published, which permits estimation of the risk of cardiovascular death in individuals 65 to 74 years.28 For its derivation, the authors used the SCORE cohorts of Norway (mainly; 86.9% of the sample included in the derivation), Italy, Belgium, and Denmark. In Spain, the SCORE Older People tables showed a lower risk than that estimated using SCORE for persons older than 65 years.29 It is thus reasonable to also promote the use of the functions validated in Spain: the Framingham-Wilson calibrated by REGICOR4,16,17 and by FRESCO,11 which also increases the upper age limit to 79 years.

To perform effective prevention of cardiovascular diseases, the models for predicting individual risk should follow the TRIPOD guidelines.30 These recommendations promote transparency in the publication of the methods used in order to facilitate their evaluation.

The FRESCO study combines data from 12 Spanish cohorts that are representative of the populations of different reference areas recruited between 1991 and 2004. The study has a large sample size, exhaustive 10-year follow-up of cardiovascular events, standardized data collection methodology for component cohorts, and broad representativeness of the Spanish population.11 Despite the robustness of these characteristics, there may be some residual heterogeneity among study components. First, the coefficients of the SCORE-C and SCORE-LOW-HDL functions were not available because they have not been published.8 Accordingly, the use of the online HeartScore calculator was required, even though it was designed for the health care setting9 and rounds results to whole numbers. The absence of decimal places prevented calculation of the area under the curve of the 2 functions. This limitation also affected the calibration analysis, which was restricted to 4 risk groups instead of the risk sextiles and deciles (such as in the case of the SCORE-LOW function) used classically to evaluate the accuracy of the cardiovascular risk functions.15,16