Intended for healthcare professionals

  1. Research
  2. Effectiveness of mRNA...
  3. Effectiveness of mRNA-1273, BNT162b2, and BBIBP-CorV vaccines against infection and mortality in children in Argentina, during predominance of delta and omicron covid-19 variants: test negative, case-control study
CCBYNC Open access
Research

Effectiveness of mRNA-1273, BNT162b2, and BBIBP-CorV vaccines against infection and mortality in children in Argentina, during predominance of delta and omicron covid-19 variants: test negative, case-control study

BMJ 2022; 379 doi: https://doi.org/10.1136/bmj-2022-073070 (Published 30 November 2022) Cite this as: BMJ 2022;379:e073070

Linked Editorial

Effectiveness of covid-19 vaccines in adolescents and children
  1. Juan Manuel Castelli, infectious disease specialist1,
  2. Analia Rearte, professor, epidemiologist1 2,
  3. Santiago Olszevicki, data scientist1,
  4. Carla Voto, infectious disease specialist1,
  5. María Del Valle Juarez, epidemiologist1,
  6. Martina Pesce, epidemiologist1,
  7. Agustina Natalia Iovane, epidemiologist1,
  8. Mercedes Paz, epidemiologist1,
  9. María Eugenia Chaparro, epidemiologist1,
  10. Maria Pia Buyayisqui, epidemiologist1,
  11. María Belén Markiewicz, epidemiologist1,
  12. Mariana Landoni, information systems engineer1 3,
  13. Carlos María Giovacchini, epidemiologist1 3,
  14. Carla Vizzotti, minister of health1
  1. 1Ministerio de Salud de la Nación Argentina, Ciudad Autónoma de Buenos Aires, Argentina
  2. 2Escuela de Medicina, Universidad Nacional de Mar del Plata, Mar del Plata, Argentina
  3. 3Área BI Dirección Nacional de Sistemas de Información, del Ministerio de Salud de la Nación
  1. Correspondence to: A Rearte analiarearte{at}gmail.com
  • Accepted 2 November 2022

Abstract

Objective To estimate the effectiveness of a two dose vaccine schedule (mRNA-1273, BNT162b2, and BBIBP-CorV) against SARS-CoV-2 infection and covid-19 related death and short term waning of immunity in children (3-11 years old) and adolescents (12-17 years old) during periods of delta and omicron variant predominance in Argentina.

Design Test negative, case-control study.

Setting Database of the National Surveillance System and the Nominalized Federal Vaccination Registry of Argentina.

Participants 844 460 children and adolescents without previous SARS-CoV-2 infection eligible to receive primary vaccination schedule who were tested for SARS-CoV-2 by polymerase chain reaction or rapid antigen test from September 2021 to April 2022. After matching with their corresponding controls, 139 321 (60.3%) of 231 181 cases remained for analysis.

Exposures Two dose mRNA-1273, BNT162b2, and BBIBP-CorV vaccination schedule.

Main outcome measures SARS-CoV-2 infection and covid-19 related death. Conditional logistic regression was used to estimate the odds of SARS-CoV-2 infection among two dose vaccinated and unvaccinated participants. Vaccine effectiveness was estimated as (1–odds ratio)×100%.

Results Estimated vaccine effectiveness against SARS-CoV-2 infection was 61.2% (95% confidence interval 56.4% to 65.5%) in children and 66.8% (63.9% to 69.5%) in adolescents during the delta dominant period and 15.9% (13.2% to 18.6%) and 26.0% (23.2% to 28.8%), respectively, when omicron was dominant. Vaccine effectiveness declined over time, especially during the omicron period, from 37.6% (34.2% to 40.8%) at 15-30 days after vaccination to 2.0% (1.8% to 5.6%) after ≥60 days in children and from 55.8% (52.4% to 59.0%) to 12.4% (8.6% to 16.1%) in adolescents.

Vaccine effectiveness against death related to SARS-CoV-2 infection during omicron predominance was 66.9% (6.4% to 89.8%) in children and 97.6% (81.0% to 99.7%) in adolescents.

Conclusions Vaccine effectiveness in preventing mortality remained high in children and adolescents regardless of the circulating variant. Vaccine effectiveness in preventing SARS-CoV-2 infection in the short term after vaccination was lower during omicron predominance and decreasing sharply over time.

Trial registration National Registry of Health Research IS003720.

Figure1

Introduction

The worldwide vaccination strategy against covid-19 changed the evolution of the pandemic. The reduction of morbidity and mortality in adults, who were the first prioritized population given their high fatality rate, was remarkable. Unlike this population, infection in childhood and adolescence is usually mild, although severe cases with sequelae (for example, multisystem inflammatory syndrome) and even fatal outcomes have also been reported, especially in those with comorbidities.1 In Argentina, the paediatric population was tested with molecular methods (for example, reverse transcription polymerase chain reaction (RT-PCR)) and immunochromatographic assays (for example, rapid antigen test) (fig 1, top) and represented about 8% of the total number of confirmed cases, with a case fatality rate of 0.05%.2 Comparisons of effectiveness by diagnostic method for this population are not available.

Fig 1
Fig 1

Top panel: incident cases and positivity rate for covid-19 according to year, epidemiological week, and type of test. Middle panel: vaccination coverage according to year and epidemiological week by age group and dose applied. Bottom panel: distribution of circulating SARS-CoV-2 variants according to year and epidemiological week. Weeks 37-52 2021 and 1-16 2022. Ag=antigen; PCR=polymerase chain reaction. Vertical dotted line indicates change of variant predominance from delta to omicron

Notwithstanding that vaccination in this age group has generated multiple debates, it was recommended by international scientific societies such as the American Academy of Pediatrics in May 2021 and Argentina’s National Immunization Technical Advisory Group (NITAG).34 Moreover, in 2022, the World Health Organization determined that, despite the fact that it is not a priority group,5 progress should be made in the paediatric population in countries with high vaccination coverage in adults. Vaccination should aim not only for individual benefit but also for a collective benefit, as it would serve to increase overall population immunity and to reduce global transmission.

In Argentina, the vaccination strategy against covid-19 aimed to vaccinate 100% of the prioritised population in a gradual and progressive manner. The prioritisation was established according to the risk of severity of the disease, risk of exposure, and social vulnerability. Following the recommendations of the NITAG, vaccination of adolescents aged 12 to 17 years began in the second half of 2021, with the mRNA-1273 (Spikevax, Moderna) and BNT162b2 (Comirnaty, Pfizer-BioNTech) vaccines, followed by children aged 3 to 11 years who were vaccinated with BBIBP-CorV (Sinopharm) (fig 1, middle).6789

Although a systematic review found that low-middle income countries may have larger proportion of paediatric fatality due to covid-19 than high income countries,10 evidence on the effectiveness of vaccination, especially to prevent mortality in children and adolescents, is scarce.11121314 Some studies evaluated vaccine effectiveness for inactivated vaccines, but only one included BBIBP-CorV and only two reported effectiveness in preventing infection in children under 5 years old.15161718 The study that evaluated BBIBP-CorV vaccine effectiveness in children was conducted in Buenos Aires Province (Argentina) and estimated that vaccine effectiveness in preventing hospital admission in 3-11 year old children was 76.4% (95% confidence interval 62.9% to 84.5%). However, this study did not evaluate effectiveness for preventing infection or mortality or loss of effectiveness over time.17

Moreover, although several studies reported a decline in effectiveness over time with the BNT162b2 vaccine in children or adolescents, scarce evidence exists of changes in effectiveness with inactivated vaccines or of differences related to SARS-CoV-2 variants. Furthermore, despite the fact that studies on the effectiveness of covid-19 vaccines usually include cases diagnosed with both RT-PCR and rapid antigen tests, comparisons of effectiveness by diagnostic method for the paediatric population are not available.

Therefore, the aim of this study was to evaluate the effectiveness of two dose schedules of mRNA-1273, BNT162b2, and BBIBP-CorV vaccines in preventing SARS-CoV-2 infection and death related to covid-19 in children and adolescents, aged 3-17 years, during periods of delta and omicron BA.1 predominance between September 2021 and April 2022 in Argentina. Additionally, we aimed to assess the effect of the diagnostic method used for estimating vaccine effectiveness, the effectiveness of different combinations of mRNA vaccines, and short term waning of immunity in children and adolescents.

Methods

Study population, setting, and design

This study followed a test negative, case-control design. These designs attempt to provide control over confounding due to differential case ascertainment, access to care, and health seeking behaviour. They are considered powerful enough to estimate the effectiveness of vaccines and have been widely used for this purpose in SARS-CoV-2, influenza, and other viruses.1920

Argentina started vaccination of adolescents (aged 12-17 years) in August 2021 and of young children (aged 3-11 years) in October 2021. All participants aged 3 to 17 years who were tested for SARS-CoV-2 between 12 September 2021 and 23 April 2022 (considering the delta predominance period as until 25 December 2021 and the omicron predominance period as since then) (fig 1, bottom), without a previous positive test since 1 June 2021, were eligible for inclusion (fig 2). We restricted analyses to participants who sought RT-PCR or antigen testing in healthcare facilities or testing centres. We included participants only once in the study period, whether as a case or as a control. The first positive test in the study period was considered inclusion as a case in the primary analysis, regardless of the number of previous negative tests. Controls were those participants who tested only negative in the study period and were selected on the date of their first test. To assess vaccine effectiveness against covid-19 associated mortality, we included as cases only those participants who tested positive in the short time before death. Additionally, the National Epidemiology Department evaluated each case with the jurisdictional epidemiology department to corroborate the cause of death.

Fig 2
Fig 2

Flowchart of study population. RT-PCR=reverse transcription polymerase chain reaction. *Minimum 14 days between last dose and test. †Between 1 and 14 days from any dose to test; data inconsistencies

Data sources and definitions

The study used epidemiological surveillance data from the National Surveillance System (SNVS 2.0), preserving the confidentiality of the participants according to the Helsinki declaration and local regulations. The notification of suspected cases of covid-19 and their confirmation and outcomes were done by certified users (professionals, technicians, administrative and health authorities of the 24 districts) of the public healthcare, private healthcare, and social security subsectors. For the death registry, each district systematically reviewed and verified data from other death records, such as bureaus of vital records (death certificate data), hospitals, and funeral companies. These data were incorporated into SNVS 2.0. Vaccine information was reported in the Nominalized Federal Vaccination Registry (NOMIVAC), including all vaccinated participants’ date of vaccination, number of doses, type of vaccine, and vaccination centre. We followed the STROBE checklist.21

We considered 3-11 year old children to be fully vaccinated if they had received two doses of BBIBP-CorV and had received the second dose a minimum of 14 days before testing. We considered 12-17 year old adolescents to be fully vaccinated if they had received two doses of a homologous or heterologous schedule with BNT162b2 and/or mRNA-1273 and had received the second dose a minimum of 14 days before testing. Individuals who did not meet these criteria were included in the analyses in this article in three additional categories: those who had received only one dose a minimum of 14 days before testing (partially vaccinated), those who received three doses with the same minimum interval, and others. The third category includes those who were vaccinated in the two weeks before their test, those who received another vaccine, and those with inconsistent or incomplete vaccination data. Information about variants could not be established at an individual level, and we used the prevalent variant from surveillance data when analysing our results.

Outcomes

The primary outcome of this study was SARS-CoV-2 infection, confirmed by antigen or RT-PCR testing. The secondary outcome was death related to SARS-CoV-2 infection.

Statistical analyses

We matched cases and controls one to one on province of residence, week of testing, type of test (antigen or genomic), presence of comorbidities, sex, and age (±1 year). Ties during matching were automatically and randomly resolved by the algorithm. Matching improves statistical efficiency in case-control studies and yields unbiased estimates when a model accounting for matching, such as conditional logistic regression, is used. The effect of the matching factors on the outcome cannot be estimated in matched sets and thus is a considerable disadvantage of matching. To compare characteristics between cases and test negative controls, we did descriptive analyses for the unmatched and matched cohorts. P values shown refer to Student’s t test, Mann-Whitney U test, or Kruskal-Wallis rank sum test for differences in numerical variables or χ2 tests for proportions.

We used conditional logistic regression to compare the odds of SARS-CoV-2 infection as a function of vaccination status, taking the matching pairs into consideration. We estimated vaccine effectiveness as (1–odds ratio)×100% and presented it with its corresponding 95% confidence interval by using the profile likelihood method. We estimated the same intervals by using bootstrapping as a robustness analysis. We created 1000 resampled datasets of matched case-control pairs. The total number of participants in each sample was equal to that of the original dataset (or the subset, for subgroup analysis), but differences within datasets were due to allowing replacement when sampling. For each of these datasets, we fitted a conditional logistic regression model and calculated vaccine effectiveness as in the main analysis. We calculated bootstrapping based vaccine effectiveness by taking the mean of the 1000 resampled datasets and defined its confidence interval limits as the empirical centiles 0.25 and 97.5.

We classified vaccination status in different levels depending on the subanalysis. The unvaccinated group was the selected reference group in all the analyses. Other levels of the vaccination status variable refer to participants who were partially vaccinated (at least 14 days between first dose and test), those who received three doses, and others (details in Data sources and definitions). The level of interest of this study is the fully vaccinated group with at least 14 days between their second dose and the diagnostic test.

We did several subanalyses. Firstly, we split data according to the study period (on 25 December 2021) considering the change in the dominant variant (from delta to omicron). In the last week of the delta period, 68.2% of cases were identified as due to delta and 31.7% as due to omicron. During the first week of the omicron period, 79.2% of the studied cases were assigned to this variant.22 For a second subanalysis, we used another split in age subgroups (3-11 and 12-17 years).

Seeking evidence of waning immunity, in a third subanalysis we divided the fully vaccinated group into four levels according to the time between their second dose and the test (15-30, 31-45, 46-60, ≥61 days). For a fourth subanalysis, we split the same level to assess vaccine effectiveness by product in the adolescent subpopulation, with all possible combinations of BNT162b2 and mRNA-1273 vaccines against both variants.

Fifthly, we did three sensitivity analyses. We assessed the effect of the type of test in the estimation by doing a subanalysis estimating RT-PCR and antigen based vaccine effectiveness separately. We fitted a linear regression model and used the non-parametric Mann-Whitney-Wilcoxon test to assess whether the difference between estimations was different from zero. For the linear model, PCR based estimates were the predictor variable and antigen based estimates were the dependent variable, as most test negative, case-control studies use PCR tests to assign participants a case status. We then repeated the main analysis excluding matching pairs in epidemiological weeks with zero fully vaccinated, to assess how not holding the positivity assumption might bias our estimates. Lastly, we excluded participants who had a previous infection to assess whether these participants might be biasing our estimates.

Finally, for a fatality analysis, we matched participants with covid-19 associated death one to four to covid-19 negative participants by using the same matching criteria as the one used in the primary analysis estimating vaccine effectiveness against SARS-CoV-2 infection. On this second matched case-control group, we estimated vaccine effectiveness against mortality through an adjusted conditional logistic regression model.

Data preprocessing was carried out with PostgreSQL (PostgreSQL Global Development Group). We used R software (R Development Core Team, version 3.6.1) for all statistical analyses.

Public and patient involvement

This was an unfunded study using routine surveillance data sources. No patients were involved in setting the research question or the outcome measures, nor were they involved in developing plans for design or implementation of the study. No patients were asked to advise on interpretation or writing up of results. There are no plans to directly disseminate the results of the research to study participants.

Results

The initial case-control cohort included 844 460 participants, of whom 421 359 sought RT-PCR testing and 423 101 were tested with immunoassays for antigen detection, between 12 September 2021 and 23 April 2022. Those who tested positive (231 181) had a higher proportion of reported comorbidities (5.2% v 4.9%; P<0.001), were less likely to be male (47.4% v 50.7%; P<0.001), had sought to be tested in more advanced epidemiological weeks and were more frequently tested with antigen detection (65.4% v 44.3%; P<0.001) (supplementary table A). Furthermore, they were slightly older (mean age 11.9 v 10.8 years; P<0.001) and had significant differences in the regional distribution (supplementary table B).

After matching, 139 321 (60.3%) of 231 181 cases with their corresponding controls remained for analysis. As a result of exact matching on presence of comorbidities, sex, type of test, province (supplementary table C), and week of testing, the distribution of those variables in cases and controls are equal. Age was matched with a tolerance of 1 year, yielding non-significant differences between the two groups (table 1).

Table 1

Population characteristics after matching, by test result and vaccination status. Values are numbers (percentages) unless stated otherwise

View this table:

The fully vaccinated participants represented 54% (150 966/278 642) of the matched case study group. Of these fully vaccinated participants, 73 409 (48.6%) were included as cases and the remainder as controls. In the 12-17 year old fully vaccinated subgroup, 86 338/91 108 (94.8%) received a homologous two dose schedule (64 197 (70.5%) BNT162b2; 22 141 (24.3%) mRNA-1273) and 4770 (5.2%) received a heterologous one (4197 (4.6%) mRNA-1273 and then BNT162b2; 573 (0.6%) BNT162b2 and then mRNA-1273). All children (3-11 years old) included as fully vaccinated received a two dose schedule of BBIBP-CorV (table 2). The median time from second dose to test was 66 (interquartile range 40) days for the 12-17 year old subgroup and 54 (32) days for the 3-11 year old subgroup.

Table 2

Vaccination schedule and vaccine effectiveness against infection, by combination and period. Values are numbers (percentages) unless stated otherwise

View this table:

Primary analysis on vaccine effectiveness in preventing infection

Figure 3 shows vaccine effectiveness in preventing infection according to the predominant variant and age group. Vaccine effectiveness was considerably higher in all age groups when we analysed only the period of delta variant predominance. Vaccine effectiveness was 64.2% (95% confidence interval 61.6% to 66.5%) for all ages, 61.2% (56.4% to 65.5%) for children (3-11 years), and 66.8% (63.9% to 69.5%) for adolescents (12-17 years). Estimates of vaccine effectiveness decreased markedly during the period of omicron predominance. Vaccine effectiveness was 19.9% (18.0% to 21.8%) for all ages, with a slightly higher value in adolescents who received mRNA based vaccines (26.0%, 23.2% to 28.8%) than in children who received BBIBP-CorV (15.9%, 13.2% to 18.6%). Assessment of robustness by means of bootstrapping yielded almost identical results with narrower confidence intervals (supplementary table D).

Fig 3
Fig 3

Vaccine effectiveness of two dose schedules against SARS-CoV-2 infection, by age group and study period. CI=confidence interval

Waning immunity

As shown in figure 4, during the delta period, vaccine effectiveness declined according to time since vaccination from 68.4% (64.1% to 72.2%) at 15-30 days to 65.2% (44.0% to 78.4%) at ≥61 days in children and from 74.8% (71.3% to 77.9%) to 56.3% (50.2% to 61.7%) in adolescents (supplementary table E). During the omicron period, the initial effectiveness was lower and the decline over time was steeper. Vaccine effectiveness dropped from 37.6% (34.2% to 40.8%) to 2.0% (1.8% to 5.6%) in children and from 55.8% (52.4% to 59.0%) to 12.4% (8.6% to 16.1%) in the adolescent subpopulation.

Fig 4
Fig 4

Vaccine effectiveness (with 95% confidence interval) of two dose schemes against SARS-CoV-2 infection, by age group, study period, and days since second dose

Subanalysis by vaccine type (adolescents)

As shown in table 2, all combinations of mRNA vaccine products showed similar results in adolescents. During the delta period, vaccine effectiveness was 70.2% (66.8% to 73.1%) for the homologous schedule of mRNA-1273 and 64.1% (60.5% to 67.3%) for BNT162b2. The heterologous schedule of mRNA-1273 then BNT162b2 yielded a vaccine effectiveness of 66.3% (54.0% to 75.4%), and that for BNT162b2 then mRNA-1273 was 88.9% (66.1% to 96.4%). In the omicron period, homologous schedules of mRNA-1273 and BNT162b2 showed a vaccine effectiveness of 17.9% (14.0% to 21.5%) and 28.1% (25.2% to 30.8%), respectively. Heterologous schedules showed 40.6% (29.4% to 50.0%) for participants receiving the BNT162b2 vaccine first and 31.5% (26.3% to 36.4%) for those who received the mRNA-1273 vaccine first.

Sensitivity analyses

On the hypothesis that estimations might be affected by the type of test used to select cases and controls, we analysed the results for RT-PCR and antigen test based vaccine effectiveness estimates (supplementary table F). The slope for the linear regression between RT-PCR and antigen based estimates of vaccine effectiveness was 0.933 (95% confidence interval 0.892 to 0.974), showing that antigen based vaccine effectiveness estimates tend to be slightly lower than RT-PCR based estimates, rejecting the null hypothesis that the slope equals one (P=0.004) (supplementary figure A). Additionally, we did a Mann-Whitney-Wilcoxon test to assess the relative width of the confidence interval between the two techniques. It yielded a non-significant P value of 0.61.

To evaluate how not holding the positivity assumption could affect our estimations, we did a sensitivity analysis excluding matching pairs in epidemiological weeks with zero fully vaccinated participants for each age subgroup. This occurred only during the delta period, and for both age groups the confidence intervals had a wide overlap with the results reported in the main analysis (supplementary table G).

Next, we did a sensitivity analysis excluding participants with previous infection to evaluate whether our estimates might be biased due to previously acquired immunity. After exclusion of participants who fulfilled this criterion, vaccine effectiveness estimates had a wide overlap with the results reported in the main analysis (supplementary table H).

Vaccine effectiveness in preventing mortality

Fifty one deaths related to covid-19 were notified among cases, of which seven occurred in the delta period (all in unvaccinated participants). Of the 51, 30 were unvaccinated participants (16 children; 14 adolescents), eight were partially vaccinated (three children; five adolescents), nine were fully vaccinated (six children; three adolescents), and four had other status (two children; two adolescents). Two further deaths were excluded after validation that the cause of death was not related to covid-19. Following one-to-four case-control matching, 44 (90%) of 49 deceased participants were successfully matched (table 3). After adjusting a conditional logistic regression model, we found that vaccine effectiveness against mortality in the 3-17 year old group for the two dose schedule was 89.3% (73.9% to 95.6%). When we considered only the omicron period, vaccine effectiveness remained high at 88.1% (70.7% to 95.2%). Considering age subgroups, vaccine effectiveness against death related to SARS-CoV-2 infection during omicron predominance was 97.6% (81.0% to 99.7%) for adolescents and 66.9% (6.4% to 89.8%) for children.

Table 3

Characteristics of covid-19 related deaths and matched controls. Values are numbers (percentages) unless stated otherwise

View this table:

Discussion

This study provides real world evidence for the effectiveness of mRNA-1273, BNT162b2, and BBIBP-CorV vaccines in preventing infection and mortality in children and adolescents in Argentina. To our knowledge, this is one of the first studies that reports vaccine effectiveness in children under 5 years old, in addition to evaluation of mortality and analysis of waning for BBIP-CoV and mRNA-1273 vaccines. Our results suggest that the primary vaccination schedule is effective in preventing mortality in children and adolescents with covid-19 regardless of the circulating SARS-CoV-2 variant. Vaccine effectiveness in preventing mortality in children vaccinated with BBIP-CoV was lower than in adolescents vaccinated with mRNA vaccines; however, owing to the scarcity of events, analysis of mortality by age group yielded results with wide confidence intervals. Our estimates also suggest that two doses of vaccine are effective in preventing SARS-CoV-2 infection in children and adolescents in the short term. Vaccine effectiveness was considerably higher when delta was the predominant circulating variant, but a significant decrease was observed over time, especially during omicron predominance.

Comparison with other studies

By the time of our study, only BBIBP-CorV was indicated in children, so we could not compare vaccine effectiveness in children by vaccine type. Evidence is variable taking into consideration that published studies differ in age groups and time to vaccination periods. Nevertheless, BNT162b2 vaccine effectiveness in preventing infection during omicron predominance has been evaluated, with vaccine effectiveness between 29.4% and 51% being reported.122324 These results are similar to the vaccine effectiveness reported in our study and in other studies that evaluated inactivated vaccines (CoranaVac) and reported vaccine effectiveness of 39.8% and 38.2%.1618

In adolescents, we evaluated different vaccine schedules. Compared with the delta period, vaccine effectiveness during the omicron period was lower for both evaluated schedules (homologous and heterologous), in consistency with the literature. A previous study of immunogenicity in adults in Argentina evaluated heterologous schedules of adenoviral vectored, inactivated, and mRNA vaccines and endorsed heterologous vaccination strategies.25 In our study, which evaluated mRNA-1273 and BNT162b2 vaccines, heterologous schedules showed a comparable to superior vaccine effectiveness in comparison with homologous schedules. Nevertheless, further investigation may be needed to evaluate different vaccine schedules.

In addition, our results are consistent with other published studies that found high short term protection induced by the primary vaccination regimen against SARS-CoV-2 infection during the delta predominance period.262728 However, since the emergence and spread of the omicron variant of concern, a reduction in vaccine effectiveness in preventing infection has been observed in adults and children.13293031 A population based study in Norway reported that one and two doses of mRNA vaccine (BNT162b2) in adolescents protected against infection with the delta variant, reaching an effectiveness of 91% after the primary vaccination schedule and declining to 53% for the omicron variant.28 Fowlkes and colleagues described a decrease in vaccine effectiveness against infection in adolescents for the primary mRNA vaccine series (BNT162b2) from 87% to 59%, comparing delta and omicron variants.12 In Chile, Jara and colleagues reported vaccine effectiveness in preventing infection for a complete schedule of inactivated vaccine (Sinovac’s CoronaVac) in 6-11 year old children of 75.8% during the delta circulating period. In another study, Jara and colleagues reported a lower vaccine effectiveness of 38.2% during omicron circulation for the same vaccine in 3-5 year old children.1516 Vaccine effectiveness reported by Jara and colleagues is similar to that observed with BBIBP-CorV in children in our study.

Available evidence suggests that the vaccine induced immune response could decrease over time, with a reduction in the effectiveness of SARS-CoV-2 vaccines in preventing infection, particularly in the context of the emergence of new variants of the virus with potential ability to partially evade the host immune response.3233 Our results highlight not only a decrease in effectiveness when comparing delta with omicron but also, and especially during omicron predominance, a significant decrease with time since vaccination. The decrease observed with BBIBP-CorV in children in our study is consistent with published evidence. Fleming-Dutra and colleagues did a test negative, case-control analysis to evaluate the vaccine effectiveness of BNT162b2 in preventing infection among children and adolescents during omicron variant predominance and reported that two months after the second dose the estimated vaccine effectiveness had declined in both age groups.34 Similarly, Dorabawila and colleagues concluded that the risk of infection and hospital admission was higher in unvaccinated children and adolescents, although the protection declined over time from vaccination.30 Sacco and colleagues reported a decrease in vaccine effectiveness for BNT162b2 in preventing infection over time, and Klein and colleagues found no evidence of protection for adolescents aged 12-17 years after receipt of two doses of BNT162b2 ≥150 days earlier during omicron predominance.2329 In a Norwegian study, protection against omicron after a complete primary regimen in adolescents was reduced to 23% from 63 days post-vaccination, whereas a smaller reduction was documented for delta.28 Amir and colleagues observed a rapid reduction in protection against omicron of two doses of mRNA vaccine (BNT162b2) in adolescents aged 12-15, with an increase in the rate of confirmed infections from 60 days after the second dose.35 Evidence of a decrease in vaccine effectiveness after inactivated vaccines is lacking.

Although the previously cited studies suggest a reduction in vaccine effectiveness for SARS-Cov-2 infection over time, observational studies have shown that vaccine effectiveness against severe disease is higher and more stable. Data from Brazil and Scotland in adolescents aged 12-17 years during the omicron wave showed a reduction in BNT162b2 vaccine effectiveness for symptomatic infection from 98 days after receipt of the primary schedule. However, protection against hospital admission and death with covid-19 remained above 80% in the same time interval.36 In Canada, vaccine effectiveness for two doses of BNT162b2 against severe disease with omicron in the same age group was 85%, with a stable trend over time.37 In children, Tan and colleagues reported a vaccine effectiveness of 82.7% in preventing hospital admission for BNT162b2.38 For inactivated vaccines, vaccine effectiveness of 59.2% and 64.6% have been reported for prevention of hospital admission with Sinovac’s CoronaVac.1618

Our results are consistent with a study conducted in Buenos Aires Province (Argentina) that reported a two dose vaccine effectiveness of 78.0% for hospital admission in 3-17 year olds evaluated during delta/omicron circulation.17 As expected, we found that vaccine effectiveness for infection was lower and vaccine effectiveness for mortality was higher than these estimations for protection against hospital admissions, especially in adolescents. In this study, estimated effectiveness against hospital admissions was higher when delta was predominant, although it remained higher than 65% during the omicron period.17 This study, which took place in an Argentinian province, was the first one to evaluate vaccine effectiveness of BBIBP-CorV in children between 3 and 11 years old. However, it did not evaluate vaccine effectiveness in preventing infection or mortality or decrease in vaccine effectiveness over time.17

Our study evaluated vaccine effectiveness in children and adolescents, including different types of diagnostic tests. Although statistically significant, the real life implications of these differences might not be important for public health agents. According to our results, antigen based estimates tended to be 6.7% lower (P=0.004) than the PCR based one for the same population and provide results with a comparable variance. Therefore, the advantages of using antigen tests alone or in addition to RT-PCR tests in vaccine effectiveness studies would outweigh the slight differences in estimations. Tan and colleagues reported differences in vaccine effectiveness by type of diagnostic test; however, they highlighted that in disease management protocols, the country included PCR testing for people with more severe symptoms or coexisting conditions, so they used PCR results as proxy of increased severity of illness.38

Strengths and limitations of study

The study has several strengths. Firstly, the sample size was large and included all children and adolescents tested and reported to the National Surveillance System in Argentina during the study period. Secondly, the high quality of the epidemiological surveillance (especially for laboratory and mortality data) and vaccination records used confers robustness to the results.

Limitations of our study include that some information, such as symptoms and hospital admissions, was incomplete and was consequently not included in any analysis. Also, as persistent potential confounders, misclassification, or selection bias might have been present, we decided to minimise them by matching with strict criteria on all the available confounders and doing univariate conditional logistic regression analyses. We also did several sensitivity analyses to assess the effect of some unmeasured phenomena. Nevertheless, unobserved confounding might exist and bias odds ratio estimations. The generalisability of our results is limited to the subpopulation who had access to testing. Additionally, as misclassification might have occurred as a result of non-100% specificity and sensitivity of covid-19 tests, we included a sensitivity analysis to assess how test related problems might affect the vaccine effectiveness estimations, which showed that antigen based estimates of vaccine effectiveness tended to be lower. Other limitations include that each age group used different vaccines, so any conclusion based on their differences should not be attributed exclusively to the vaccine technology or to age related factors. Finally, information about variants could not be established at an individual level, and we used the prevalent variant from surveillance data when analysing our results.

Conclusions

Vaccine effectiveness remained high for preventing mortality in children and adolescents regardless of the predominant circulating variant. Vaccines were effective in preventing SARS-CoV-2 infection in the short term after vaccination, being lower when the omicron variant was predominant and decreasing sharply over time. Heterologous mRNA schedules showed comparable to superior vaccine effectiveness in comparison with homologous schedules. In summary, vaccinating children is an important public health measure that will prevent mortality in this population, especially in periods of high viral circulation.

What is already known on this topic

  • mRNA and inactivated covid-19 vaccines are effective in preventing severe disease and infection in children and adolescents in the short term after vaccination, but data on mortality are lacking

  • Waning of protection against infection has been described, especially for mRNA vaccines, but evidence for inactivated vaccines in children is limited

What this study adds

  • Vaccine effectiveness remained high for preventing mortality in children and adolescents aged 3-17 years, regardless of the predominant circulating variant

  • Waning of protection against infection was described and evaluated for both types of vaccines

  • Heterologous mRNA schedules showed comparable to superior vaccine effectiveness in comparison with homologous schedules

Ethics statements

Ethical approval

Covid-19 surveillance was included as an obligatory notifiable disease by RM 680/2020 (https://www.boletinoficial.gob.ar/detalleAviso/primera/227324/20200331). As secondary data were used, the study protocol was subject to an internal review by the ethical area of the Research Department of the Ministry of Health and was found to be fully compliant with all regulatory requirements. As no regulatory issues were identified, a full ethical review was deemed not to be necessary.

Data availability statement

The data that support the findings of this study will be available for researchers who provide a methodologically sound proposal after it is approved, on request from the corresponding author.

Acknowledgments

CMG works on behalf of the Área de Vigilancia del Ministerio de Salud de la Nación. This work was possible thanks to the health system workers who carried out the compilation and notification of covid-19 cases to the National Surveillance System throughout the Argentine territory and to the Nominalized Federal Vaccination Registry System on people vaccinated against covid-19. We extend our gratitude to the epidemiology departments, the expanded immunisation programmes, and the clinical and laboratory surveillance coordinators throughout the country for their role in coordinating the entire surveillance network in each jurisdiction.

Footnotes

  • Contributors: AR, CV, and JMC conceived and supervised the study. AR was in charge of the project administration. AR and SO designed the methods. SO performed and interpreted the formal analysis. AR, SO, CV, MDVJ, MP, MEC, MP, and ANI conceptualised the study and wrote the original draft. CMG critically revised the manuscript. MPB, MBM, and ML developed the supporting algorithms and curated the data. All authors reviewed, edited and gave final approval for the version to be submitted. AR is the guarantor. The corresponding author attests that all listed authors meet authorship criteria and that no others meeting the criteria have been omitted.

  • Funding: No external funding.

  • Competing interests: All authors have completed the ICMJE uniform disclosure form at https://www.icmje.org/disclosure-of-interest/ and declare: no support from any organisation for the submitted work; JMC and CV were involved in the decision making process of the vaccination campaign in Argentina; no other relationships or activities that could appear to have influenced the submitted work.

  • The lead author (the manuscript’s guarantor) affirms that the manuscript is an honest, accurate, and transparent account of the study being reported; that no important aspects of the study have been omitted; and that any discrepancies from the study as planned (and, if relevant, registered) have been explained.

  • Dissemination to participants and related patient and public communities: Results from this research will be shared through conferences with expert immunisation advisory committees for Argentina and public health officials overseeing Argentina’s covid-19 immunisation campaign. The findings will also be further disseminated through the National Health Ministry website, social media channels, and news media.

  • Provenance and peer review: Not commissioned; externally peer reviewed.

http://creativecommons.org/licenses/by-nc/4.0/

This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.

References

  1. ↵
    1. Mehta NS,
    2. Mytton OT,
    3. Mullins EWS,
    4. et al
    . SARS-CoV-2 (COVID-19): What Do We Know About Children? A Systematic Review. Clin Infect Dis2020;71:2469-79. doi:10.1093/cid/ciaa556 pmid:32392337
    OpenUrlCrossRefPubMed
  2. ↵
    Ministerio de Salud de la Nación Argentina. Sala de situación covid-2019 Nuevo Coronavirus 2019 - Argentina. https://www.argentina.gob.ar/sites/default/files/2022/02/sala-ninez-adolescencia-21-03-22-se11.pdf.
  3. ↵
    1. Committee on Infectious Diseases
    . COVID-19 Vaccines in Children and Adolescents. Pediatrics2021;148:e2021052336. doi:10.1542/peds.2021-052336 pmid:33980697
    OpenUrlFREE Full Text
  4. ↵
    Ministerio de Salud de la Nación Argentina. Reunión de Comisión Nacional de Inmunizaciones 15 de julio de 2021. https://www.argentina.gob.ar/sites/default/files/2021/04/2021-07-15-vacunacion-covid-adolescentes.pdf.
  5. ↵
    World Health Organization. WHO SAGE roadmap for prioritizing uses of COVID-19 vaccines: an approach to optimize the global impact of COVID-19 vaccines, based on public health goals, global and national equity, and vaccine access and coverage scenarios, first issued 20 October 2020, updated: 13 November 2020, updated: 16 July 2021, latest update: 21 January 2022. https://apps.who.int/iris/handle/10665/351138.
  6. ↵
    Ministerio de Salud de la Nación Argentina. Actualización lineamientos técnicos Campaña Nacional de vacunación contra la COVID-19. 2021. https://bancos.salud.gob.ar/recurso/actualizacion-de-los-lineamientos-tecnicos-campana-nacional-de-vacunacion-contra-la-covid.
  7. ↵
    Ministerio de Salud de la Nación Argentina. Resolución 688/2021. Autorízase con carácter de emergencia la vacuna SARS COV-2 (células vero) inactivada, desarrollada por el Laboratorio Beijing Institute of Biological Products de la República Popular China. 2021. https://www.argentina.gob.ar/normativa/nacional/resoluci%C3%B3n-688-2021-347264.
  8. ↵
    Ministerio de Salud de la Nación Argentina. Resolución 2711/2021. Autorízase para personas mayores de 12 años de edad, con carácter de emergencia la vacuna SPIKEVAX (ARNm-1273), desarrollada por el Laboratorio ModernaTX Inc. 2021. https://www.boletinoficial.gob.ar/detalleAviso/primera/250495/20211005.
  9. ↵
    Administración Nacional De Medicamentos. Alimentos Y Tecnología Médica. Autorízase a Pfizer S.R.L la extensión de la indicación en personas de 12 años de edad y mayores. 2021. http://www.anmat.gov.ar/boletin_anmat/Julio_2021/Dispo_5569-21.pdf.
  10. ↵
    1. Kitano T,
    2. Kitano M,
    3. Krueger C,
    4. et al
    . The differential impact of pediatric COVID-19 between high-income countries and low- and middle-income countries: A systematic review of fatality and ICU admission in children worldwide. PLoS One2021;16:e0246326. doi:10.1371/journal.pone.0246326 pmid:33513204
    OpenUrlCrossRefPubMed
  11. ↵
    1. Xia S,
    2. Zhang Y,
    3. Wang Y,
    4. et al
    . Safety and immunogenicity of an inactivated COVID-19 vaccine, BBIBP-CorV, in people younger than 18 years: a randomised, double-blind, controlled, phase 1/2 trial. Lancet Infect Dis2022;22:196-208. doi:10.1016/S1473-3099(21)00462-X pmid:34536349
    OpenUrlCrossRefPubMed
  12. ↵
    1. Fowlkes AL,
    2. Yoon SK,
    3. Lutrick K,
    4. et al
    . Effectiveness of 2-Dose BNT162b2 (Pfizer BioNTech) mRNA Vaccine in Preventing SARS-CoV-2 Infection Among Children Aged 5-11 Years and Adolescents Aged 12-15 Years - PROTECT Cohort, July 2021-February 2022. MMWR Morb Mortal Wkly Rep2022;71:422-8. doi:10.15585/mmwr.mm7111e1 pmid:35298453
    OpenUrlCrossRefPubMed
  13. ↵
    1. Powell AA,
    2. Kirsebom F,
    3. Stowe J,
    4. et al
    . Effectiveness of BNT162b2 against COVID-19 in adolescents. Lancet Infect Dis2022;22:581-3. doi:10.1016/S1473-3099(22)00177-3 pmid:35325619
    OpenUrlCrossRefPubMed
  14. ↵
    1. Price AM,
    2. Olson SM,
    3. Newhams MM,
    4. et al.,
    5. Overcoming Covid-19 Investigators
    . BNT162b2 Protection against the Omicron Variant in Children and Adolescents. N Engl J Med2022;386:1899-909. doi:10.1056/NEJMoa2202826 pmid:35353976
    OpenUrlCrossRefPubMed
  15. ↵
    1. Jara A,
    2. Undurraga EA,
    3. Flores JC,
    4. et al
    . Effectiveness of an Inactivated SARS-CoV-2 Vaccine in Children and Adolescents: A Large-Scale Observational Study. Nat Med2022;28:1377-80. doi:10.1038/s41591-022-01874-4 pmid:35605637
    OpenUrlCrossRefPubMed
  16. ↵
    1. Jara A,
    2. Undurraga EA,
    3. Zubizarreta JR,
    4. et al
    . Effectiveness of CoronaVac in children 3-5 years of age during the SARS-CoV-2 Omicron outbreak in Chile. Nat Med2022;28:1377-80. doi:10.1038/s41591-022-01874-4 pmid:35605637
    OpenUrlCrossRefPubMed
  17. ↵
    1. González S,
    2. Olszevicki S,
    3. Gaiano A,
    4. et al
    . Effectiveness of BBIBP-CorV, BNT162b2 and mRNA-1273 vaccines against hospitalisations among children and adolescents during the Omicron outbreak in Argentina: A retrospective cohort study. Lancet Reg Health Am2022;13:100316. doi:10.1016/j.lana.2022.100316 pmid:35872665
    OpenUrlCrossRefPubMed
  18. ↵
    1. Florentino PTV,
    2. Alves FJO,
    3. Cerqueira-Silva T,
    4. et al
    . Vaccine effectiveness of CoronaVac against COVID-19 among children in Brazil during the Omicron period. Nat Commun2022;13:4756. doi:10.1038/s41467-022-32524-5 pmid:35963844
    OpenUrlCrossRefPubMed
  19. ↵
    1. De Serres G,
    2. Skowronski DM,
    3. Wu XW,
    4. Ambrose CS
    . The test-negative design: validity, accuracy and precision of vaccine efficacy estimates compared to the gold standard of randomised placebo-controlled clinical trials. Euro Surveill2013;18:20585. doi:10.2807/1560-7917.ES2013.18.37.20585 pmid:24079398
    OpenUrlCrossRefPubMed
  20. ↵
    World Health Organization. Design of vaccine efficacy trials to be used during public health emergencies—points of considerations and key principles. https://www.who.int/docs/default-source/blue-print/working-group-for-vaccine-evaluation-(4th-consultation)/ap1-guidelines-online-consultation.pdf.
  21. ↵
    1. von Elm E,
    2. Altman DG,
    3. Egger M,
    4. Pocock SJ,
    5. Gøtzsche PC,
    6. Vandenbroucke JP,
    7. STROBE Initiative
    . The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) Statement: guidelines for reporting observational studies. Int J Surg2014;12:1495-9. doi:10.1016/j.ijsu.2014.07.013 pmid:25046131
    OpenUrlCrossRefPubMed
  22. ↵
    Ministerio de Salud de la Nación Argentina. Situación de nuevas variantes SARS-CoV-2 en Argentina - Informe técnico SE8/2022. 2022. https://www.argentina.gob.ar/sites/default/files/2022/02/informe_vigilancia_genomica_se08_2022.pdf.
  23. ↵
    1. Klein NP,
    2. Stockwell MS,
    3. Demarco M,
    4. et al
    . Effectiveness of COVID-19 Pfizer-BioNTech BNT162b2 mRNA Vaccination in Preventing COVID-19-Associated Emergency Department and Urgent Care Encounters and Hospitalizations Among Nonimmunocompromised Children and Adolescents Aged 5-17 Years - VISION Network, 10 States, April 2021-January 2022. MMWR Morb Mortal Wkly Rep2022;71:352-8. doi:10.15585/mmwr.mm7109e3 pmid:35239634
    OpenUrlCrossRefPubMed
  24. ↵
    1. Cohen-Stavi CJ,
    2. Magen O,
    3. Barda N,
    4. et al
    . BNT162b2 Vaccine Effectiveness against Omicron in Children 5 to 11 Years of Age. N Engl J Med2022;387:227-36. doi:10.1056/NEJMoa2205011 pmid:35767475
    OpenUrlCrossRefPubMed
  25. ↵
    1. Rearte A,
    2. Castelli JM,
    3. Rearte R,
    4. et al
    . Effectiveness of rAd26-rAd5, ChAdOx1 nCoV-19, and BBIBP-CorV vaccines for risk of infection with SARS-CoV-2 and death due to COVID-19 in people older than 60 years in Argentina: a test-negative, case-control, and retrospective longitudinal study. Lancet2022;399:1254-64. doi:10.1016/S0140-6736(22)00011-3 pmid:35303473
    OpenUrlCrossRefPubMed
  26. ↵
    1. Glatman-Freedman A,
    2. Hershkovitz Y,
    3. Kaufman Z,
    4. Dichtiar R,
    5. Keinan-Boker L,
    6. Bromberg M
    . Effectiveness of BNT162b2 Vaccine in Adolescents during Outbreak of SARS-CoV-2 Delta Variant Infection, Israel, 2021. Emerg Infect Dis2021;27:2919-22. doi:10.3201/eid2711.211886 pmid:34570694
    OpenUrlCrossRefPubMed
  27. ↵
    1. Reis BY,
    2. Barda N,
    3. Leshchinsky M,
    4. et al
    . Effectiveness of BNT162b2 Vaccine against Delta Variant in Adolescents. N Engl J Med2021;385:2101-3. doi:10.1056/NEJMc2114290 pmid:34670036
    OpenUrlCrossRefPubMed
  28. ↵
    1. Veneti L,
    2. Berild JD,
    3. Viksmoen Watle S,
    4. et al
    . Vaccine effectiveness with BNT162b2 (Comirnaty, Pfizer-BioNTech) vaccine against reported SARS-CoV-2 Delta and Omicron infection among adolescents, Norway, August 2021 to January 2022. medRxiv2022.03.24.22272854; .
  29. ↵
    1. Sacco C,
    2. Del Manso M,
    3. Mateo-Urdiales A,
    4. et al.,
    5. Italian National COVID-19 Integrated Surveillance System and the Italian COVID-19 vaccines registry
    . Effectiveness of BNT162b2 vaccine against SARS-CoV-2 infection and severe COVID-19 in children aged 5-11 years in Italy: a retrospective analysis of January-April, 2022. Lancet2022;400:97-103. doi:10.1016/S0140-6736(22)01185-0 pmid:35780801
    OpenUrlCrossRefPubMed
  30. ↵
    1. Dorabawila V,
    2. Hoefer D,
    3. Bauer UE,
    4. Bassett MT,
    5. Lutterloh E,
    6. Rosenberg ES
    . Risk of Infection and Hospitalization Among Vaccinated and Unvaccinated Children and Adolescents in New York After the Emergence of the Omicron Variant. JAMA2022;327:2242-4. doi:10.1001/jama.2022.7319 pmid:35559959
    OpenUrlCrossRefPubMed
  31. ↵
    1. Tseng HF,
    2. Ackerson BK,
    3. Luo Y,
    4. et al
    . Effectiveness of mRNA-1273 against SARS-CoV-2 Omicron and Delta variants. Nat Med2022;28:1063-71. doi:10.1038/s41591-022-01753-y pmid:35189624
    OpenUrlCrossRefPubMed
  32. ↵
    1. Britton A,
    2. Fleming-Dutra KE,
    3. Shang N,
    4. et al
    . Association of COVID-19 Vaccination With Symptomatic SARS-CoV-2 Infection by Time Since Vaccination and Delta Variant Predominance. JAMA2022;327:1032-41. doi:10.1001/jama.2022.2068 pmid:35157002
    OpenUrlCrossRefPubMed
  33. ↵
    1. Pierobon A,
    2. Zotto AD,
    3. Antico A,
    4. et al
    . Outbreak of SARS-CoV-2 B.1.617.2 (delta) variant in a nursing home 28 weeks after two doses of mRNA anti-COVID-19 vaccines: evidence of a waning immunity. Clin Microbiol Infect2022;28:614. doi:10.1016/j.cmi.2021.12.013 pmid:34958917
    OpenUrlCrossRefPubMed
  34. ↵
    1. Fleming-Dutra KE,
    2. Britton A,
    3. Shang N,
    4. et al
    . Association of Prior BNT162b2 COVID-19 Vaccination With Symptomatic SARS-CoV-2 Infection in Children and Adolescents During Omicron Predominance. JAMA2022;327:2210-9. doi:10.1001/jama.2022.7493 pmid:35560036
    OpenUrlCrossRefPubMed
  35. ↵
    1. Amir O,
    2. Goldberg Y,
    3. Mandel M,
    4. et al
    . Initial protection against Omicron in children and adolescents by BNT162b2.medRxiv2022.05.22.22275323; doi:10.1101/2022.05.22.22275323 .
    OpenUrlAbstract/FREE Full Text
  36. ↵
    1. Florentino PTV,
    2. Millington T,
    3. Cerqueira-Silva T,
    4. et al
    . Vaccine effectiveness of two-dose BNT162b2 against symptomatic and severe COVID-19 among adolescents in Brazil and Scotland over time: a test-negative case-control study. Lancet Infect Dis2022;22:1577-86. doi:10.1016/S1473-3099(22)00451-0 pmid:35952702
    OpenUrlCrossRefPubMed
  37. ↵
    1. Buchan SA,
    2. Nguyen L,
    3. Wilson SE,
    4. Kitchen SA,
    5. Kwong JC
    . Vaccine Effectiveness of BNT162b2 Against Delta and Omicron Variants in Adolescents. Pediatrics2022;150:e2022057634. doi:10.1542/peds.2022-057634 pmid:35706105
    OpenUrlCrossRefPubMed
  38. ↵
    1. Tan SHX,
    2. Cook AR,
    3. Heng D,
    4. Ong B,
    5. Lye DC,
    6. Tan KB
    . Effectiveness of BNT162b2 Vaccine against Omicron in Children 5 to 11 Years of Age. N Engl J Med2022;387:525-32. doi:10.1056/NEJMoa2203209 pmid:35857701
    OpenUrlCrossRefPubMed
Back to top