Intended for healthcare professionals

CCBY Open access
Research

SARS-CoV-2 lateral flow assays for possible use in national covid-19 seroprevalence surveys (React 2): diagnostic accuracy study

BMJ 2021; 372 doi: https://doi.org/10.1136/bmj.n423 (Published 02 March 2021) Cite this as: BMJ 2021;372:n423

Read our latest coverage of the coronavirus outbreak

  1. Maya Moshe, research technician1,
  2. Anna Daunt, clinical research fellow1 2,
  3. Barnaby Flower, clinical research fellow1 2,
  4. Bryony Simmons, doctoral student1,
  5. Jonathan C Brown, research associate1,
  6. Rebecca Frise, research technician1,
  7. Rebecca Penn, doctoral student1,
  8. Ruthiran Kugathasan, doctoral student1,
  9. Claire Petersen, senior divisional research nurse2,
  10. Helen Stockmann, research nurse2,
  11. Deborah Ashby, professor3,
  12. Steven Riley, professor3,
  13. Christina Atchison, principal clinical academic fellow3,
  14. Graham P Taylor, professor1,
  15. Sutha Satkunarajah, innovation fellow4,
  16. Lenny Naar, head of design strategy4,
  17. Robert Klaber, consultant2,
  18. Anjna Badhan, research associate1,
  19. Carolina Rosadas, research associate1,
  20. Federica Marchesin, research technician1,
  21. Natalia Fernandez, scientific officer1,
  22. Macià Sureda-Vives, doctoral student5,
  23. Hannah Cheeseman, research associate1,
  24. Jessica O’Hara, research technician1,
  25. Robin Shattock, professor1,
  26. Gianluca Fontana, senior policy fellow4,
  27. Scott J C Pallett, consultant physician6 7,
  28. Michael Rayment, consultant6,
  29. Rachael Jones, lead clinician6,
  30. Luke S P Moore, honorary clinical senior lecturer1 6,
  31. Hutan Ashrafian, chief medical officer8,
  32. Peter Cherapanov, professor1 9,
  33. Richard Tedder, professor1,
  34. Myra McClure, professor1,
  35. Helen Ward, professor3,
  36. Ara Darzi, professor4,
  37. Paul Elliott, professor2 3 10,
  38. Graham S Cooke, professor1,
  39. Wendy S Barclay, professor1 2
  40. on behalf of the React study team
  1. 1Department of Infectious Disease, Imperial College London, School of Medicine, St Mary’s Hospital, Praed Street, London W2 1NY, UK
  2. 2Imperial College Healthcare NHS Trust, St Mary’s Hospital, London, UK
  3. 3School of Public Health, Imperial College London, St Mary’s Hospital, London, UK
  4. 4Institute for Global Health Innovation, Imperial College London, London, UK
  5. 5Synthetic Biology Group, London Institute of Medical Sciences, Imperial College London, London, UK
  6. 6Chelsea and Westminster NHS Foundation Trust, London, UK
  7. 7Centre of Defence Pathology, Royal Centre for Defence Medicine, Queen Elizabeth Hospital, Birmingham, UK
  8. 8Department of Surgery and Cancer, Imperial College London, London, UK
  9. 9Chromatin Structure and Mobile DNA Laboratory, Francis Crick Institute, London, UK
  10. 10National Institute for Health Research (NIHR) Health Protection Research Unit (HPRU) in Chemical and Radiation Threats and Hazards, Imperial College London, London, UK
  1. Correspondence to: M Moshe maya.moshe18{at}imperial.ac.uk
  • Accepted 11 February 2021

Abstract

Objective To evaluate the performance of new lateral flow immunoassays (LFIAs) suitable for use in a national coronavirus disease 2019 (covid-19) seroprevalence programme (real time assessment of community transmission 2—React 2).

Design Diagnostic accuracy study.

Setting Laboratory analyses were performed in the United Kingdom at Imperial College, London and university facilities in London. Research clinics for finger prick sampling were run in two affiliated NHS trusts.

Participants Sensitivity analyses were performed on sera stored from 320 previous participants in the React 2 programme with confirmed previous severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) infection. Specificity analyses were performed on 1000 prepandemic serum samples. 100 new participants with confirmed previous SARS-CoV-2 infection attended study clinics for finger prick testing.

Interventions Laboratory sensitivity and specificity analyses were performed for seven LFIAs on a minimum of 200 serum samples from participants with confirmed SARS-CoV-2 infection and 500 prepandemic serum samples, respectively. Three LFIAs were found to have a laboratory sensitivity superior to the finger prick sensitivity of the LFIA currently used in React 2 seroprevalence studies (84%). These LFIAs were then further evaluated through finger prick testing on participants with confirmed previous SARS-CoV-2 infection: two LFIAs (Surescreen, Panbio) were evaluated in clinics in June-July 2020 and the third LFIA (AbC-19) in September 2020. A spike protein enzyme linked immunoassay and hybrid double antigen binding assay were used as laboratory reference standards.

Main outcome measures The accuracy of LFIAs in detecting immunoglobulin G (IgG) antibodies to SARS-CoV-2 compared with two reference standards.

Results The sensitivity and specificity of seven new LFIAs that were analysed using sera varied from 69% to 100%, and from 98.6% to 100%, respectively (compared with the two reference standards). Sensitivity on finger prick testing was 77% (95% confidence interval 61.4% to 88.2%) for Panbio, 86% (72.7% to 94.8%) for Surescreen, and 69% (53.8% to 81.3%) for AbC-19 compared with the reference standards. Sensitivity for sera from matched clinical samples performed on AbC-19 was significantly higher with serum than finger prick at 92% (80.0% to 97.7%, P=0.01). Antibody titres varied considerably among cohorts. The numbers of positive samples identified by finger prick in the lowest antibody titre quarter varied among LFIAs.

Conclusions One new LFIA was identified with clinical performance suitable for potential inclusion in seroprevalence studies. However, none of the LFIAs tested had clearly superior performance to the LFIA currently used in React 2 seroprevalence surveys, and none showed sufficient sensitivity and specificity to be considered for routine clinical use.

Introduction

A detailed understanding of severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) seroprevalence is key to public health policy and in anticipating the epidemiology of the coronavirus disease 2019 (covid-19) pandemic. In contrast to routine serology assays, the use of lateral flow immunoassays (LFIAs) does not require the support of central laboratories and offers a rapid, scalable, and affordable method of testing. This approach has been used in Spain1 and in our own React 2 (real time assessment of community transmission 2) study in England2 to conduct national seroprevalence studies2 and to monitor the persistence of SARS-CoV-2 antibodies.3 The React 2 programme consists of participants self-administering LFIAs at home. Participants complete an online questionnaire and read the results by using uploaded test images.2 The first round of the programme consisted of more than 100 000 participants and was completed on 13 July 2020. The results showed a SARS-CoV-2 antibody prevalence of approximately 6% nationally, with ethnic minority groups, healthcare workers, and care home workers disproportionately affected.2 Two subsequent rounds of surveillance have now been completed, all using the same assay (Fortress Diagnostics, Northern Ireland), which was selected after rigorous clinical and laboratory evaluation4 and engagement with the public and participants.5

Diagnostics development continues at pace internationally, with over 200 LFIAs commercialised.6 Despite this effort, no LFIA evaluated to date meets the Medicines and Healthcare products Regulatory Agency criteria for approval for individual testing in the United Kingdom,7 which requires the sensitivity (proportion of people with SARS-CoV-2 infection with a positive test result) and specificity (proportion of people without SARS-CoV-2 infection with a negative test result) to exceed 98%. When used in population studies, analyses can adjust for the performance characteristics of tests that do not meet such stringent criteria. However, to ensure these adjustments are accurate, evaluation in the intended setting of use is required. At present, the evaluation of LFIAs has focused on performance in the laboratory.89 Rigorous clinical evaluation of SARS-CoV-2 LFIAs of different population subgroups (such as people admitted to hospital compared with those not admitted to hospital, or people with severe symptoms compared with those without symptoms) is urgently needed to enhance generalisability and reduce variability in sensitivity and specificity estimates.10

In this study, we continue our programme of evaluating LFIAs that have been prioritised from published evaluations and that have the potential for large scale application. The primary objective was to establish the sensitivity and specificity of new LFIAs, and to identify the most suitable candidate for deployment in future rounds of the React 2 study and potentially for individual use. A new LFIA would be considered to replace the Fortress LFIA in future rounds of React 2 seroprevalence surveys if it showed significantly superior sensitivity on finger prick testing and the capacity to be procured rapidly and at scale.

Methods

The methods reported here describe round 2 of our LFIA evaluation (React 2, study 1) and are based on the same principles as the round 1 methods.4 However, in contrast to round 1, LFIAs were only evaluated on capillary blood in the clinic if they showed equivalent or superior sensitivity and specificity on testing of sera in the laboratory compared with the Fortress LFIA currently being used for the React 2 seroprevalence studies. A comprehensive React protocol has been published and describes the study design, sampling size and strategy, and data collection and analysis of the various ongoing React 1 and React 2 studies.11

Laboratory assessment of sensitivity and specificity using sera

Initial assessment of LFIA sensitivity used sera stored from 320 participants from round 1 with a previous positive test for SARS-CoV-2 by reverse transcription polymerase chain reaction (RT-PCR) on nasopharyngeal swab. Supplementary figure i presents a detailed summary of the flow of participants during sensitivity and specificity analysis in the laboratory. Sera were stored at −80ºC, subjected to a maximum of two freeze-thaw cycles, and brought to room temperature before testing. Research technicians blinded to the reference standard assay results performed LFIA testing in the laboratory. The two reference standards used in this study—the in-house SARS-CoV-2 spike protein enzyme linked immunoassay (S-ELISA) and a hybrid double antigen binding assay (hybrid DABA)—have been shown to have high sensitivity and specificity; these reference standards have been described previously.4 The viral antigen used in the S-ELISA is the SARS-CoV-2 spike protein whereas the hybrid DABA uses the SARS-CoV-2 spike protein and receptor binding domain. This variation results in inherent differences in the detection of antibody classes between the two in-house ELISAs; therefore, a composite outcome of a positive result on S-ELISA or hybrid DABA was used as the benchmark for sensitivity analysis throughout this study. Research technicians performing LFIA testing in the laboratory were aware of whether they were testing LFIAs for sensitivity or specificity, but were blinded to clinical information about participants and the reference standard assay. Assessors of the reference standard were blinded to clinical information and LFIA test results, and repeated any borderline positive or negative samples to give a determinate result.

Sensitivity in the laboratory for detection of SARS-CoV-2 antibodies was estimated for each LFIA using a minimum of 200 sera and compared with positive results on the S-ELISA or hybrid DABA. When testing the AbC-19 LFIA, a scoring card supplied by the manufacturer was used to grade the intensity of immunoglobulin G (IgG) bands on a scale of 1-10 (supplementary table i). All other LFIA results were interpreted as either IgG positive or negative. Specificity analysis was performed on prepandemic sera collected as part of the Airwave Health Monitoring Study before August 2019.12 Round 2 used a different cohort of 500 prepandemic sera (Airwave2) from that used previously (Airwave1).4

Test selection for clinic

In round 1 of the React 2 programme,4 five LFIAs were initially evaluated using finger prick testing in the clinic. These LFIAs also underwent sensitivity and specificity analyses on sera from the assembled cohort and 500 prepandemic sera, respectively. A further six LFIAs underwent sensitivity analysis and four of these achieved sufficient sensitivity to proceed to specificity testing on 500 prepandemic sera. In round 1, specificity was performed on LFIAs that showed a sensitivity of more than 80%. The two best performing LFIAs, which also showed high specificity scores of 99.8% (Surescreen, Panbio), were identified for potential clinic evaluation in round 2 (fig 1).

Fig 1
Fig 1

Timeline and selection process for lateral flow immunoassay (LFIA) evaluation. Further information about flow of participants is included in supplementary figures i and ii. *Biosure (Mologic) II was tested with 5 µL serum (according to instructions provided at time). Manufacturer advises LFIA should be performed with 10 µL serum. React 2=real-time assessment of community transmission 2

In this study, we report the results of round 2 of the study in which seven further LFIAs were initially selected for evaluation based on the manufacturer’s performance and published data, if available (supplementary table ii), before undergoing sensitivity analysis. LFIAs with a laboratory sensitivity greater than the sensitivity of the Fortress LFIA (>84%) proceeded to specificity testing on 500 prepandemic sera. The AbC-19 LFIA was selected for clinic evaluation because it showed the best performance in the laboratory and it has the potential to be procured at scale. Altogether, three LFIAs were selected for testing in the clinic in round 2: Panbio (Abbott13), AbC-19 (TT3, Abingdon rapid test consortium14), and Surescreen.15

Participant recruitment in clinic

Clinical recruitment of participants took place over two periods. Round 2a ran from 17 June to 2 July 2020 and tested the Panbio and Surescreen tests in parallel in the same participants. Round 2b ran from 4 to 21 September 2020 and tested the AbC-19 LFIA. People who worked in one of five NHS hospitals in two NHS trusts were invited to participate. Additionally, participants who had been involved in previous rounds were invited to reattend to test new LFIAs. No participants tested the same LFIA more than once. Supplementary figure ii presents a detailed summary of the flow of participants during sensitivity evaluation in the clinic.

Eligibility criteria were broadened in round 2 (supplementary fig iii). People who were known to be or had been seropositive based on formal laboratory antibody testing performed before attending the study clinic, or people with PCR confirmed SARS-CoV-2 infection (or both) were included. Additionally, family members of staff could participate provided they had also received a positive PCR result or were previously confirmed to be seropositive on formal laboratory antibody testing. Finally, participants who were admitted to hospital with covid-19 and were previously excluded from the study could also take part. Participants were enrolled after a minimum of 21 days had passed from symptom onset or positive PCR result (whichever occurred earlier). People considered seropositive for SARS-CoV-2 antibodies on previous finger prick LFIA only were excluded.

Clinic procedure

Study clinics were run at two sites in round 2a and at a single site in round 2b. Participants were required to provide evidence of the result and date of a previous positive SARS-CoV-2 PCR or laboratory based antibody test, and to complete a questionnaire. The questionnaire included demographic information, medical history, and information detailing the timing, duration, and severity of illness caused by SARS-CoV-2 infection.

Each participant performed one or two LFIAs using capillary blood through finger prick, under the supervision of a research nurse or practitioner, before analysis against reference standards. Participants followed the protocol provided by the manufacturer and verbal instructions from trained research staff in the clinic to ensure that the test was performed correctly. Interpretation of the LFIA result by the participant and trained observer was recorded independently and photographs of the completed tests were obtained. At each attendance, a venous blood sample was taken for laboratory testing. Invalid or failed tests, where the control line was absent, were excluded from the analysis and participants repeated the test.

To enable direct comparison of performance with capillary blood in clinic and sera in the laboratory, one LFIA (AbC-19) was retested by a research technician using matched sera from clinic participants, according to the manufacturer’s protocol (supplementary table i).

Sample size

Sample size was calculated assuming 90% power, SARS-CoV-2 infection prevalence of 100%, and expected test sensitivity of 85%. To evaluate sensitivity with a two sided delta of 10%, a target sample size of 153 participants was calculated. For specificity, a sample size of 361 was calculated based on an expected specificity of 98% with a lower limit of 95%.

Performance analysis

Statistical analysis was performed as previously described.4 The primary outcome of the study was the sensitivity and specificity of each LFIA in detecting SARS-CoV-2 IgG antibodies. Sensitivity analysis included performance on finger prick self-testing (participant interpretation), finger prick self-testing (observer interpretation), and serum in the laboratory. Two comparisons were made: against confirmed previous SARS-CoV-2 infection (by PCR swab or previous laboratory antibody test) and against confirmed positive results by S-ELISA or hybrid DABA from venous samples taken at the study clinic appointment. As previously described, specificity was calculated as the proportion of known negative samples that were negative on the LFIAs. Data are presented using a binomial confidence interval of 95% and significance was denoted by a P value less than 0.05.

For comparison of clinic and laboratory performance of individual LFIAs, agreement was assessed using the κ statistic with the following interpretation: less than 0, poor agreement; 0.00-0.20, slight agreement; 0.21-0.40, fair agreement; 0.41-0.6, moderate agreement; 0.61-0.8, good agreement; and higher than 0.8, almost perfect agreement16 Analysis of antibody concentrations is presented using quantitative S-ELISA data for round 2a and 2b. S-ELISA titres showed a skewed distribution and were log10 transformed. Continuous variables are presented as medians and interquartile ranges; the Kruskal-Wallis test was used to compare continuous variables across rounds. We used Dunn’s test to evaluate pairwise comparisons when the Kruskal-Wallis test was rejected. The McNemar test was used to test differences for dependent groups. We used Fisher’s exact test to test differences between independent groups. To further explore test sensitivity, participants were divided into quarters according to S-ELISA antibody reactivity, and test sensitivity was assessed for each category. Statistical analyses were performed on Stata (version 14.2, StataCorp, TX).

Patient and public involvement

Public involvement and participant feedback have been central to the design of the React 2 programme. There has been extensive involvement from patient panels and rigorous evaluation of the usability of LFIAs included in the React 2 studies has been undertaken.5 User expressed feedback during clinics, and formal evaluation of instruction materials provided to manufacturers through patient panels have been incorporated in reports to companies.

Results

Table 1 and table 2 show full results for sensitivity and specificity testing of nine LFIAs tested in round 2 using sera in the laboratory. The Fortress LFIA, previously studied in round 1, was retested against the Airwave2 prepandemic sera and attained a specificity of 99.0% (95% confidence interval 97.7% to 99.7%). The overall specificity of Fortress LFIA was 98.8% (97.9% to 99.4%) against 1000 prepandemic serum samples. Table 1, table 2, and supplementary table iii present the sensitivity of the composite reference standards S-ELISA and hybrid DABA when tested on participants from round 1, 2a, and 2b.

Table 1

Sensitivity results for LFIAs tested on round 1, 2a, and 2b positive sera and specificity results for LFIAs tested on prepandemic negative sera. Data are percentages (95% confidence intervals)

View this table:
Table 2

Validation of positive and negative sera on reference standards. Data are percentages (95% confidence intervals)

View this table:

Forty six of 48 participants who attended clinic in round 2a, and 51 of 52 who attended clinic in round 2b underwent finger prick testing and were included in the analysis (supplementary fig ii). Table 3 and supplementary table iv present their baseline characteristics. The sensitivity of the Surescreen and Panbio LFIAs, prioritised from round 1, were 88% (82.5% to 92.2%) and 91% (85.5% to 94.3%), respectively on testing of sera, and both had 99.8% specificity (98.9% to 100%).

Table 3

Baseline characteristics of participants in round 1, 2a, and 2b (assessed on Fortress, Surescreen and Panbio, and AbC-19 lateral flow immunoassays, respectively). Data are medians (interquartile ranges) unless stated otherwise

View this table:

The Surescreen LFIA tested in the clinic attained a sensitivity of 86% (72.7% to 94.8%) compared with the benchmark ELISAs, which was not significantly different (P=0.8) than it had shown against round 1 sera in the laboratory or significantly higher (P=0.772) than the Fortress LFIA. By contrast, AbC-19 and Panbio tests showed a lower sensitivity on finger prick capillary blood in the clinic compared with sera in the laboratory (table 1, table 2, and fig 2). The sensitivity of Panbio dropped significantly (P=0.018) from 91% (85.5% to 94.3%) in the laboratory with round 1 sera to 77% (61.4% to 88.2%) on finger prick blood in clinic versus S-ELISA or hybrid DABA.

Fig 2
Fig 2

Sensitivity of lateral flow immunoassays for finger prick (self-read), finger prick (observer read), matched serum, and initial analysis with serum. S-ELISA data include sensitivity results versus the in-house SARS-CoV-2 S-ELISA; PCR or antibody confirmed data include sensitivity results versus previous confirmed SARS-CoV-2 infection by PCR or formal laboratory antibody test result. PCR=polymerase chain reaction; SARS-CoV-2=severe acute respiratory syndrome coronavirus 2; S-ELISA=spike protein enzyme linked immunoassay

Despite attaining the highest sensitivity (100%, 98.1% to 100%) during laboratory testing with stored sera in the laboratory, the sensitivity of AbC-19 LFIA with capillary blood through finger prick also reduced significantly (P<0.001) to 69% (53.8% to 81.3%) versus S-ELISA or hybrid DABA. Supplementary table v includes additional sensitivity estimates for each LFIA by method of SARS-CoV-2 diagnosis (PCR v previous formal laboratory antibody testing) and by symptom severity.

Given the possibility that the cohort tested on AbC-19 had low antibody titres, matched serum samples from the participants evaluated in the clinic were tested in the laboratory on the AbC-19 LFIA and a sensitivity of 92% (80.0% to 97.7%) was found. This result was significantly higher than the sensitivity on finger prick testing in the clinic performed on the same participants (P=0.007). Concordance between the sensitivity for finger prick and for sera from these matched samples, determined by κ score, was slight (0.07, 95% confidence interval −0.14 to 0.28) in contrast to moderate (0.56, 0.25 to 0.86) for matched samples previously tested on the Fortress LFIA.

Because of low rates of new SARS-CoV-2 infection locally at the time of round 2 clinic testing, we anticipated that a proportion of participants might have been infected at an early stage of the pandemic. The round 2b cohort had a significantly longer period since symptom onset of 173 days compared with 90 days for round 2a and 59 days for round 1 (table 3 and supplementary fig iii; P<0.001). To account for the possibility that waning of antibody titres had led to an underestimation of the AbC-19 LFIA sensitivity performance in clinic, the distribution of serum antibody concentrations was analysed for round 1, 2a, and 2b on S-ELISA (supplementary fig iii). The median antibody concentration on S-ELISA was significantly different for round 1, 2a, and 2b (P<0.001). Of samples with antibody titres in the lowest quarter (≤2.51 log10 S-ELISA or 325 ng/mL), the Fortress LFIA detected 8/12 positive samples on finger prick testing compared with 8/21 on AbC-19 (supplementary figs iv and v). Additionally, when tested using sera matched to participants in clinic, the AbC-19 LFIA detected 20/21 samples as opposed to 8/21 from the same participants tested by finger prick in clinic (supplementary table vi). Supplementary table vi presents sensitivity results for all antibody concentration quarters for the LFIAs tested on finger prick in clinic and matched sera (if applicable).

Discussion

This study shows that LFIA sensitivity is variable on serum and finger prick testing, and often differs from that stated by the manufacturer. Specificity of all LFIAs that underwent this analysis was high. One further LFIA (Surescreen) is identified as suitable for use in seroprevalence studies because it showed comparable performance to the LFIA currently used in the React 2 seroprevalence studies (Fortress). However, the performance of Surescreen was not significantly better than Fortress; as a result, Fortress is still considered the most suitable candidate for ongoing rounds of the React 2 programme.

LFIAs remain an important tool in the assessment of population seroprevalence of SARS-CoV-2 infection. Evaluation of new LFIAs often relies on performance using sera in the laboratory with only a minority of studies evaluating whole blood or capillary finger prick testing, which ultimately is the intended use.17 An accurate assessment of the performance of LFIAs with capillary blood is key to interpreting large scale seroprevalence studies, and before clinical implementation, given the reduced sensitivity compared with laboratory analysis for some tests.418

Laboratory testing is an essential component of LFIA evaluation.1518 In this study, as in previous work, we have shown that most LFIAs evaluated had lower sensitivity or specificity than reported in preliminary results by the manufacturers. Only three LFIAs in round 2, and eight of 18 LFIAs evaluated in the React 2 programme to date showed sufficient sensitivity and specificity during analysis on sera to justify progression to clinic testing.

Of the three LFIAs tested in the clinic in this study, two showed a significant difference in sensitivity between serum and finger prick testing in two tests. AbC-19, the best performing test in the laboratory (100% sensitivity, 95% confidence interval 98.1% to 100.0%; and 99.8% specificity, 98.9% to 100.0%), showed the lowest sensitivity (69%, 53.8% to 81.3%) compared with S-ELISA or hybrid DABA upon finger prick testing in the clinic. Of the remaining two tests, only the Surescreen LFIA showed a marginally higher sensitivity than the Fortress LFIA, which has been used in previous rounds of the React national seroprevalence study.

Several potential reasons could explain differences in test performance between the laboratory and clinic. A possibility is that with sequential testing, participants recruited later after acute infection are likely to have lower antibody titres. Time elapsed post symptom onset was considerably different between all three rounds. Additionally, by broadening the inclusion criteria to include those with positive serology only (as opposed to PCR positivity), later cohorts might represent milder disease or be more likely to contain participants with false positive results. However, these more varied presentations of previous SARS-CoV-2 infection are a more accurate reflection of the population for which the LFIAs are intended to be used than healthcare employees testing positive on previous PCR alone.

Distribution of antibody titres was substantially different across all three periods of testing. A rapid fall in new SARS-CoV-2 infections locally after the first wave of the pandemic, and the timing at which new LFIAs became available for evaluation meant that there was a considerable difference in time since symptom onset in rounds 1, 2a, and 2b. Therefore, we considered whether a possible threshold effect relating to antibody concentration could account for a drop off in sensitivity observed between laboratory and clinic testing. However, on evaluation of the lowest quarter of serum antibody concentration from samples across all three rounds, the Fortress LFIA detected a numerically higher proportion of samples with a low antibody titre on finger prick testing than AbC-19. Additionally, the median antibody titre in round 2a was higher than that of round 1; despite this, the Panbio LFIA also showed a (non-significant) difference in sensitivity between serum and finger prick testing. Finally, the considerable difference in sensitivity on the AbC-19 LFIA between finger prick and sera testing on matched samples from round 2b suggests other factors in test design and use could be more important.

Study limitations

Several limitations are acknowledged in this study. Important differences were found between cohorts and corresponding antibody titres. Additionally, while laboratory staff were blinded to the results of LFIA interpretation on finger prick testing, participants and research staff in the clinic were aware that all participants had evidence of previous SARS-CoV-2 infection. Therefore, LFIA interpretation in the clinic was not blinded, which could have led to an overestimation of LFIA sensitivity on self-testing. In round 2a, participants were evaluated on two LFIAs in the same appointment and it is possible that the result of the test evaluated first could have influenced the interpretation of the second test.

Conclusions

This study confirms the importance of assessing LFIAs in the intended population because laboratory results might not accurately predict performance in the clinic. We have shown that analyses should account for changes in antibody levels over time and the comparison of tests on a consistent cohort of laboratory sera remains an important part of the evaluation. Through a robust approach to LFIA evaluation, we characterised the performance of nine LFIAs, and identified one new LFIA with performance comparable to the Fortress LFIA, which has been used successfully in large seroprevalence studies. For our React 2 programme, a new LFIA would have to perform considerably better than the Fortress LFIA to outweigh the scientific value in repeating seroprevalence surveys using the same assay. At this time, no LFIA offers enough improvement in performance to merit inclusion in subsequent React 2 seroprevalence studies. Additionally, no LFIA has reached the standard set by the Medicines and Healthcare products Regulatory Agency for individual use.

What is already known on this topic?

  • Since the start of the coronavirus disease 2019 (covid-19) pandemic, over 200 lateral flow immunoassays (LFIAs) have been developed to detect severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) antibodies

  • Performance of these LFIAs is variable

  • Previous studies have focused on laboratory performance, not on capillary blood testing, which is the intended use

What this study adds

  • The LFIAs analysed showed high specificity, but variable sensitivity on sera and finger prick testing

  • One new LFIA was found to be suitable for use in seroprevalence studies, but none of the LFIAs tested meet the criteria for individual use

Ethics statements

Ethical approval

This study was conducted in concordance with the World Medical Association’s Declaration of Helsinki throughout. Ethical approval was granted for this work as part of the React 2 study approvals by South Central-Berkshire B Research Ethic Committee (REC ref: 20/SC/0206; IRAS 283805). Prepandemic negative control sera were provided by the Airwave Health Monitoring Study, which is approved by North West-Haydock Research Ethics Committee (REC ref: 19/NW/0054).

Data availability statement

Anonymised data are available on reasonable request and can be provided by the corresponding author.

Acknowledgments

We thank all the participants who volunteered for finger prick testing to help with this study. We extend our gratitude to Margaret-Anne Bevan, Helen Stockmann, Danielle Davy, Chloe Wood, Billy Hopkins, Miranda Cowen, Norman Madeja, Nidhi Gandhi, Vaishali Dave, and Narvada Jugnee who ran the React 2 antibody testing clinics. We are grateful for the support from the Imperial National Institute for Health Research Biomedical Research Centre. We acknowledge support from the Huo Family Foundation for our work on covid-19.

Footnotes

  • Contributors: All listed authors made substantial contributions to the conception or design of the work; or the acquisition, analysis, or interpretation of data for the work; and drafting the work or revising it critically for important intellectual content; final approval of the version to be published; and agreement to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved. MM and AD contributed equally to this work; GSC and WSB contributed equally to this work. The corresponding author attests that all listed authors meet authorship criteria and that no others meeting the criteria have been omitted. GSC is the guarantor.

  • Funding: This work was supported by the Department of Health and Social Care, and the National Institute for Health Research (NIHR) Biomedical Research Centre of Imperial College NHS Trust. WSB is the Action Medical Research chair in virology; GSC is supported by an NIHR professorship; AD and HW are NIHR senior investigators; DA and PE are emeritus NIHR senior investigators; PC, MM, and RT hold intellectual property rights on the hybrid DABA and this work was supported by a UK Research and Innovation/Medical Research Council grant (reference MC_PC_19078). The work in PC laboratory is supported by the Francis Crick Institute (FC001061), which receives its core funding from Cancer Research UK, the UK Medical Research Council, and the Wellcome Trust. We acknowledge support from the Huo Family Foundation for our work on covid-19. The sponsor is Imperial College London. The funders had no role in the production of this manuscript.

  • Competing interests: All authors have completed the ICMJE uniform disclosure form at www.icmje.org/coi_disclosure.pdf and declare: support from the Department of Health and Social Care, NIHR Biomedical Research Centre of Imperial College NHS Trust, UK Research and Innovation/Medical Research Council, Francis Crick Institute, Cancer Research UK, the Wellcome Trust, Huo Family Foundation and Imperial College London for the submitted work; PC, MM, and RT hold intellectual property rights on the hybrid DABA; no other relationships or activities that could appear to have influenced the submitted work.

  • The lead author (the manuscript’s guarantor) affirms that the manuscript is an honest, accurate, and transparent account of the study being reported; that no important aspects of the study have been omitted; and that any discrepancies from the study as planned (and, if relevant, registered) have been explained.

  • Dissemination to participants and related patient and public communities: Once published, the results of this study will be circulated to Imperial College Healthcare and Chelsea and Westminster NHS staff who participated in the study. The results of the study will be provided to all participants who request to see them.

  • Provenance and peer review: Not commissioned; externally peer reviewed.

http://creativecommons.org/licenses/by/4.0/

This is an Open Access article distributed in accordance with the terms of the Creative Commons Attribution (CC BY 4.0) license, which permits others to distribute, remix, adapt and build upon this work, for commercial use, provided the original work is properly cited. See: http://creativecommons.org/licenses/by/4.0/.

References