Background: Rapid IgM/IgG antibody tests were largely used in lieu of RT-PCR tests as part of COVID-19 public health response activities in Lima, Peru. To assess their utility, we explored the relationship between the time since onset of several COVID-19-related symptoms and the sensitivity of a rapid combined IgM/IgG antibody test. Methods: We collected data from a community sample of individuals (n = 492) who received concurrent RT-PCR and rapid IgM/IgG antibody testing between May 2020 and March 2021. We estimated the sensitivity of the antibody test, against the RT-PCR test, by weeks since symptom onset via segmented regression analysis. Results: The overall sensitivity of the rapid IgM/IgG antibody test was 46.7% (95% CI, 42.4–51.2%). Among 372 (75.6%) participants who reported COVID-19-related symptoms, sensitivity increased from 30.4% (95% CI, 24.7–36.6%) in week 1 after symptom onset to 83.3% (95% CI, 41.6–98.4%) in week 4. The test sensitivity increased by 31.9% (95% CI, 24.8–39.0%) per week until week 2 to 3, then decreased by − 6.0% (95% CI, − 25.7–13.7%) per week thereafter. Conclusion: Rapid antibody tests are a poor substitute for RT-PCR testing, regardless of presenting symptoms. This highlights the need for future pandemic planning to include timely and equitable access to gold-standard diagnostics, treatment, and vaccination.