The novel, highly transmissible severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) has triggered a pandemic of acute respiratory illness worldwide and remains a huge threat to the healthcare system’s capacity to respond to COVID-19. Elderly and immunocompromised patients are at increased risk for a severe course of COVID-19. These high-risk groups have been identified as developing diminished humoral and cellular immune responses. Notably, SARS-CoV-2 RNA remains detectable in nasopharyngeal swabs of these patients for a prolonged period of time. These factors complicate the clinical management of these vulnerable patient groups. To date, there are no well-defined guidelines for an appropriate duration of isolation for elderly and immunocompromised patients, especially in hospitals or nursing homes. The aim of the present study was to characterize at-risk patient cohorts capable of producing a replication-competent virus over an extended period after symptomatic COVID-19, and to investigate the humoral and cellular immune responses and infectivity to provide a better basis for future clinical management. In our cohort, the rate of positive viral cultures and the sensitivity of SARS-CoV-2 antigen tests correlated with higher viral loads. Elderly patients and patients with diabetes mellitus had adequate cellular and humoral immune responses to SARS-CoV-2 infection, while immunocompromised patients had reduced humoral and cellular immune responses. Our patient cohort was hospitalized for longer compared with previously published cohorts. Longer hospitalization was associated with a high number of nosocomial infections, representing a potential hazard for additional complications to patients. Most importantly, regardless of positive SARS-CoV-2 RNA detection, no virus was culturable beyond a cycle threshold (ct) value of 33 in the majority of samples. Our data clearly indicate that elderly and diabetic patients develop a robust immune response to SARS-CoV-2 and may be safely de-isolated at a ct value of more than 35.