There is increasing evidence linking long-term fine particulate matter (PM2.5) exposure to negative health effects. However, the relative influence of each component of PM2.5 on health risk is poorly understood. In a cohort study in the contiguous United States between 2000 and 2017, we examined the effect of long-term exposure to PM2.5 main components and all-cause mortality in older adults who had to be at least 65 years old and enrolled in Medicare. We estimated the yearly mean concentrations of six key PM2.5 compounds, including black carbon (BC), organic matter (OM), soil dust (DUST), nitrate (NO3–), sulfate (SO42–), and ammonium (NH4+), using two independently sourced well-validated prediction models. We applied Cox proportional hazard models to evaluate the hazard ratios for mortality and penalized splines for assessing potential nonlinear concentration–response associations. Results suggested that increased exposure to PM2.5 mass and its six main constituents were significantly linked to elevated all-cause mortality. All components showed linear concentration–response relationships in the low exposure concentration ranges. Our research indicates that long-term exposure to PM2.5 mass and its essential compounds are strongly connected to increased mortality risk. Reductions of fossil fuel burning may yield significant air quality and public health benefit.