Growing evidence suggests that fine particulate matter (PM2.5) likely increases the risks of dementia, yet little is known about the relative contributions of different constituents. Here, we conducted a nationwide population-based cohort study (2000 to 2017) by integrating the Medicare Chronic Conditions Warehouse database and two independently sourced datasets of high-resolution PM2.5 major chemical composition, including black carbon (BC), organic matter (OM), nitrate (NO3−), sulfate (SO42−), ammonium (NH4+), and soil dust (DUST). To investigate the impact of long-term exposure to PM2.5 constituents on incident all-cause dementia and Alzheimer’s disease (AD), hazard ratios for dementia and AD were estimated using Cox proportional hazards models, and penalized splines were used to evaluate potential nonlinear concentration–response (C-R) relationships. Results using two exposure datasets consistently indicated higher rates of incident dementia and AD for an increased exposure to PM2.5 and its major constituents. An interquartile range increase in PM2.5 mass was associated with a 6 to 7% increase in dementia incidence and a 9% increase in AD incidence. For different PM2.5 constituents, associations remained significant for BC, OM, SO42−, and NH4+ for both end points (even after adjustments of other constituents), among which BC and SO42− showed the strongest associations. All constituents had largely linear C-R relationships in the low exposure range, but most tailed off at higher exposure concentrations. Our findings suggest that long-term exposure to PM2.5 is significantly associated with higher rates of incident dementia and AD and that SO42−, BC, and OM related to traffic and fossil fuel combustion might drive the observed associations.