Objective: Sex steroid hormones potently shape brain functions, including those critical to maintain mental health such as serotonin signaling. Use of oral contraceptives (OCs) profoundly changes endogenous sex steroid hormone levels and dynamics. Recent register-based studies show that starting an OC is associated with increased risk of developing depression. Here, we investigate whether use of OCs in healthy women is associated with a marker of the serotonin system in terms of serotonin 4 receptor (5-HT4R) brain imaging. Methods: [¹¹C]SB207145-PET imaging data on 53 healthy women, of whom 16 used OCs, were available from the Cimbi database. We evaluated global effects of OC use on 5-HT4R binding in a latent variable model based on 5-HT4R binding across cortical and subcortical regions. Results: We demonstrate that OC users have 9–12% lower global brain 5-HT4R binding potential compared to non-users. Univariate region-based analyses (pallidostriatum, caudate, hippocampus, amygdala, anterior cingulate cortex, and neocortex) supported the global effect of OC use with the largest difference present in the hippocampus (−12.8% (95% CI [−21.0; −3.9], P_corrected = 0.03). Conclusion: We show that women who use OCs have markedly lower brain 5-HT4R binding relative to non-users, which constitutes a plausible molecular link between OC use and increased risk of depressive episodes. We propose that this reflects a reduced 5-HT4R gene expression, possibly related to a blunted ovarian hormone state among OC users.