Understanding individual variability in response to physical activity is key to developing more effective and personalised interventions for healthy ageing. Here, we aimed to unpack individual differences by using longitudinal data from a randomised-controlled trial of a 12-month muscle strengthening intervention in older adults. Physical function of the lower extremities was collected from 247 participants (66.3 ± 2.5 years) at four time-points. At baseline and at year 4, participants underwent 3 T MRI brain scans. K-means longitudinal clustering was used to identify patterns of change in chair stand performance over 4 years, and voxel-based morphometry was applied to map structural grey matter volume at baseline and year 4. Results identified three groups showing trajectories of poor (33.6%), mid (40.1%), and high (26.3%) performance. Baseline physical function, sex, and depressive symptoms significantly differed between trajectory groups. High performers showed greater grey matter volume in the motor cerebellum compared to the poor performers. After accounting for baseline chair stand performance, participants were re-assigned to one of four trajectory-based groups: moderate improvers (38.9%), maintainers (38.5%), improvers (13%), and decliners (9.7%). Clusters of significant grey matter differences were observed between improvers and decliners in the right supplementary motor area. Trajectory-based group assignments were unrelated to the intervention arms of the study. In conclusion, patterns of change in chair stand performance were associated with greater grey matter volumes in cerebellar and cortical motor regions. Our findings emphasise that how you start matters, as baseline chair stand performance was associated with cerebellar volume 4 years later.