Objectives Changes in socioeconomic status (SES) during life may impact health in old age. We investigated whether social mobility and childhood and adulthood SES are associated with trajectories of health-related quality of life (HrQoL) over a 17-year period. Methods We used data from the Helsinki Birth Cohort Study (n = 2003, 46% men, mean age 61.5 years). Social mobility was derived from childhood SES, obtained from healthcare records, and register-based adulthood SES. Results Logistic regression models showed that lower adulthood SES was associated with lower physical HrQoL trajectories. Among men low (OR 3.95, p < .001), middle (OR 2.20, p = .006), and declining lifetime SES (OR 2.41, p = .001) were associated with lower physical HrQoL trajectories compared to men with high SES. Socioeconomic status was not associated with mental HrQoL trajectories. Discussion Declining SES during life course may have negative health consequences, while improving SES is potentially as beneficial as high SES to later-life health among men.