Introduction: Physical activity (PA) has been suggested to protect against old-age cognitive deficits. However, the independent role of childhood/youth PA for adulthood cognitive performance is unknown. This study investigated the association between PA from childhood to adulthood and midlife cognitive performance. Methods: This study is a part of the Cardiovascular Risk in Young Finns Study. Since 1980, a population-based cohort of 3596 children (age, 3–18 yr) have been followed up in 3- to 9-yr intervals. PA has been queried in all study phases. Cumulative PA was determined in childhood (age, 6–12 yr), adolescence (age, 12–18 yr), young adulthood (age, 18–24 yr), and adulthood (age, 24–37 yr). Cognitive performance was assessed using computerized neuropsychological test, CANTAB (N = 2026; age, 34–49 yr) in 2011. Results: High PA in childhood (A = 0.119; 95% confidence interval [CI], 0.055–0.182) and adolescence (A = 0.125; 95% CI, 0.063–0.188) were associated with better reaction time in midlife independent of PA in other age frames. Additionally, an independent association of high PA in young adulthood with better visual processing and sustained attention in midlife was observed among men (A = 0.101; 95% CI, 0.001–0.200). There were no associations for other cognitive domains. Conclusions: Cumulative exposure to PA from childhood to adulthood was found to be associated with better midlife reaction time. Furthermore, cumulative PA exposure in young adulthood and adulthood was associated with better visual processing and sustained attention in men. All associations were independent of participants PA level in other measured age frames. Therefore, a physically active lifestyle should be adopted already in childhood, adolescence, and young adulthood and continued into midlife to ensure the plausible benefits of PA on midlife cognitive performance.