Despite the importance of the developing cerebellum on cognition, the associations between physical fitness and cerebellar volume in adolescents remain unclear. We explored the associations of physical fitness with gray matter (GM) volume of VI, VIIb and Crus I & II, which are cerebellar lobules related to cognition, in 40 (22 females; 17.9 ± 0.8 year-old) adolescents, and whether the associations were sex-specific. Peak oxygen uptake (V̇O2peak) and power were assessed by maximal ramp test on a cycle ergometer, muscular strength with standing long jump (SLJ), speed-agility with the shuttle-run test (SRT), coordination with the Box and Block Test (BBT) and neuromuscular performance index (NPI) as the sum of SLJ, BBT and SRT z-scores. Body composition was measured using a dual-energy X-ray absorptiometry. Cerebellar volumes were assessed by magnetic resonance imaging. V̇O2peak relative to lean mass was inversely associated with the GM volume of the cerebellum (standardized regression coefficient (β) = −0.038, 95% confidence interval (CI) -0.075 to 0.001, p = 0.044). Cumulative NPI was positively associated with the GM volume of Crus I (β = 0.362, 95% CI 0.045 to 0.679, p = 0.027). In females, better performance in SRT was associated with a larger GM volume of Crus I (β = −0.373, 95% CI -0.760 to −0.028, p = 0.036). In males, cumulative NPI was inversely associated with the GM volume of Crus II (β = −0.793, 95% CI -1.579 to −0.008 p = 0.048). Other associations were nonsignificant. In conclusion, cardiorespiratory fitness, neuromuscular performance and speed-agility were associated with cerebellar GM volume, and the strength and direction of associations were sex-specific.