To assess the relationship between physical activity (PA) in work, transport, domestic, and leisure-time domains (with sitting time included) and health-related quality of life (HRQoL) among young adult men. The long version of IPAQ and SF-36 Health Survey were used to assess PA and HRQoL, respectively, in 1425 voluntary 20- to 40-year-old Finnish male participants. Participants were divided into tertiles (MET-h/week): Lowest tertile (<38 MET-h/week), Middle tertile (38-100 MET-h/week), and Highest tertile (>100 MET-h/week). The IPAQ domain leisure-time PA predicted positively the Physical Component Summary (PCS) (β = 0.11, 95% CI: 0.06 to 0.16) and Mental Component Summary (MCS) (β = 0.11, 95% CI: 0.05 to 0.16) dimensions. Occupational PA predicted negative relationships in the PCS (β = −0.13, 95% CI: −0.19 to −0.07), and sitting time predicted negative relationships in the MCS dimension (β = −0.13, 95% CI: −0.18 to −0.07). In addition, a linear relationship was found between total PA level (including sitting time) and all of the IPAQ domains (<0.001). The Middle tertile had the highest leisure-time PA (38% of total PA), whereas the highest sitting time (28%) and lowest occupational PA (8%) were found in the Lowest tertile. The Highest tertile had the highest occupational PA (61%), while the leisure-time PA was the lowest (16%). Different PA domains appear to have positive and negative relationships to mental and physical aspects of HRQoL. Relatively high leisure-time PA indicated a better HRQoL regardless of the amount of total PA, while occupational PA and higher daily sitting time related negatively to HRQoL.