We studied the effects of mother–infant interaction and maternal pre- and postnatal psychological distress on children’s social-emotional problems and competences, as well as whether interaction quality moderates the association between distress and children’s outcomes. Maternal pre- and postnatal psychological distress were measured using the SCL and EPDS questionnaires, whereas mother–infant interaction was measured when the child was 8 months old using the EA Scales. Children’s social-emotional development was measured using the BITSEA questionnaire at 2 years old and using the SDQ questionnaire at 4 years old, where higher maternal structuring was associated with fewer social-emotional problems in children and higher maternal sensitivity was associated with greater social-emotional competence in children at 2 years old. Further, higher postnatal distress was found associated with greater social-emotional problems at 2 years old, though neither these effects nor moderating effects at 4 years old were observed after multiple-comparison corrections. Our findings support direct associations of both mother–infant interaction and maternal postnatal psychological distress with children’s social-emotional development during toddlerhood.