This study examines the direct and indirect effects of home numeracy and literacy environment, and parental factors (parental reading and math difficulties, and parental education) on the development of several early numeracy and literacy skills. The 265 participating Finnish children were assessed four times between ages 2.5 and 6.5. Children’s skills in counting objects, number production, number sequence knowledge, number symbol knowledge, number naming, vocabulary, print knowledge, and letter knowledge were assessed individually. Parents (N = 202) reported on their education level, learning difficulties in math and reading (familial risk, FR), and home learning environment separately for numeracy (HNE) and literacy (HLE) while their children were 2.5 years old and again while they were 5.5 years old. The results revealed both within-domain and cross-domain associations. Parents’ mathematical difficulties (MD) and reading difficulties (RD) and home numeracy environment predicted children’s numeracy and literacy skill development within and across domains. An evocative effect was found as well; children’s skills in counting, number sequence knowledge, number symbol identification, and letter knowledge negatively predicted later home numeracy and literacy activities. There were no significant indirect effects from parents’ RD, MD, or educational level on children’s skills via HLE or HNE. Our study highlights that parental RD and MD, parental education, and the home learning environment form a complex pattern of associations with children’s numeracy and literacy skills starting already in toddlerhood.