We consider an extension of the Standard Model with a singlet sector consisting of a real (pseudo)scalar and a Dirac fermion coupled with the Standard Model only via the scalar portal. We assume that the portal coupling is weak enough for the singlet sector not to thermalize with the Standard Model allowing the production of singlet particles via the freeze-in mechanism. If the singlet sector interacts with itself sufficiently strongly, it may thermalize within itself, resulting in dark matter abundance determined by the freeze-out mechanism operating within the singlet sector. We investigate this scenario in detail. In particular, we show that requiring the absence of inflationary isocurvature fluctuations provides lower bounds on the magnitude of the dark sector self-interactions and in parts of the parameter space favors sufficiently large self-couplings, supported also by the features observed in the small-scale structure formation.