If a house is sufficiently insulated, it need not actively reject heat outdoors, for example with air conditioners. It is sufficient to store the heat generated within by the inhabitants and their devices (lights, appliances, etc.) in a seasonal thermal store until the environment is cool enough for the heat to flow to it passively, given the opportunity.
What’s the worst-case scenario? One person dissipates around 2000 kcal/day, about 100W. Household appliances are usually a few times this; let’s say 400W in total.
Let’s suppose we store cold in the form of ice during the winter, and that it will have to last six months.
400 J/s · 6 months · 30 days/month · 86400 s/day = 6.2208 GJ
Melting ice consumes 75 kcal/kg.
6.2 GJ/person · (1/75) kg/kcal · 4.2 cal/J = 0.35 Gg/person = 350 Mg/person
350 tons of water: 350 m³, a little more due to the expansion of ice. At a height of 5 m, that would be 70 m²: basically a bedroom or two of space. Or, if it were below an entire floor of 16m × 16m = 256 m², it’d be 1½m of height of ice. Per person. If four people live in the house, it’ll be 6 meters of depth of ice below the house — 1 meter per month.
If we’re talking about storing cold in a material without a change of phase, we’d have just 20° or 30° of ΔT, which is 20–30 kcal/kg, if it were water, or 7–20 kcal/kg if it were something else — in total 4× to 10× the ice, by weight — 1400 to 3500 tons per person. Nevertheless, it could be a similar volume of rock or soil, since they are some 2× to 4× as dense as ice.
In practice, it’s probably possible in most places to radiatively couple to a comfortable temperature almost every day, so a six-month thermal store is extreme overkill. However, these calculations show that even a six-month purely passive thermal store is feasible.