Anyone else experience initial exposure being worse?

I'm starting to get into the rhythm of identifying exposures in my day-to-day life (50% of buildings are water damaged). I feel like I'm getting better at using my intuition to guide me, but one thing is tripping me up:

Let's say I enter a moldy building for the first time (in a while). I will quickly feel symptoms coming on and flag the building as likely bad. The problem is that if I enter the building again on day 2, the exposure is less obvious, and symptoms are slower to materialize and not quite as severe, to the point where I start to doubt my assessment from day 1.

Is this an experience others share?