Sorry about your luck. Some of it may just be coincidence.
I used to get colds all the time until I got in the habit of washing my hands immediately post work out. I also avoid using my hands to wipe sweat from my face during my workout. Have had just 2 colds in the last 2 years.