If you're bulking, I would go for at least 1%.
I think soon enough it will be discovered that there are some anabolic/"performance" fats in dairy. Think about how drinking milk is associated with heightened serum IGF-1 levels. It can't be a function of the calcium/galactose/protein content, so it has to be a product of either lipid &/or enzymatic properties in the dairy. I haven't found a "smoking gun" abstact yet, but given how much milk research I've read, I'm pretty convinced there are a lot of beneficial growth factors in dairy, with organic dairy obviously being the best source.
Milk fat also has documented anti-carcinogenic properties, that I know for sure.
Plus, just from a common sense standpoint, all the different constituents in milk seem to act together in synergy. BCAA-rich protein, calcium, ACE-inhibiting enzymes, galactose, sphingomyelin, butyric acid--milk is one of the best examples of natural synergy in regards to a food product's contents that you really can't duplicate through artificial means. It represents more than the sum of its parts essentially, from a nutritional standpoint. My theory is if nature put some fat in there, it's doing something. I think drinking whole milk is unnecessary calories and saturated fat, but I think it makes some sense to drink 1 or 2% if you're on a mass phase, assuming you're complementing your diet with EFAs to begin with.