Depends on your goals. If I'm just trying to lose a few pounds then no, I don't worry about losing the dairy. Now If I'm trying to lose some serious pounds then yeah, I cut out the dairy. The lactose is not needed while cutting. Hell, any form of sugar is not needed while cutting.