To get really big as a natty you have to be fat too (or just finished precontest dieting or something of the like), your body does not want all that expensive muscle tissue hanging around without plenty of fat reserves to fuel it in case of a food shortage. Once upon a time we were all genetically hulking neanderthals, famines came and only us puny weaklings could scrounge enough calories to survive. So modern humans are generally built to be lighter and use a minimal amount of food to accomplish what we need to (which is not really lifting weights).
This all goes out the window when you start artificially altering your body's chemistry obviously