The mother's genes affect how many babies more than any other factor. Some inbred laboratory lines consistently have 3-4 pups, and other consistently have 18+. The cage size shouldn't matter much, if at all...
My experience has been that when my mice are in one of the bigger boxes, it takes them MUCH longer to breed, if they do at all. I prefer to keep breeding pairs in the smaller boxes that have a nesting box of some kind and not much to play on. I keep the toys out because it seems the doe has less places to hide or get away from the buck.
I have not experienced a noticable differnce in the number of babies between a small box and big box. I personally prefer to have them in a shorter smallish box when the pups are born. I move them to a larger box when they start moving around, and then to a nursery cage when they are 4 weeks that has a gazillion things to climb up on and in. I do not give the younger babies toys because it makes them harder to catch and in my opiion less social.
Maria, I agree completely about removing clutter from the 'honeymoon suite', but I don't think tank size matters. I've gone both ways, using a smallish plastic CritterCarrier for some, using 5 1/2 gallon for others. The only difference that I see comes with the buck regarding whether or not he has the right behaviors toward the doe. Bucks that chase and harass the doe have problems, where as bucks that groom and cuddle get 'er done. I also feel that you're right about having a doe in a smallish tank up until birth, and not cluttering whatever tank you use so the little ones don't learn to run and hide. Sometimes I use a nesting box in a tank for just the first week, and then remove it. Also, I don't put wheels in with litters of younglings, ever since I saw one go flying off to hit the plexiglass; it illuminated some deaths among the babies, and reinforced the need to use wheels with solid ng surfaces.
You're new! I just noticed...posting before I finish my first cup of coffee...welcome!! *yawn* Welcome!
If the groups in the smaller cages aren't overcrowded, then it shouldn't make much of a difference, no. As long as the buck has scented all the territory, he'll feel comfortable getting cozy with the does. If they're a little cramped for space, though, in your smaller cages, stress could be affecting their willingness to copulate. Sex comes second to comfort for them, so if they're stressed out for any reason they won't give you litters as frequently. You could try giving them things to climb on and in, to try and increase the surface area in the cage where mice can be, and see if that increases your rates. Usually, though, in my experience so far, some bucks are just more eager breeders than others.