The cost-benefit ratio of group living is thought to vary with group size: individuals in ‘optimally sized’ groups should have higher fitness than individuals in groups that are either too large or too small. However, the relationship between group size and individual fitness has been difficult to establish for long-lived species where the number of groups studied is typically quite low. Here, we present evidence for optimal group size that maximizes female fitness in a population of geladas (Theropithecus gelada). Drawing on 14 years of demographic data, we found that females in small groups experienced the highest death rates, while females in mid-sized groups exhibited the highest reproductive performance. This group size effect on female reproductive performance was largely explained by variation in infant mortality (and, in particular, by infanticide from immigrant males) but not by variation in reproductive rates. Taken together, females in mid-sized groups are projected to attain optimal fitness due to conspecific infanticide and, potentially, predation. Our findings provide insight into how and why group size shapes fitness in long-lived species.