SARS-CoV-2 Omicron (B.1.1.529) BA.4 and BA.5 sub-lineages, first detected in South Africa, have changes relative to Omicron BA.1 including substitutions in the spike receptor binding domain. Here we isolated live BA.4 and BA.5 viruses and measured BA.4/BA.5 neutralization elicited by BA.1 infection either in the absence or presence of previous vaccination as well as from vaccination without BA.1 infection. In BA.1-infected unvaccinated individuals, neutralization relative to BA.1 declines 7.6-fold for BA.4 and 7.5-fold for BA.5. In vaccinated individuals with subsequent BA.1 infection, neutralization relative to BA.1 decreases 3.2-fold for BA.4 and 2.6-fold for BA.5. The fold-drop versus ancestral virus neutralization in this group is 4.0-fold for BA.1, 12.9-fold for BA.4, and 10.3-fold for BA.5. In contrast, BA.4/BA.5 escape is similar to BA.1 in the absence of BA.1 elicited immunity: fold-drop relative to ancestral virus neutralization is 19.8-fold for BA.1, 19.6-fold for BA.4, and 20.9-fold for BA.5. These results show considerable escape of BA.4/BA.5 from BA.1 elicited immunity which is moderated with vaccination and may indicate that BA.4/BA.5 may have the strongest selective advantage in evading neutralization relative to BA.1 in unvaccinated, BA.1 infected individuals.