No, SWR isn’t the only concern. Depending on where you place a ground plane antenna there will be some directional bias. Ideally you would place it in the center of the ground plane, but because of the short wavelength of UHF, a ground plane antenna isn’t nearly as sensitive to placement and a lot of people are satisfied even with a ground plane antenna mounted at the very edge of a ground plane, such as the edge of a tailgate or hood.
As @WRYZ926 says, as long as you have at least 6 inches of ground plane in any direction from the mount you should be fine.