Here's a video that (overly) dramatizes the differences in SWR and insertion losses for UHF connectors vs. N connectors. However, while the narrator makes it seem pretty dramatic, the actual numbers don't represent noticeable differences in real life. Reflected power isn't truly lost unless you are using lossy cable and the reflected power is converted to heat. However, for a person obsessed with having nearly perfect metrics, the differences might cause a loss of sleep.
Finally, a word about adapters. Adapters are convenient, but if you're ordering cable custom made, just get an N male on one end and a PL-259 on the other and make sure they're good connectors, such as M&P.