When someone is listening to you over the radio, the only thing they know about you is going to be your callsign, assuming you aren't an idiot and have not broadcast your full name for everyone to hear.
So usually, over the radio, which is what the context of this discussion is, it is VERY difficult to look up someone's address without going through the FCC website.