You are correct in that this repeater owner has no right to tell someone they can't access those frequencies. If the users are using the repeater for more than an incidental contact, then the owner does have the right to request they stop, or he can turn the repeater off for a period. The rules are a little murky regarding private repeaters, but the uncommon solution would be for the repeater owner to use some common sense regarding the fact the repeater is on-the-air to begin with.