The RT97 has a duplexer built in. A duplexer allows the receiver portion of a repeater to hear while the transmitter is transmitting. That’s why the transmitter must transmit at a separate frequency. A duplexer is a filter to prevent the outgoing signal from the transmitter from being received by the receiver.
The duplexer must be precisely tuned to the frequency you program into the radio. If not, the repeater simply won’t work, unless you use two antennas, one for transmit and one for receive, and separate them, usually vertically on a tower.
I don’t know if that’s your problem, but unless you program your RT97 to match how the duplexer is tuned, your repeater won’t.
450m-520m sounds like the bandwidth or range of frequencies your RT97 can be programmed to (450 MHz to 520 MHz). That says nothing about what frequency it’s actually set to, just its capabilities.
There’s no special programming that makes it work better with an Anytone 878/868. The RT97 is strictly analog FM. Boxcar already explained that the receive frequencies on the repeater are the converse of what’s on any radios you intend to use.
Best wishes. I’m hopeful that you can get it sorted out. My limited understanding of duplexer tuning informs me that I don’t have the right equipment. It would be easier to program the radio to match how the duplexer is already tuned.