I only use one remote. The tv remote. I just enable HDMI-CEC

I keep the Apple TV remote around for extremely rare situations where that doesn’t work but even then, my cell phone has a built in Apple TV remote as well, which makes it even less necessary