Depends on what sort of RL you are doing. If you are trying to train agents to play small games with vision the agent will need a small cnn to process images. This will need a gpu and what you have should be enough.

I was training on atari for a while with 1080ti. The games run on the cpu so you need a decent cpu as well.