Yes, solder wire leads to the pot, and instead of board-mounting it, solder the wires to the board. Then you can move the board where ever you want, and leave the pot in place. Just snip the legs off the pot.
You have a ton of space in that enclosure, relative to the side of the board.