I've only done the inner walls of the hole itself (if that makes sense). Stick knife in hole (carefully to not hit powder coat, obviously), scrape the perimeter.
I assume inside would also work, the only point is that some part of the jack makes a reliable connection.
I guess technically only clearing the inner walls might not be reliable since it could be possible to have the jack be positioned perfectly in the middle of the hole, not touching any walls, in which case clearing inside the enclosure instead like KR Sound showed would be better, but in practice I've never had that happen.