I did some more testing, switched R6 out for a trimmer so I could bias Q1 at exactly 5V. That didn't change much, but switching R4 to a 1k made the gain range much more useable.
However, the level is still quite weak. At full blast it's fine, but could go a bit higher, and then 90% of the pot's...