I'm not suggesting that you add another IC plus 8 transistors to your current design. I'm talking about your next design.
Use four transistors as "sinking" column drivers and use a 74HC595 and eight transistors as "sourcing" segment (row) drivers. The OE (Output Enable) line on the 74HC595 can be used for full "fade to black" PWM brightness control.
Since you're using sockets for the LEDs, you should be able to install the four column driver transistors underneath the LED sockets, if you need to. The 74HC595 and eight segment driver transistors shouldn't take up too much board space. You might also consider using an SMD version of the 74HC595.
Mike