no no no, with 8 x 12 matrix, you turn all 8 of your one port on, with whatever pattern for that column. Then you turn the column on. This means LEDs are on for 1/12 (or 1/16th) of the time, not 1/96th.
since your LEDs use a resistor on the top port, none is required on the bottom, you can just send them straight to ground. This is a little harder with a decoder that drives to ground, but all you need to do is add another transistor to invert the drive... for example....
View attachment 68735