You might also find that you need a resistor for each LED. I realize this will significantly increase the parts count, but if you use only one resistor per column, the brightness of the LED's in that column will be directly related to how many are lit at any given time.
For example: if your resistor is sized to allow 25ma of current to pass and you have only one LED on the column lit, it will be very bright, as it will be using the entire 25ma. However, if you have 5 LED's on the column lit, the available 25ma will be distributed among them and each will use 5ma. If all 8 on the column are lit, the current available for each drops to 3.125ma.
Something to consider.