Cliff notes: Use 300 ohm current limiting resistors on each cathode and don't worry about any other discreet transistors or switches for controlling the LEDs
Each individual pin is rated for 40 mA on the AVR, so assuming 10 mA per diode the 'master' pin would only have to sink 30 mA of current. Unless you want a single master to control ALL led banks....
If you're driving the chip at 5v, you'll need about a 300 ohm resistor between the output pins and the cathode.
With 32 IO pins, 4 pins per LED group, thats a max of 8 tri-color LEDs you could drive, 24 sources and 8 sinks. 240 mA source and sink if all LEDs are lit. The Vcc and GND pins are rated at 200 mA, but I'm not certain if its 200 mA EACH pin or 200 mA TOTAL. I think its each pin, and on the PQFP there are 2 Vcc pins (not counting AVcc which is PORTA power) and 2 GND pins (not counting AGND which also should be grounded). The PDIP only has one each so technically you could exceed the ratings, IF you're driving the max possible LED banks, but you're probably not....