Assuming you have common cathode displays, you could use a 74HC139.
Design for a LED current of 6 mA and parallel the inputs and outputs since the max output current is 25 mA and you need 6 * 4 = 24 mA which is close to the 25 mA limit. So you can guard against this by paralleling the 2 sections.