The simplest way, with the lowest component count, and smallest size, would be to use a micro-controller. A PIC 16F628 would do nicely, your total project would consist of the 16F628, the LED display, 10 series resistors, two push button switches, two pull-up resistors for the switches (you could use the weak pull-ups on PortB instead of these), and a 5V supply.
By using a SM chip you could make it really small, but even without SM, an 18 pin DIL chip isn't very big.
You would have to write the software though, but my tutorials contain most of what you would need.