You could use a CD40110 (up/down counter 7-segment decoder) controlled by push-buttons . That will allow you to remove points, in case of an error, as well as add them.
You can use one push-button to count up and one push-button to count down. Each push of the button changes the count by one. Both switches must have a debounce circuit.
If you want to get fancy, you could add a auto-increment circuit, which would change the count at a slow rate (say 2 counts per second) after you held the button down for a short-period.