A "window comparator" circuit should do that.
There is an example configuration about a third of the way down this page:
**broken link removed**
An LM393 dual comparator should be suitable.
Connect the variable resistor in series with another resistor, eg. 3.3K, across power, and feed the window comparator from the junction.
Connect the green led, a normal diode and a resistor in series across power - LED cathode to ground, resistor to positive.
Connect the red LED between the comparator outputs (cathode) and the junction of the diode and resistor.
You probably need more like 4.5 - 6V for it to work properly.