For less than 1µF, ceramic or film capacitors are typically used. Above that use aluminum electrolytic (cheapest) or tantalum electrolytic (more expensive but smaller and better performance).
With a 9v battery this is a routine application. Use a 50 - 100 uF aluminum electrolytic. An additional 1-10 uF Tantalum is often added for noise supression. The Tantalum is faster acting. Is the load noisey, such as a small motor? If its just LED's etc, then anything will work. If the circuit includes a microcontroller than noise suppresion is essential.
Note that a 1.7 drop out may somewhat limit battery life. You can get low dropout regulators < 1 volt, for example the LM2940.
Yes you are correct. However even a motor on the battery will inject noise into the electronics. Apparenltly no motors are involved so the issue is theoretical.