Take a look of the cover of this example relay. 12V means how much voltage you need to power up the magnetic coil that connect and disconnect the relay's contacts.
There are different coil voltages, including 5, 6, 12 or 24VDC. You can find 120VAC coil types, like the following one.
View attachment MHA.jpg
On the first picture, the second marking CONTACT 250V/10A indicates the maximum voltage and current that the relay contacts can handle. As far as I know doesn't matter if it's AC or DC, someone more experienced will correct me if I'm wrong.
Now, some relay types require that the current that powers the coils must be DC. If you try to power that type of relay using AC, it will turn on and off 50 or 60 times per second, rendering it useless (and maybe destroying the unit).
I understand that you want an extremely simple design, but using only a relay presents some flaws.
1. Your lights may turn on during a daytime outage or when they are not really needed, wasting the batteries.
2. If you decide to use disposable batteries, changing the batteries of every room of the house would be tedious.
3. Using rechargeable batteries require a charge controller circuit to avoid overcharging and over-discharging.
4. I'm not sure if i'm correct of this one, but I think that continuous energization of the relay coil may shorten its life.
Spending a little more money on a proper circuit may be a better investment.