Hello there,
You also need to check the voltage level across the two 150uf caps, as i see you did use 35v rated devices and that's a little close to the actual voltage you might measure there, but at high line the voltage could go above 35v that's why i was saying that 50v devices would be preferred. That's quite important too or the caps could become damaged. Do you have any 50v caps around?
The caps on the rectifier are determined by calculating the min voltage of the ripple waveform with full load current flowing. The idea is to have enough capacitance so that for any low line condition the rectifier still puts out enough voltage to run the load properly. For example, if the load requires at least 12v (including regulator overhead) and 100uf allows the voltage to dip below 12v (say 11v) then we need to add more capacitance. If we add 50uf in parallel with the 100uf and that keeps the voltage above 12vdc (at low line) then we're good to go, except we might also want to factor in a little capacitor aging in which case we can make the total capacitance 150 percent higher, which would work out to about 220uf for this example.
Most people do a simulation to determine if the capacitance is acceptable, but i have a special program developed for these rectifier circuits that is much faster than a simulation if you would like to try it. With a decent computer the simulations are not that slow anymore though anyway, so if you have a circuit simulator you can try that first if you like. You should realize though that in the real world there is always some input series resistance presented by the windings on the transformer, and that helps with the filtering too although it does lower the output voltage a little more. You can estimate this resistance by measuring the input and output characteristics of the transformer but im not sure if you want to be bothered by those details, especially since you have test equipment you can use to measure the real world performance once you get it up and running.
The other caps (0.1uf) are taken from what the manufacturer recommends, although you have the right idea already: check it with a scope and look for oscillations. You need to check at both low line and at high line too however, which you can take to be plus or minus 15 percent of the nominal line voltage. You should also check your max dc voltage level across the caps at high line too as i was saying above. If you dont have a variac on hand to raise and lower the input line voltage, you can estimate by multiplying the voltage measured across the caps at nominal line by 1.15 for high line and 0.85 for low line. You dont want the voltage going above this level or below the level needed for the LM317 to regulate into your required load (about 2.5v above output).