Yes, there is a formula but it's a bit complex.
I knew that the internet has a better explanation than I could give you so I opened 'google' (www.google.com) and typed in "heatsink calculations".

And here it is:http://us.st.com/stonline/books/ascii/docs/2651.htm

You can do your own calculations now
Have fun,
Klaus

Well, in general you calculate a heatsink this way:

You got:
Ptot = max. power produced by the semiconductor in Watt
Tambient = ambient temperature in Kelvin

From the datasheet:
Tmax = max. temperature of the semicond.
Rjc = junction - case thermal resistance in K/W

We calculate:
Rja = junction - ambient thermal resistance
Rca = case - ambient thermal resistance (-> the heatsink)

Rja = (Tmax - Tambient) / Ptot ... (Tmax, Tambient in Kelvin; Ptot in watt)

Rca = Rja - Rjc

Rca is the max. thermal resistance your heatsink should be. i.e. if you got 3.5K/W I would use a heatsink of 3K/W - just to be sure

We assume that the heatsink is mounted with thermal conducting matter (you know, the white stuff between the heatsinks and the semiconductors),
so we can neglect this additional thermal resistance. If you want to get sure, assume a max. thermal resistance between case and heatsink Rch with 0.2K/W:
Rca = Rja - Rjc - Rch

I hope this is all correct!

Best regards,
Hannes / Austria

PS: if your circuit is in a box, Tambient can easy be about 10-15K higher than the ambient temperature. Make sure air can easily get into and out of the box.