Tmax > T + P × Rt
Tmax- maximum junction temperature
T - maximum expected ambient temperature
Rt -thermal resistance (Celsius/Watt or Kelvin/Watt) - the sum of all the thermal resistances between junction and ambient (heatsink and thermal grease included if applicable). Treating this variable as multiple terms and solving for the one representing the heatsink is how you size the heatsink. If you have multiple thermal paths to ambient, you sum up the thermal resistances in each path separately (like series resistors) and then run those sums through the same equation you use to find the equivalent of in parallel parallel resistors to get a single equivalent thermal resistance. One example might be if you have a top-side heatsink on your IC but it also has significant heatsinking to ambient through the PCB.
P - power dissipation - I^2 x R if it's juts constant current with no high frequency PWM switching. If high frequency switching is involved then you have account for that power dissipation and heating as well. It's rather complicated to calculate though. You'll have to do some reading if you are interested in that. Also, do not forget to account for R increasing as you heat up. There is a graph that shows you the resistance vs temperature. To figure it out exactly you have to start with any temperature (a guess really) and use the corresponding resistance in the calculation to get a new temperature, then go back to the chart to see if the new temperature matches the one you started with. If it doesn't you go through the process again using the corresponding resistance for the new temperature. Or...you could just assume worst case and use the resistance when the FET is at maximum temperature (which is usually 1.5x to 2x the room temperature resistance). You can look at the chart for a more accurate number.