First, the power transistor has ~ the same voltage across it as the 317, except that it has 4A flowing through it, so (20-5)*4 = 60W.
If your heat sink's thermal resistance is 3.5 degC/W, then with 75W, the temperature of the heat sink would be 25degC (ambient temperature) + 3.5*75 = 25 + 262degC = 287degC, which is way, way too hot.
The max temperature you want at the heatsink is ~85degC (still hot enough to burn your hand). Starting from a 25degC ambient, that would mean that the temperature rise of your sink above ambient would have to be 85-25 = 60degC. Since you are dissipating 75W, the thermal resistance of the sink would be 60degC/75W or 0.8degC/W, which you will find will require like 4000cm². Are you convinced that this is dead end, yet?
This is not the way you should be doing this! To get 5V * 5A = 25W to the load, you are dissipating 60+15=75W in the regulator. This is where you should be using a buck switch-mode regulator. Alternatively, start with a different transformer so that the unregulated voltage is much closer to 5V.