Its going to be a messy calculation, since there are many factors that come into play. How many layers, ambient temperature, etc. Derating could be as low as 75% of capacity, for 1 layer, up to 25% for say 6 or more layers. It's the combined mutual heating of the layers that cause the derating.
Maybe some company has a similar item, like a self coiling garage extension, and they have done the derating already, and you could base your estimate on theirs.
Try a google search for "derating cables on reels" or similar.