I dunno. If I wanted to find out, I would write some equations...
However, intuitively, if you think about what is happening with the RC charging curve, there is only one instant in time as the capacitor charges when the product of voltage*current matches the maximum power transfer point. If you could keep the source loaded such that the downstream load is always at 2.5V/20.833mA = 120Ω for the entire time the capacitor is charging, then it seems it would happen faster than the simplistic case.
To try this out, I did this simple sim.
First circuit is the brute force RC charge-up. Note that the dark blue trace is the voltage vs time of C1 and the red trace is the power extracted from the source. Note that the peak power happens as you would expect when C1 is at 2.5V, and the peak power is ~52mW.
Second circuit charges C2 with a magic behavioral current source which automatically adjusts its current so that it's power output is a constant 52mW. The green trace is the voltage at C2, and the light blue trace is the power output of B1 (i.e. constant at 52mW).
Note that the charging curve of C2 is always ahead of C1. Not a huge win, but the second circuit always wins, especially if you insist on charging C1 all the way to 5.000V
Now, all you have to do is figure out how to build B1....