A true peak detector does exactly what the name implies. It detects and holds the highest voltage that a signal has ever been. It does not average the signal. Even if the voltage was at that point for only a microsecond once, that is the peak voltage that will be detected and held. The cap is charged to the peak voltage, but the diode prevents any discharge.
Fundamentally, a peak detector is a diode and a capacitor. Quite often, the diode gets wrapped in an op-amp circuit resulting in what is called a precision rectifier. What the op-amp does is offset the voltage drop of the diode so that it can be used with sub millivolt signals.
To get an average of a voltage, you would use a resistor and a capacitor. then the cap charges and discharges based on the time and magnitude of the signal input signal.
Note: The description above is assuming an ideal circuit. In reality, series inductance and resistance can inhibit the ability of a cap to capture very short peaks. And leakage currents in the cap, diode and other components can cause the peak voltage to drift over time. Nevertheless, practical detectors can be built to hold a peak voltage for a few seconds. In many cases, a resistor will be places in parallel with the cap to drop the peak over time. Or, a switch may be placed across the cap to reset the peak value to zero.