The problem with the H11F1M (and similar opto isolators) is the small range of linear operation. If you look at Figure 2 in the data sheet you will see that the voltage across the device must be less than ±0.05V to stay within the linear resistance range (the near vertical part of the curves). If the voltage is greater than that, the device saturates and limits the signal to a constant current (the horizontal part of the curves).
Thus in order for that device to operate as a variable resistor without significant distortion, you would need to limit the voltage across it to about ±50mV pp.
If you put a resistor in series with the emitter to ground of the transistor you are using to drive the device, you will get a more linear response (500Ω would give about 2mA of collector current per volt of base voltage). Also if you put a Silicon diode in series with the bottom of the pot element to ground you will get the needed 0.6V offset to bias the transistor on at the minimum pot rotation.