A4 is electrically different than other port pins in regards to output. It can pull the pin to ground when you write a 0 to the reg like any other pin. However it has no transistor to pull it up when you write a 1 to the reg. It is essentially tristated even though TRISA4 may be 0.
Be sure that for all other pins on PortA you may do, any time you set just one bit you assign the LAT, not the port itself. LATA0=1 not RA0=1. The reason being is that the RA0=0 means the hardware will read the current value of all PortA, even those defined as inputs, modifies, and writes it back to the port. So A1 may still be an input but may get a "1" written to it. That means when you change it to an output again, it will output a 1 instead of the 0 you wrote to it at the top of the code. The safer thing is to assign LATA1=0 before every time you assign TRISA1=0, but it's still not universally safe since ISRs can happen between the two instructions which write to another RAx pin.