If you leave out the LM317, then if the solar panel can deliver more than 0.5A, you will still need a current limiter to protect the battery. As a minimum, you will need a power FET to disconnect the panel from the battery when the correct voltage is reached.