That's excatly the problem. Like Pommie said, you must turn segments off before switching digits.
Imagine it, old segment info is active, you switch digit, the second digit gets this old data, then you change segment data and only now second digit gets this new data.
Oh you're right! I had switched the order of when the segments are turned on, but that doesn't even matter, because the old data is still in the register. I do have to turn off the segment completely.
Thanks guys!