Yes, you need pullups on either your rows or columns. You don't need (mustn't have) the pull downs as the way you read the pad is by making 1 pin an output and setting it low. I.E. if you make A0 output and low then A3, 6 & 7 will be low if the top line of switches were pressed. A1 will give the next row and so on.
Mike.