Hello there,
How about take your outputs from the Q' outputs instead of the Q outputs ? (Q' represents the "Q not" output)
That will give you a down count with the normal connection of Q outputs to next stage CLK inputs which is the way you normally connect them for an up count. So you'd just be taking your outputs from the inverted Q outputs that's the only difference.
In fact, you can get an up count and a down count at the same time this way simply by taking the up count from the Q outputs and the down count from the Q' outputs. Note however that when the up count is at 0000 the down count is at 1111.