Do a KCL:
5A + 2Vx/2 = Vx/4. Therefore, Vx = -20/3
This is wrong; you left out the current in the dependent source.
Let Id be the current in the dependent source. Then you would have:
5A + 2Vx/2 = Vx/4 + Id
But, interestingly, you got a value for Vx which is exactly the negative of the correct value.
To solve for the open circuit output voltage, Vth, note that the current from the current source must pass to ground through the 4Ω and 6Ω resistors. Let V1 be the same as Vx and let V2 be the voltage at the top of the 6Ω resistor. Thus we can write:
V1/4 + V2/6 = 5
The difference between V2 and V1 is just the voltage provided by the dependent source, so we can write:
V2 - V1 = 2*V1 (since V1 is the same as Vx)
Solving these two equations simultaneously, we get:
V1 = 20/3
V2 = 20
so Vth = 20
Now let's calculate the short circuit output current. This is the current through the 2Ω output resistor when its right end is connected to ground.
This changes our previous first equation to:
V1/4 + V2/6 + V2/2 = 5
We still have for the second equation:
V2 - V1 = 2*V1
Solving these two simultaneously, we get:
V1 = 20/9
V2 = 20/3
The short circuit output current, Isc, is then (20/3)/2 = 10/3
Rth is given by Vth/Isc = 20/(10/3) = 6Ω