I never claimed to be the end all authority on radio design; however, I do have experience in HF radio design, and one thing I learned while working in this area is that the RF front stage sets many parameters that will dictate the overall radio performance. IP3 is one, Noise figure is the other. If you ignore these then you can expect the sort of audio sample that you have supplied for us earlier.
My suggestion to you is this:
First change your RF front end to a part with Better IP3 specs, and NF, and in your case NF would be a big thing. Also if you have the gear, test your front end for these parameters.
A weakly designed front end will ripple on down to the rest of your already whimpy unit... and your output will just be droopy.
Have you ever done a cascaded gain distribution of your design which included NF and IP3.
There are Gain distribution spreedsheets available.