Possible problems:
The differential input impedance of your circuit is unreasonably high, at around 1 Mohm (R3 to R38). It would be better if you could lower the impedance to something in the range of a few hundred ohms, but I don't know what your source impedance is, so cannot say for sure what is the best value. If you are sending high bandwidth signals through the coax, you may also need to pay attention to impedance matching, which may force your load impedance down to 75 or 50 ohms.
Construction is important in cases like this. We need to know how you connected the shield of each coax cable, as this is not shown on your schematic. Please also explain how you have connected C103 and C107 since these are your common mode input filters and need to be connected to a common point.
Definite problems:
Your schematic is not drawn well. You should put all components on one page and directly connect the G and S nodes so that the signal flow is more clear. You should also use standard symbols if you want others to understand your circuit. In this case I mean that the method you use to show "No-connects" as if they were the legs of a spider on your op amp symbols is non standard. You should take greater care in allowing enough room so that your capacitor symbols don't touch each other and so that your component labels don't overlap any other lines or symbols.