Multi-Master 6 SBC VME Bus Issue - At a LOSS Please Help!

[CBRFrenzie929]

I am currently working on a rather complex project involving 6 Power PC SBC's in a VME Chassis. All 6 SBC's are VME Bus masters running with geographical position priority arbitration (left most board highest priority and right most board lowest priority). We recently switched to a different manufacturer for our VME Chassis and are now experiencing an extremely perplexing problem with all of the boards going into a continuous reset loop after a couple hours of running. The reset signal is being asserted by the highest priority SBC (left most in the VME Chassis) via the SYSRESET line on the VME Bus after approx. 1.5 - 2 hours of running and then continues to do so indefinitely, or until the system is powered off and back on again. All Power PC SBC's are running a version of VxWorks as the Kernel with each SBC's having near identical Kernel configurations. My team is currently at a loss as to why a new VME Chassis would cause this behavior, as the project has been working successfully for several years prior to the VME Chassis change.

PLEASE IF ANYONE has seen something similar to this issue or has some sort of input that may be helpful I am all ears!!!! We are currently in the process of getting the software for a VME Bus analyzer approved for use and are hoping that tool may be of some help in resolving this issue.

Please and thank you to anyone and everyone that reads this post!

 

[AnalogKid]

Assuming that you have played musical chairs with boards in slot one, and the problem is at least semi-repeatable . . .

Since the problem appeared with the new chassis, it and the chassis almost certainly are related. The chassis might have a system monitor or power-on reset module that is independent of the POR in the slot 1 bus controller. The time lag feels like a thermal soak. Some power supplies have a POR output signal, a hidden something that might go sideways. Another hidden gem is an AC Fail signal out of the supply. When connected directly to the backplane, it can force a SYSRESET. And finally, the forgotten cousin, SYSFAIL. Any scope with 3 or more channels can capture transient activity on these three utility bus lines.

Who made the old chassis, and who makes the new one? Also, power supply make and model for the new chassis?

And, where are you located?

ak

 

[CBRFrenzie929]

Assuming that you have played musical chairs with boards in slot one, and the problem is at least semi-repeatable . . .

Since the problem appeared with the new chassis, it and the chassis almost certainly are related. The chassis might have a system monitor or power-on reset module that is independent of the POR in the slot 1 bus controller. The time lag feels like a thermal soak. Some power supplies have a POR output signal, a hidden something that might go sideways. Another hidden gem is an AC Fail signal out of the supply. When connected directly to the backplane, it can force a SYSRESET. And finally, the forgotten cousin, SYSFAIL. Any scope with 3 or more channels can capture transient activity on these three utility bus lines.

Who made the old chassis, and who makes the new one? Also, power supply make and model for the new chassis?

And, where are you located?

ak

Thank you AnalogKid for your reply! Thermal soak is one of the first things that came to mind for us as well. The power supply does have a FAIL output signal that is asserted when an overload or thermal issue takes place and we have monitored this signal. Unfortunately this signal is never asserted and the power supply rails all remain steady when the problem occurs. Also that signal is not connected to the VME chassis and in fact the only lines that are connected from the power supply are the +5.0V, +/- 12.0V and Ground connections. This is actually the second power supply we have connected to the chassis as we were trying to rule as many factors out from the beginning as possible. I currently have a 4 channel scope connected to the 3 power supply rails and the SYSRESET line on the utility bus line and have confirmed this. Although, you make a good point about the other 2 signals on the utility bus, AC Fail and SYSFAIL. I am going to setup a new test and monitor those lines to see what the relationship is between them and SYSRESET when the problem occurs.

I made another discovery recently that I should note and maybe this may be helpful. When any of the other SBC's are removed from the system (can be any board from #2-6) the problem does not occur and the system runs for several hours on end without issue. This really has us scratching our heads.... Also note, I have swapped out the board from slot #1 with a brand new board and the issue remains the same.

The old chassis' are made by APW Electronic Solutions and are probably 10+ years old. The new chassis is made by Schroff. We are located in the midwest USA.

Thank you for your thoughts and ideas! I will report back with my findings from this new test.

 

[AnalogKid]

Back in my day, Schroff made their own backplanes and I don't remember who MuPAc (APW) used. The three Utility Bus signal should have terminals or a connector somewhere, usually on the back side. If any of them have any wires, that's a trail to follow.

OTOH - the single board / multi board behavior smells like a bus arbitration issue. Are the old and new backplanes auto-bus-grant? If so, active or passive?

ak