Gordon,You failed to calibrate the levels to and from the machine relative to what the dbx wants to see, regardless of whether you use it or not. Even when bypassed, the levels are very critical; otherwise, the dbx will get confused. Also, I hope you never attempted to calibrate the dbx unit itself - this is a job best left to the pros with the right gear and knowledge.
The secret is to start by sending 1 kHz to the machine (in Input), then switching the dbx in and out, adjusting the oscillator from the console until there is no (or minimal) level shift. If you can accomplish this, you're giving the machine the signal it wants. This may not agree with the signal your console is sending or with the machine's VU meters (at the moment). If so, that's another issue.
Once you are sending the correct signal, switch the dbx Out and adjust the input cal for 0 VU. I'm assuming you calibrated the reference tape to 0 VU. If not, do that also. Then (assuming the bias is correctly adjusted) re-adjust record level so that output and input agree. Also, keep in mind that the meters, as I recall, reflect the dbx compression (when used). Switch the dbx "out" to check levels, then "in" to record knowing the meter will move less because the compression ratio is 2:1. Properly aligned, the dbx should work as intended, making the machine usable for something other than rap and speed metal.