Not a deal-killer in normal use. You adapt to the minor unbalance without thinking. Might be different with a very heavy single pannier, but guessing you're not hauling rock most days.
What you might notice, though, is the difference in handling when you carry the same weight low in a pannier...