Get a rear view mirror that doesn't stick out past your bar ends. That seems like a reasonable solution. I had a bar end mirror for a while but it got destroyed by hitting something as your described. I purchased a better design that was under the grip on the left side.
The reason why bar end...