Oh Stephan, you certainly can.
It depends on they amount of use, the type of ammunition used, how the weapon is maintained and the environment it's used in (dusty, salty, etc.), but you certainly can even with pistols wear them out or cause enough wear to seriously affect the accuracy even with...