If the guy is fat, really old, a lonely weirdo or asthmatic I could see how he is "offended" with you getting out to enjoy the great outdoors. Otherwise carry some mace and be ready to use it should he confront you on the street. And/or invest in a good action camera.
