One more clue: it does not happen when the rides start, only after a few minutes or riding...
What does that say to you? I have checked wheels, brakes, handlebar, chain, pedals, saddle, ... but I am no expert so it could still be one of those things. (I am still thinking that is a cable/plastic...