I couldn't reproduce the noise with the bike upside down, but I tested it without the motor, and although I could hear the noise less, it was still there.
I heard two fairly loud “clicks” recently, when I was pushing hard on the pedal. I'm going to clean it up and test it again.
The bike was...