I see some valid points in your post but I certainly don't feel that the remotes are fiddly and definitely prefer them over constantly glancing down at the TCU to change the power settings and having to remove either hand from the bar.
As for the drive side port, I didn't really encounter any issue with squeezing the plug ends through the holes. Yes, one hole is 5mm and the other is 4mm but I pulled each one individually through the larger hole and never had to drill out and enlarge the smaller hole. If both holes were 4mm then it likely would have required some drilling. It would be nice to know what replacement ports Specialized is including in the parts package with new bikes. Perhaps someone in the know can enlighten us. Part # for it is mentioned in the instructions that came with the remotes.
That being said, I could see the buttons being difficult to locate if they were buried under bar wrap but having them exposed is certainly not a problem for me and represents a much safer method over manually pushing the button on the TCU.
I think that there is going to be compromise in any scenario whether it be the Creo or the Levo remote.