I'd make two observations: the first is that it seems the vast majority of e-bike fires I hear about involve homebuilt bikes with homebuilt batteries. The other observation is that literally millions of e-bikes are sold every year across Europe, Canada, the United States, Australia, and New Zealand. Yet only a handful of e-bike fires not involving homebuilt batteries occur every year.
A reasonable person would conclude that the risk of your e-bike catching on fire is probably somewhat less than 1 in 100,000, and probably substantially less than that if you aren't using a DIY system.
Note that a 1 in 100,000 fire risk is insanely high for a consumer product.
Having said all that, I'd recommend being damned careful with your batteries. If you drop them you should assume that you are going to have to dispose of them and purchase new ones. If you have a significant crash on your e-bike you should also assume that you should at a minimum have your batteries inspected and if any damage is found you should of course replace them.
Where I live charging outside isn't an option -- the environment outside is more flammable than the inside of my garage, and a battery fire outside would likely quickly be catastrophic. So at least during biking season I clear the truck out of the garage and charge the bike in the garage, where it has a whole vehicle stall all to itself and will be a safe distance from the walls and on a concrete floor.
For peace of mind if anything else I also have a kevlar hoverboard bag and some concrete flower pots that I store the e-bike batteries in during the off-season.
Also for peace of mind if anything else there is some safety in numbers -- if you are riding on a fairly popular brand of bike or fairly popular e-bike platform (e.g. Bosch) chances are you won't be the first lucky winner in the fire lottery and if there is some kind of defect you are more likely to be somewhat protected by a recall notice. For smaller bike manufacturers and no-name batteries the sample sizes are likely too small and it will be harder to detect a safety defect in the first place.