There are a wide variety of cells in the market, and if you know what's inside your battery and have a technical bent, the charging and storage temperatures can be looked up. Some cells are rated for storage at -40C and charged at 0C.
Most owners will not have that info, so best to use the commonly quoted numbers of storage/operation down to -20C (4F) and recharge above 10C (50F). Freezing will damage the chemistry if you try to recharge. Cold temperatures will reduce the capacity if you run the battery, but won't damage it.
I don't bring my bikes inside, here in northern Illinois, We (wife/I) will bundle up and ride when it's above 40F, but stay home when it's icy, Once in a great while, I'll want to recharge and it's too cold (under 40F) in the garage. Then I will put the battery in a plastic bag to let it warm up in the house without condensation.