Innovation and Technology Today published a helpful article on 7 basic steps you should take to secure IoT / Smart Home devices. Number 4 on that list suggests disconnecting devices from the internet when they are not in use. Disconnected devices are useless to hackers, immune to viruses and invisible to searching bots.
If you look at the hours of the day that you use your smart speaker, control your smart lights, thermostat, garage door, et., usage is concentrated to an hour or so in the morning, and a few hours at night on weekdays for most people. Keeping all devices connected 24X7 adds little to no convenience but increases the liabilities. There is no reason to keep them connected to the internet when your sleeping or away from home, and when they are disconnected you have one less thing to worry about.