Do I have to leave Bluetooth on for my Kevo to work?

Yes, in order for the Kevo lock to recognize your eKey, Bluetooth must be enabled.