Upcoming Storage Change -- Got Back-ups?
-
TL;DR: Format of data in
storagedirectory will soon change. Make sure you are backing up the contents of that directory in its entirety, and you preserve your backups for an extended period, particularly the backup you take right before upgrading to the build containing this change (date of that is still to be determined, but soon). The old data format will remain readable (so you'll be able to read your pre-change backups) for the foreseeable future.In support of a number of other changes in the works, I have found it necessary to change the storage format for Reactor objects in
storageat the physical level.Until now, plain, standard JSON has been used to store the data (everything under the
storagedirectory). This has served well, but has a few limitations, including no real support for native JavaScript objects likeDate,Map,Set, and others. It also is unable to store data that contains "loops" — objects that reference themselves in some way.I'm not sure exactly when, but in the not-too-distant future I will publish a build using the new data format. It will automatically convert existing JSON data to the new format. For the moment, it will save data in both the new format and the old JSON format, preferring the former when loading data from storage. I have been running my own home with this new format for several months, and have no issues with data loss or corruption.
A few other things to know:
- If you are not already backing up your
storagedirectory, you should be. At a minimum, back this directory up every time you make big changes to your Rules, Reactions, etc. - Your existing JSON-format backups will continue to be readable for the long-term (years). The code that loads data from these files looks for the new file format first (which will have a
.dvalsuffix), and if not found, will happily read (and convert) a same-basenamed.jsonfile (i.e. it looks forruleid.dvalfirst, and if it doesn't find it, it tries to loadruleid.json). I'll publish detailed instructions for restoring from old backups when the build is posted (it's easy). - The new
.dvalfiles are not directly human-readable or editable as easily as the old.jsonfiles. A new utility will be provided in thetoolsdirectory to convert.dvaldata to.jsonformat, which you can then read or edit if you find that necessary. However, that may not work for all future data, as my intent is to make more native JavaScript objects directly storable, and many of those objects cannot be stored in JSON. - You may need to modify your backup tools/scripts to pick up the new files: if you explicitly name
.jsonfiles (rather than just specifying the entirestoragedirectory) in your backup configuration, you will need to add.dvalfiles to get a complete, accurate backup. I don't think this will be an issue for any of you; I imagine that you're all just backing up the entire contents ofstorageregardless of format/name, that is the safest (and IMO most correct) way to go (if that's not what you're doing, consider changing your approach). - The current code stores the data in both the
.dvalform and the.jsonform to hedge against any real-world problems I don't encounter in my own use. Some future build will drop this redundancy (i.e. save only to.dvalform). However, the read code for the.jsonform will remain in any case. - This applies only to persistent storage that Reactor creates and controls under the
storagetree. All other JSON data files (e.g. device data for Controllers) are unaffected by this change and will remain in that form. YAML files are also unaffected by this change.
This thread is open for any questions or concerns.
- If you are not already backing up your








