Upcoming Storage Change -- Got Back-ups?

[toggledbits]

TL;DR: Format of data in storage directory will soon change. Make sure you are backing up the contents of that directory in its entirety, and you preserve your backups for an extended period, particularly the backup you take right before upgrading to the build containing this change (date of that is still to be determined, but soon). The old data format will remain readable (so you'll be able to read your pre-change backups) for the foreseeable future.

In support of a number of other changes in the works, I have found it necessary to change the storage format for Reactor objects in storage at the physical level.

Until now, plain, standard JSON has been used to store the data (everything under the storage directory). This has served well, but has a few limitations, including no real support for native JavaScript objects like Date, Map, Set, and others. It also is unable to store data that contains "loops" — objects that reference themselves in some way.

I'm not sure exactly when, but in the not-too-distant future I will publish a build using the new data format. It will automatically convert existing JSON data to the new format. For the moment, it will save data in both the new format and the old JSON format, preferring the former when loading data from storage. I have been running my own home with this new format for several months, and have no issues with data loss or corruption.

A few other things to know:

• If you are not already backing up your storage directory, you should be. At a minimum, back this directory up every time you make big changes to your Rules, Reactions, etc.
• Your existing JSON-format backups will continue to be readable for the long-term (years). The code that loads data from these files looks for the new file format first (which will have a .dval suffix), and if not found, will happily read (and convert) a same-basenamed .json file (i.e. it looks for ruleid.dval first, and if it doesn't find it, it tries to load ruleid.json). I'll publish detailed instructions for restoring from old backups when the build is posted (it's easy).
• The new .dval files are not directly human-readable or editable as easily as the old .json files. A new utility will be provided in the tools directory to convert .dval data to .json format, which you can then read or edit if you find that necessary. However, that may not work for all future data, as my intent is to make more native JavaScript objects directly storable, and many of those objects cannot be stored in JSON.
• You may need to modify your backup tools/scripts to pick up the new files: if you explicitly name .json files (rather than just specifying the entire storage directory) in your backup configuration, you will need to add .dval files to get a complete, accurate backup. I don't think this will be an issue for any of you; I imagine that you're all just backing up the entire contents of storage regardless of format/name, that is the safest (and IMO most correct) way to go (if that's not what you're doing, consider changing your approach).
• The current code stores the data in both the .dval form and the .json form to hedge against any real-world problems I don't encounter in my own use. Some future build will drop this redundancy (i.e. save only to .dval form). However, the read code for the .json form will remain in any case.
• This applies only to persistent storage that Reactor creates and controls under the storage tree. All other JSON data files (e.g. device data for Controllers) are unaffected by this change and will remain in that form. YAML files are also unaffected by this change.

This thread is open for any questions or concerns.

[toggledbits]

OK, everyone, it's almost time. Sorry for the long pause. Life takes over sometimes.

The aforementioned updates will be in the next build, and I am working on wrapping everything up to get that build out later this week.

Please make backups of your Reactor data as advised in the head post here. As I said then/there, I've been working with this for months now, with no issues, but my world is not your world or everyone else's world, so there's always the possibility I don't see or have an issue that you do.

Prior to releasing this new build in the latest channel, the 26011 build will become the stable channel head. In addition to your backups, that gives you a relatively quick path (especially docker users) to get back onto 26011 if there's a showstopper.

This build will also include unified room groups: DynamicGroupController will manage "rooms" (or areas or locations or whatever your hub calls them). If you have two different hubs with devices in the "Living Room," there will be one "Living Room" group with the combined set of devices. The per-controller room groups generated by VeraController and EzloController will be disabled by default, and existing room groups created by these controllers will be marked as dead entities (and eventually purged). If you already use DynamicGroupController to manually create your own room groups via configuration, you can either keep that (and disable DGC's new behavior, if you wish) or switch to DGC's version. An updated version of ZWaveJSController that supports this functionality will be released simultaneously with the core build.