Organizing/ structuring rule sets and rules

[toggledbits]

Build 21258

• Warning and clear when changing operator on condition that would clear condition options.
• Fix Vera importer geotag handling.
• VeraController: add binary x_vera_geotag.in_region attribute.
• HassController: implement x_hass_system.call_service on system entities.

[toggledbits]

Build 21267

• Change the evaluation algorithm for global variables to make them reactive. That is, global variables are now automatically reevaluated when dependent global variables or entities (from use getEntity()) change. They will also drive evaluation of rules when a rule-based expression or trigger condition references a global variable that changes.
• A new version of lexpjs supporting the above changes is being used; this version includes local scope for the interior of most statements, which may break some expressions (in particular, those that breach etiquette and create variables inside loops but access them outside).
• First version with localization capability. Users interested in localizing the user interface for their language/country can read the how-to documentation.

Expressions Changes

This release comes with some significant changes to the operation of global expressions in particular, and some expression tweaks throughout.

First, the expression language has been given the concept of scope; specifically, the statements each, first, do...done, if then else endif now have local scopes for their interior expressions. Variables created in the interior of these statements are not visible outside of these statements (they are implicitly local), unless the variable already exists in an enclosing scope. This is probably not going to cause anyone much heartburn on existing expressions with these statements. I don't see many of you pushing the limits in this area of the expression language that would make it an issue. We'll handle exceptions as they come up.

Second, and more importantly, Reactor now evaluates and treats global variables differently, and I think you'll agree this is a good change and brings more useful, even expected, behavior. The details:

1. Global variables are now reactive; for example, when a global variable that contains a getEntity() expression detects that the entity is modified, the global variable's expression is automatically recalculated. In previous versions of MSR, the variable was not re-evaluated until a rule requested the global variable's value.
2. Global variables are recalculated when they reference another global variable, and that global variable changes (for whatever reason). If, for example, you have a rule that periodically fetches JSON data from a web site and stores the response in a global variable, and other global variables that draw on that data, those dependent variables will be reevaluated when the variable holding the response is modified.
3. Rules will be reactive to global variable changes that do not involve entities. Previously, MSR rules would only re-evaluate on a change to a global variable if that variable involved a getEntity() expression. In fact, Reactor was not reacting to the variable change, it was reacting to the entity change. Now, Reactor truly reacts to the global variable change, which means it will also respond to the change in values of case #2 above -- if a rule is dependent on a global variable, any change to that global variable for any reason will cause the rule to re-evaluate, and based on its conditions, possibly change state.
4. You can disable the dependency and auto-recalculation of a global variable, because sometimes its useful to do that.

Given these changes, it is pretty easy to create reference loops. There's nothing stopping you from creating a variable A and compute its value from B, and then create B and have it set its value based on A. This will get you loop errors and a stop on evaluation; it won't be subtle. Similar abuses between global variables and rules and the overzealous use of Set Variable actions can also lead to rule loops throttling. Any time you use Set Variable in a rule-based reaction where that rule also uses the variable in triggers, you are running the risk of such a loop, so be careful.

Reactor for Vera doesn't have global variables, so at the time it was implemented as a new feature of MSR, simplifying assumptions were made. I hope you'll agree this is a useful evolution of their behavior.

Docs are updated, so make sure you check them out.

Localization

This is the first release that has the bulk of my recent work on localization. I'm barely passable at German, certainly not enough so to do a translation, and know enough Spanish to order a beer and get slapped by members of the opposite sex, but that's about it for me for languages (I used to teach Latin in an elementary school, but I doubt there's much demand for that translation). So while the system now recognizes the language configuration of your browser and host system automatically, it won't be able to present in any language other than US English because that's the only string file available. If any native speakers of other languages (including UK and Australian English) would like to step up and do a localization, let me know — that would be your work product to own and distribute as an add-on (e.g. through Github, etc.) to the community, and optionally include in Reactor distributions if you're willing (as a redistributed/third-party work with attribution on the About page). The US English file, which is the reference for all strings and starting point for translations, can be found in this Github repository: https://github.com/toggledbits/msr-localization

[toggledbits]

Build 21270

• Fix an issue with EzloController not processing device match expressions correctly on newer lexpjs.
• Add matchEntities() function (see docs)
• Further localization improvements and fixes.

[toggledbits]

Build 21275

NOTICE: This revision incorporates a breaking change to the recently-introduced matchEntities() expression function. See below.

• VeraController: Devices of type ...:FloodSensor1 now use the leak_sensor (specific) capability; they maintain their previous use of the generic binary_sensor capability for backward compatiblity with existing automations.
• EzloController: Fix a stack overflow that could occur under certain boundary conditions.
• OWMController: Fix a broken context initialization for lexpjs causing user-configured "extra data" fields with expressions to throw a soft error and report an incorrect (null) value.
• Improve interaction between Engine and UI when global expressions are created: they are now given an initial evaluation and their dependency tree built at that time. Previously, a rule reference or manual UI interaction (e.g. "Try Now" button press) was required to initialize it.
• Improve interaction between Engine and UI when global variables are deleted: the deleted variable is now removed from the Engine's evaluation dependency tree to avoid spurious "Expression Not Found" exceptions in the log for the then-deleted variable. These would stop after a restart, because the dependency tree was rebuilt at that time, but it should not take a restart to fix the tree.
• Fix an error in the parameters of the isRuleSet and isRuleEnabled expression functions that would cause a constructor error when global expressions using these functions were initialized at system startup, before the Engine had loaded the rules. This error was exposed by the changes made to support global variable dependency graphs.
• When deleting system objects, wait until all notifications have been given and returned, to assure that the object being deleted is not actually deleted before the notified object has a chance to do whatever it needs to do. The former behavior had no adverse effect; this was a change to gain new behavior necessary to support the global variable dependency tree update on delete mentioned above.
• Fix a bug in the recording of the timestamp of modification of a global variable; it was using real time rather than virtual time, so when the engine was configured for "Test Time", it was recording incorrect (i.e. non-test time) values.
• The check for circular dependencies in the evaluation of global variables was relaxed to allow self-references (e.g. series = push( series, value )).
• BREAKING: The matchEntities() function now returns only a list of canonical IDs for matched entities. You will need to use the getEntity() function to fetch current data for the entity if you need it. This was done because the previously-returned list of entity objects could grow "stale" (the data in the objects reflects the state at the time, not ever-changing and ongoing state as attributes change). And yes, this is exactly the kind of thing I said I try to avoid, but sometimes you just don't know all the side-effects until you see the thing in the wild. Sorry, everyone.
• New expression functions: fileExists(), fileRead(), fileWrite(). See the docs.
• Incorporates a hotfix made to 21270 for the SetVariable action on Vera hubs.

[toggledbits]

Build 21277

• HubitatController: Fix a problem with setting RGB color.
• Fix an issue causing "Object not found" exceptions when evaluating a getEntity() call in a subcontext of a rule-based expression (e.g. within an each loop).
• Fix a number of issues required for localization (thank you, @Crille ).

[toggledbits]

Build 21281

• In prior versions, a reference to a variable (rule or global) that does not exist would cause an alert and "fail soft" by returning null. This (old) behavior actually hides such errors and makes them more difficult to locate, and generally speaking, you shouldn't be referring to things that don't exist. The new behavior is to throw an exception during the evaluation, which will more clearly highlight the error and make it easier to find and correct. If for some reason you still need the old behavior, you can set deprecated_undefined_global_soft_fail to true in the engine section of reactor.yaml, and such errors will soft-fail as before. The old behavior is deprecated, however, and will be removed in a few months.
• PR 0000260: Fix failure of HTTP Request (really underlying variable handling) to notice change in array of objects returned by request.
• PR 0000259: Need to more clearly highlight variables with errors in the expression editor. Failing expressions now receive orange highlight bars.
• Add typeof() expression function.
• Additional localization changes.
• New HTTP API additions: add alert, dismiss alert
• Bless Home Assistant to 2021.10.0

[toggledbits]

Build 21286

• InfluxFeed now supports InfluxDB version 1.8+ and 2.0. InfluxDB versions prior to 1.8 are no longer supported. Please see the documentation for configuration details, as the configuration requirements for 1.x and 2.0 are different. Users of InfluxFeed also need to npm update --nosave if they are bare-metal installed.
• The behavior of rules when throttled has been improved. Note, however, that any time a rule is throttled, events could be missed (e.g. if a light that is a dependency of a rule is turned on and back off quickly when throttled, the rule may not "see" the on event).
• Ongoing documentation updates.
• Bless Hass to 2021.10.4
• First version to contain a foreign language translation, Swedish! Thank you @Crille for your work on the translation and helping me get localization working.
• Address possible evaluation loop while using SetVariable in a rule reaction on a rule variable, where the value is an expression that contains a getEntity() reference.
• Documentation updates
• Further i18n fixes (thank you @Crille)

[toggledbits]

Build 21292

• PR 0000262: Fix an error in handling date/time conditions with M/D across new year boundary.
• PR 0000261: HubitatController: Resolve an order of operations issue preventing extended attributes from being assigned as primary in local config.
• PR 0000258: (i18n) Fix string reference for condition option
• HassController: treat controller reporting NOT_RUNNING as an error and disconnect/retry
• Bless Hass to 2021.10.6

[toggledbits]

Build 21294

• PR 000263: Fix spurious log messages after deleting a global expression that contained a getEntity().
• Fixed a UI issue with an expression being sorted to the top of the list then ending up at the bottom.
• The expression function strftime() is now locale-aware.
• Docker containers are now running on nodejs version 16.

[toggledbits]

Build 21297

• PR 0000264: Fix an error in the rule editor on a particular value test used in several places that could cause the entered value of a condition option to not be saved (most notable on the "repeats within" field, but also found in two other locations).
• Update to lexpjs 21296 to get sort() function for arrays, and user-definable functions from expressions. Default sort in Reactor Engine for the implemention of expression sort() function is (host) locale-aware. Please refer to the documentation for details.

[toggledbits]

Build 21306

Note: this release includes changes to package dependencies. Bare-metal users please run npm update --no-save to resolve.

• HTTP Request action now has explicit control over HTTP auth, with type selection (none, Basic, Digest) and credential fields (username and password).
• PR 0000265: Correct a string that is not translated in the alerts status panel.
• i18n: Found an additional string that was not localizable.
• The HTTP Request action will now suppress all alerts for any error it encounters, not just HTTP errors, if the "no error alerts" option is checked.
• Update lexpjs to get new array set functions: arrayIntersection(), arrayUnion(), arrayDifference(), arrayExclusive() and arrayConcat().

[toggledbits]

Build 21307

• PR 0000266: An issue introduced at 21306 saves certain substitutions with the wrong data type.
• HubitatController: Resolve (hopefully) a race condition at startup.

[toggledbits]

Build 21313

• Bless Home Assistant to version 2021.11.2
• New system capability button to express state of a single button or contact closure.
• PR 0000268: Sun condition shows "not between" as "between" in rule detail card.
• Add /alert/:tag HTTP API endpoint (DELETE method); this deprecates /alert/:tag/dismiss.

[toggledbits]

Build 21321

• HassController: Additional updates for event handling (init support).
• HubitatController: Map additional capabilities.
• Internal support additionals for MQTTController

MQTTController - Build 21321

• Fix incorrect scope reference is tasmota_generic_relay payload.
• Additional helpful debug.

Upgrading MQTTController, which is a standalone/add-on component, is the same procedure as initial installation.

[toggledbits]

Build 21331

Please note that Hubitat has released version 2.30 of its firmware, but it has not yet been tested with HubitatController and is not yet supported.

• DynamicGroupController: new controller to build dynamic groups. Refer to the docs for configuration information.
• Logger: new recycle flag, when true, each startup rotates the log file so the new run starts in a fresh file (makes some debugging easier); the file mode of the logs created can now be set by setting the mode key.
• Bump recommended nodejs version to 16.13.0; versions 14 and 15 will continue to be supported through March 31, 2022; docker containers are built with recommended versions, and require no user action, but bare-metal users should check their node version (node -v) and if necessary make a plan to upgrade before 3/31.
• Update to lexpjs 31330; adds built-in median( array ) function; provides extended first...in syntax with (new) optional result expression; refer to docs.
• PR 0000270: [HubitatController] Apparently Hubitat needs action pacing like VeraController (that is, it can be overwhelmed if too many actions are sent at once). Pacing is now enabled with a default of 25ms delay between actions.
• PR 0000269: [UI] Rename Reaction dialog for global reactions prepopulates name from start of editor session rather than most recent name given if rename is used several times within one session.
• Bless Home Assistant to version 2021.11.5

MQTTController - Build 21331

• More carefully guard sensors (in templates) from payloads that don't contain the sensor data (i.e. avoid setting null value if an update is published with data missing).
• Additional templates for Tasmota and Shelly.
• Documentation updates and clarifications.

[toggledbits]

Build 21332

• DynamicGroupController: Fix missing initialization in preload for some selectors (blocking bug, no PR in Mantis).
• Documentation build fix so all documents receive correct left navigation.
• Condition operator is [not] EMPTY has been added to ease checking of group member arrays and other values for emptiness; see the docs Entity Attribute conditions for details on what "empty" means in this context.

[toggledbits]

Build 21336

If you have been with Reactor since before 21075 (March 15, 2021), your logging.yaml configuration may have a relative path in the name keys of the default streams (e.g. ./logs/reactor.log). These must be changed to just a simple filename (e.g. reactor.log). Reactor has been warning you at startup (in the log file) about this since 21075, if it applies to you (i.e. your logging config uses a relative path), but if you haven't seen it, you may not have changed it. The time to check it/change it is now, because the next version of Reactor that is released will not allow pathnames (relative or absolute) in this key, and will not run if it finds one.

• DynamicGroupController: map actions in known capabilities of selected objects, so that groups can perform actions on their members. For example, performing the power_switch.off action on a group will do so on all of its members that support that action.
• Logging: Fix support of log streams so that subsystems declaring their own log streams operate correctly; fix a number of long-standing annoyances in the configuration and handling (no changes to user configuration requirements except as noted above); improved "panic mode" (used when no configured stream is writable).
• Dashboard: improved layout for string_sensor capability objects.
• HubitatController: Bless firmware 2.3.0.113
• HubitatController: extend recent additions to capabilities (string_sensor) to the mode and HSM entities. Reduce reliance on the custom/hub-specific extension capability.
• HubitatController: New "type guessing", experimental, to try to reduce effort for this hub in particular in determining device types and primary attributes. This is a work in progress (but then, what isn't?).
• PR 0000271: Fix a missing localization in rule detail panel, operator in group titles (reported by @Crille).
• Improvements to rpi-install.sh script to handle upgrade of systemd script when upgrading nodejs.
• A number of internal fixes related to linting of the code.
• As always, a number of documentation updates.

[toggledbits]

Build 21337

• Document additional keys for logging in the dist-config template. There is also existing, but not well-organized, docs on the Logging page of the manual (which at the moment lives under Troubleshooting).
• HubitatController: ensure x_hubitat and x_hubitat_extra_attribute capabilites are declared and extended so their attributes can be assigned as primary.
• PR 0000272: Localization issue in input fields for date/time condition.
• PR 0000273: Pulse after sustained for delay is shortened by the delay.

[toggledbits]

Build 21338

This is a hotfix build for users of Hubitat only, and specifically for users of Hubitat Modes. If you do not use Hubitat Modes, or Hubitat at all, you do not need to upgrade to this version.

• HubitatController: Fix handling of mode query response (injection from additional changes in 21336).

[toggledbits]

Build 21342

• PR 0000275: Fix sun angle not updating in reactor_system>sun (Sun Information) entity.
• PR 0000274: Fix missing localization of condition operand labels in rule editor.
• HassController: More tightly map reported service_data field types into Reactor types (to improve UI experience on Hass-native actions).
• Support "object" data type in action data more universally, not just in controllers that need it.