Organizing/ structuring rule sets and rules

[toggledbits]

Build 21228 has been released. Docker images available from DockerHub as usual, and bare-metal packages here.

• Home Assistant up to version 2021.8.6 supported; the online version of the manual will now state the current supported versions;
• Fix an error in OWMWeatherController that could cause it to stop updating;
• Unify the approach to entity filtering on all hub interface classes (controllers); this works for device entities only; it may be extended to other entities later;
• Improve error detail in messages for EzloController during auth phase;
• Add isRuleSet() and isRuleEnabled() functions to expressions extensions;
• Implement set action for lock and passage capabilities (makes them more easily scriptable in some cases);
• Fix a place in the UI where 24-hour time was not being displayed.

[toggledbits]

Build 21236

• Prototypical support for Ezlo Atom and PlugHub in EzloController;
• Be less aggressive with module loading in app.js for more permissive first-time startup;
• Simplify reaction stop for child tasks;
• Numerous small bug fixes/gnats in UI;
• Documentation updates and corrections.

Note that Generation 1 Atoms are supported in this version, but only through the remote access API (so it's cloud-dependent). Gen2 and later Atoms/PlugHubs can run with local/LAN access (no cloud dependency).

[toggledbits]

Build 21237

• EzloController: Tone down the warnings about disabling certain ciphers for Atom; that's only applicable to local access (don't need warnings when remote access is used).
• PR 0000245: Fixed an issue with a hang caused by the attempt to actively stop a reaction that is blocked waiting for a child reaction (a reaction group) to finish.
• PR 0000252: Enable power_switch capability for siren category on eZLO hubs.

[toggledbits]

Build 21239

• Expression functions strftime() and format() added. See the updated documentation.
• Direct browser (via HTML attributes) to disable spellcheck on various fields where it's inappropriate (e.g. HTTP Request action URL, headers and data).
• Updated to latest lexpjs, which has itself been updated to (hopefully) support Unicode alphabetic characters in identifier names.
• Allow expressions to have Unicode characters in their names (parity with lexpjs).
• Additional preparation for localization in UI and engine. For example, non-US users should notice that lists sorted alphabetically now appear in a more natural order for their configured language, rather than "ASCII" (character code ordinal) sorting.

[toggledbits]

Build 21243

• HubitatController: Resolve exception thrown when attempting to set mode.
• HassController: Blessed Hass version 2021.8.8

[toggledbits]

Build 21256

• Fix issue with HTTP Request action post body causing exception;
• The maximum wait time for controllers to become available when Reactor starts is now configurable via the startup_wait key in the engine section of reactor.yaml. Users of the similarly-named startup_delay, which is a fixed delay, are encouraged to use startup_wait instead.
• The Rule detail display in the Rule Sets list was not correctly showing some non-primitive values (objects and arrays) correctly.
• HassController: Blessed Hass version 2021.9.6

[toggledbits]

Build 21258

• Warning and clear when changing operator on condition that would clear condition options.
• Fix Vera importer geotag handling.
• VeraController: add binary x_vera_geotag.in_region attribute.
• HassController: implement x_hass_system.call_service on system entities.

[toggledbits]

Build 21267

• Change the evaluation algorithm for global variables to make them reactive. That is, global variables are now automatically reevaluated when dependent global variables or entities (from use getEntity()) change. They will also drive evaluation of rules when a rule-based expression or trigger condition references a global variable that changes.
• A new version of lexpjs supporting the above changes is being used; this version includes local scope for the interior of most statements, which may break some expressions (in particular, those that breach etiquette and create variables inside loops but access them outside).
• First version with localization capability. Users interested in localizing the user interface for their language/country can read the how-to documentation.

Expressions Changes

This release comes with some significant changes to the operation of global expressions in particular, and some expression tweaks throughout.

First, the expression language has been given the concept of scope; specifically, the statements each, first, do...done, if then else endif now have local scopes for their interior expressions. Variables created in the interior of these statements are not visible outside of these statements (they are implicitly local), unless the variable already exists in an enclosing scope. This is probably not going to cause anyone much heartburn on existing expressions with these statements. I don't see many of you pushing the limits in this area of the expression language that would make it an issue. We'll handle exceptions as they come up.

Second, and more importantly, Reactor now evaluates and treats global variables differently, and I think you'll agree this is a good change and brings more useful, even expected, behavior. The details:

1. Global variables are now reactive; for example, when a global variable that contains a getEntity() expression detects that the entity is modified, the global variable's expression is automatically recalculated. In previous versions of MSR, the variable was not re-evaluated until a rule requested the global variable's value.
2. Global variables are recalculated when they reference another global variable, and that global variable changes (for whatever reason). If, for example, you have a rule that periodically fetches JSON data from a web site and stores the response in a global variable, and other global variables that draw on that data, those dependent variables will be reevaluated when the variable holding the response is modified.
3. Rules will be reactive to global variable changes that do not involve entities. Previously, MSR rules would only re-evaluate on a change to a global variable if that variable involved a getEntity() expression. In fact, Reactor was not reacting to the variable change, it was reacting to the entity change. Now, Reactor truly reacts to the global variable change, which means it will also respond to the change in values of case #2 above -- if a rule is dependent on a global variable, any change to that global variable for any reason will cause the rule to re-evaluate, and based on its conditions, possibly change state.
4. You can disable the dependency and auto-recalculation of a global variable, because sometimes its useful to do that.

Given these changes, it is pretty easy to create reference loops. There's nothing stopping you from creating a variable A and compute its value from B, and then create B and have it set its value based on A. This will get you loop errors and a stop on evaluation; it won't be subtle. Similar abuses between global variables and rules and the overzealous use of Set Variable actions can also lead to rule loops throttling. Any time you use Set Variable in a rule-based reaction where that rule also uses the variable in triggers, you are running the risk of such a loop, so be careful.

Reactor for Vera doesn't have global variables, so at the time it was implemented as a new feature of MSR, simplifying assumptions were made. I hope you'll agree this is a useful evolution of their behavior.

Docs are updated, so make sure you check them out.

Localization

This is the first release that has the bulk of my recent work on localization. I'm barely passable at German, certainly not enough so to do a translation, and know enough Spanish to order a beer and get slapped by members of the opposite sex, but that's about it for me for languages (I used to teach Latin in an elementary school, but I doubt there's much demand for that translation). So while the system now recognizes the language configuration of your browser and host system automatically, it won't be able to present in any language other than US English because that's the only string file available. If any native speakers of other languages (including UK and Australian English) would like to step up and do a localization, let me know — that would be your work product to own and distribute as an add-on (e.g. through Github, etc.) to the community, and optionally include in Reactor distributions if you're willing (as a redistributed/third-party work with attribution on the About page). The US English file, which is the reference for all strings and starting point for translations, can be found in this Github repository: https://github.com/toggledbits/msr-localization

[toggledbits]

Build 21270

• Fix an issue with EzloController not processing device match expressions correctly on newer lexpjs.
• Add matchEntities() function (see docs)
• Further localization improvements and fixes.

[toggledbits]

Build 21275

NOTICE: This revision incorporates a breaking change to the recently-introduced matchEntities() expression function. See below.

• VeraController: Devices of type ...:FloodSensor1 now use the leak_sensor (specific) capability; they maintain their previous use of the generic binary_sensor capability for backward compatiblity with existing automations.
• EzloController: Fix a stack overflow that could occur under certain boundary conditions.
• OWMController: Fix a broken context initialization for lexpjs causing user-configured "extra data" fields with expressions to throw a soft error and report an incorrect (null) value.
• Improve interaction between Engine and UI when global expressions are created: they are now given an initial evaluation and their dependency tree built at that time. Previously, a rule reference or manual UI interaction (e.g. "Try Now" button press) was required to initialize it.
• Improve interaction between Engine and UI when global variables are deleted: the deleted variable is now removed from the Engine's evaluation dependency tree to avoid spurious "Expression Not Found" exceptions in the log for the then-deleted variable. These would stop after a restart, because the dependency tree was rebuilt at that time, but it should not take a restart to fix the tree.
• Fix an error in the parameters of the isRuleSet and isRuleEnabled expression functions that would cause a constructor error when global expressions using these functions were initialized at system startup, before the Engine had loaded the rules. This error was exposed by the changes made to support global variable dependency graphs.
• When deleting system objects, wait until all notifications have been given and returned, to assure that the object being deleted is not actually deleted before the notified object has a chance to do whatever it needs to do. The former behavior had no adverse effect; this was a change to gain new behavior necessary to support the global variable dependency tree update on delete mentioned above.
• Fix a bug in the recording of the timestamp of modification of a global variable; it was using real time rather than virtual time, so when the engine was configured for "Test Time", it was recording incorrect (i.e. non-test time) values.
• The check for circular dependencies in the evaluation of global variables was relaxed to allow self-references (e.g. series = push( series, value )).
• BREAKING: The matchEntities() function now returns only a list of canonical IDs for matched entities. You will need to use the getEntity() function to fetch current data for the entity if you need it. This was done because the previously-returned list of entity objects could grow "stale" (the data in the objects reflects the state at the time, not ever-changing and ongoing state as attributes change). And yes, this is exactly the kind of thing I said I try to avoid, but sometimes you just don't know all the side-effects until you see the thing in the wild. Sorry, everyone.
• New expression functions: fileExists(), fileRead(), fileWrite(). See the docs.
• Incorporates a hotfix made to 21270 for the SetVariable action on Vera hubs.

[toggledbits]

Build 21277

• HubitatController: Fix a problem with setting RGB color.
• Fix an issue causing "Object not found" exceptions when evaluating a getEntity() call in a subcontext of a rule-based expression (e.g. within an each loop).
• Fix a number of issues required for localization (thank you, @Crille ).

[toggledbits]

Build 21281

• In prior versions, a reference to a variable (rule or global) that does not exist would cause an alert and "fail soft" by returning null. This (old) behavior actually hides such errors and makes them more difficult to locate, and generally speaking, you shouldn't be referring to things that don't exist. The new behavior is to throw an exception during the evaluation, which will more clearly highlight the error and make it easier to find and correct. If for some reason you still need the old behavior, you can set deprecated_undefined_global_soft_fail to true in the engine section of reactor.yaml, and such errors will soft-fail as before. The old behavior is deprecated, however, and will be removed in a few months.
• PR 0000260: Fix failure of HTTP Request (really underlying variable handling) to notice change in array of objects returned by request.
• PR 0000259: Need to more clearly highlight variables with errors in the expression editor. Failing expressions now receive orange highlight bars.
• Add typeof() expression function.
• Additional localization changes.
• New HTTP API additions: add alert, dismiss alert
• Bless Home Assistant to 2021.10.0

[toggledbits]

Build 21286

• InfluxFeed now supports InfluxDB version 1.8+ and 2.0. InfluxDB versions prior to 1.8 are no longer supported. Please see the documentation for configuration details, as the configuration requirements for 1.x and 2.0 are different. Users of InfluxFeed also need to npm update --nosave if they are bare-metal installed.
• The behavior of rules when throttled has been improved. Note, however, that any time a rule is throttled, events could be missed (e.g. if a light that is a dependency of a rule is turned on and back off quickly when throttled, the rule may not "see" the on event).
• Ongoing documentation updates.
• Bless Hass to 2021.10.4
• First version to contain a foreign language translation, Swedish! Thank you @Crille for your work on the translation and helping me get localization working.
• Address possible evaluation loop while using SetVariable in a rule reaction on a rule variable, where the value is an expression that contains a getEntity() reference.
• Documentation updates
• Further i18n fixes (thank you @Crille)

[toggledbits]

Build 21292

• PR 0000262: Fix an error in handling date/time conditions with M/D across new year boundary.
• PR 0000261: HubitatController: Resolve an order of operations issue preventing extended attributes from being assigned as primary in local config.
• PR 0000258: (i18n) Fix string reference for condition option
• HassController: treat controller reporting NOT_RUNNING as an error and disconnect/retry
• Bless Hass to 2021.10.6

[toggledbits]

Build 21294

• PR 000263: Fix spurious log messages after deleting a global expression that contained a getEntity().
• Fixed a UI issue with an expression being sorted to the top of the list then ending up at the bottom.
• The expression function strftime() is now locale-aware.
• Docker containers are now running on nodejs version 16.

[toggledbits]

Build 21297

• PR 0000264: Fix an error in the rule editor on a particular value test used in several places that could cause the entered value of a condition option to not be saved (most notable on the "repeats within" field, but also found in two other locations).
• Update to lexpjs 21296 to get sort() function for arrays, and user-definable functions from expressions. Default sort in Reactor Engine for the implemention of expression sort() function is (host) locale-aware. Please refer to the documentation for details.

[toggledbits]

Build 21306

Note: this release includes changes to package dependencies. Bare-metal users please run npm update --no-save to resolve.

• HTTP Request action now has explicit control over HTTP auth, with type selection (none, Basic, Digest) and credential fields (username and password).
• PR 0000265: Correct a string that is not translated in the alerts status panel.
• i18n: Found an additional string that was not localizable.
• The HTTP Request action will now suppress all alerts for any error it encounters, not just HTTP errors, if the "no error alerts" option is checked.
• Update lexpjs to get new array set functions: arrayIntersection(), arrayUnion(), arrayDifference(), arrayExclusive() and arrayConcat().

[toggledbits]

Build 21307

• PR 0000266: An issue introduced at 21306 saves certain substitutions with the wrong data type.
• HubitatController: Resolve (hopefully) a race condition at startup.

[toggledbits]

Build 21313

• Bless Home Assistant to version 2021.11.2
• New system capability button to express state of a single button or contact closure.
• PR 0000268: Sun condition shows "not between" as "between" in rule detail card.
• Add /alert/:tag HTTP API endpoint (DELETE method); this deprecates /alert/:tag/dismiss.

[toggledbits]

Build 21321

• HassController: Additional updates for event handling (init support).
• HubitatController: Map additional capabilities.
• Internal support additionals for MQTTController

MQTTController - Build 21321

• Fix incorrect scope reference is tasmota_generic_relay payload.
• Additional helpful debug.

Upgrading MQTTController, which is a standalone/add-on component, is the same procedure as initial installation.