Moving MSR from a QNAP container to RP 5 - some issues

[toggledbits]

Build 21072

• 0000108: Editing an extended attribute after having a system attribute selected applies validation rules of system attribute
• 0000107: Improve button consistency between rules and reactions
• 0000104: Error when attempting to recreate variable used in rule
• 0000095: Also fix in Reactions (reselect of action if device changes)
• 0000100: Make sl_CentralSceneAttr available for Vera

The x_vera_state capability is gone, and has been replaced with x_vera_svc_XXXX capabilities, wherein XXXX is the service ID on the Vera device. This slows startup, as additional queries must be made of every Vera device at initialization; the user_data query does not provide sufficient information. The Hubitat and Hass controllers also model this behavior. It's very tidy on Hubitat and Hass, I'm happy to say (and there is no performance penalty on these platforms).

The bad news (Vera users): If you are using x_vera_state attributes in your rules, you will need to edit them and choose the newer capability. Good news: this model will probably never change.

[toggledbits]

Build 21073:

• 0000110: Problem with formatting and reselection of Interval conditions in "relative to condition TRUE". Good time to mention that this mode is now DEPRECATED. This functionality was long ago replaced in Reactor for Vera by pulse output mode, but I didn't deprecate the feature in RFV at the time; should have. It is deprecated now in MSR.

Ongoing UI work, including exporting of group to new Rule, and importing of Rule to group.

[toggledbits]

Build 21074

• 0000110 - Interval condition not restoring related condition (option) correctly on edit
• 0000111 - Hass identified service not available (androidtv)
• 0000112 - Expression error
• 0000113 - Bridged Vera systems shows no devices (corrupt geofencing data)
• No PR - Fix precedence issue with lexpjs "each" and "first" statements

This version is also ready for testing under systemd on RPi (I've been running it that way all day). I will publish more on that later; I'm being called to the dinner table.

[toggledbits]

Build 21075:

• 0000114: Fix loss of condition options on device change
• 0000115: Add restart button to Tools page
• No PR: Update lexpjs to fix && and || return values.

Continuing work on UI cleanups are in the fore these days. Also work getting the HTTP API squared away sufficiently that I can port my automated test tool and start writing new test scripts.

[toggledbits]

Build 21076:

• 0000122 - Pushover notifications fail (dup/same cause as 119)
• 0000121 - "in" operator malfunction (Conditions)
• 0000120 - getEntity() fails to find entity when using name
• 0000119 - SMTP Notifications fail
• 0000106 - Include ID in entities list of expression builder
• 0000063 - Reorganize entity list in triggers and constraints

PRs 63 and 106 are addressed by a long-planned change finally implemented which I call the "Entity Picker". It operates similar to the Entities tab at the top leve of the UI, providing filters for source controller, group and capability. It also provides a name search that filters as you type. And, it remembers the last set of filters, so when you bring it up, your last search is intact. This has been deployed in all locations where entity selection was previously done with a drop-down menu.

[toggledbits]

Build 21077:

• 0000123: Rename of rule sometimes doesn't stick
• 0000124: Request action URL check doesn't allow variable substitution

Notification now supports (hopefully) CallMeBot. The dist-config contains an updated notifications.yaml with a section for CMB. It looks like this:

notification:
  # other method configurations not shown

  # CallMeBot
  CallMeBot:
    profiles:
      default:
        # api_url - Access URL for CallMeBot API being used
        api_url: http://api.callmebot.com/start.php
        #
        # api_key - (if needed) API key (use for Facebook API, WhatsApp API, etc.).
        #           If the API you are using does not require it, leave it
        #           commented out or blank.
        #api_key: "place api key here"
        #
        # source - (if needed) Source field for some APIs. If yours does not require
        #          it, leave it commented out or blank.
        #source: "source-value"
        #
        # phone - (if needed) Phone field for some APIs. If yours does not require
        #         it, leave it commented out or blank.
        #phone: "phone-number"
        #
        # user - (if needed) User field for some APIs. If yours does not require
        #        it, leave it commented out or blank.
        #user: "username"
        #
        # lang - (optional) Language/voice; default "en-US-Standard-C".
        #        See CallMeBot docs for possible values.
        #lang: "en-US-Standard-C"
        #
        # rpt - (optional) Number of times to repeat message; default: 1.
        #rpt: 1

If you run into an issue, logs will help here.

[toggledbits]

Build 21078:

• 0000122: Prowl notifications not working
• 0000125: Add name search to Entities list (like Entity Picker)
• 0000129: Exit after creating rule leaves empty rule
• 0000130: Improve "last value" display for expressions

[toggledbits]

Build 21080:

• 0000135: Fix animation when rule SETs
• 0000127: Overly verbose error messages on expression syntax errors

Additional work:

• Rule status detail now shows expression values.
• Sun info now available in a "Sun Information" entity managed by the system controller.
• "Solar noon" option added to "Sunrise/Sunset" conditions.

[toggledbits]

Build 21081:

• 0000139: Unconfigured notification methods causing incomplete UI display/missing actions

Additional:

• In the system location, one can now set name to the name of the desired location for default weather (sometimes OpenWeatherMap comes up with some odd place names). The city name will be used by default.
• Work on expression substitutions in conditions is feature-complete but not thoroughly tested yet. Note that substitutions are only permitted in Entity Attribute and Expression Value condition operands, and in about half of the condition options. The latter is very much by design, as the proper operation of the condition options requires a deterministic (stable, unchanging) value between evaluations for proper operation of the option in many cases (e.g. pulse timing and counts cannot be set by expression values).
• The evaluation context for rules is now more persistent, which allows a global expression that is referred to twice or more in a rule or other global expressions to be only evaluated once (previously, the global variable was evaluated on every reference, which could have side-effects for expressions used as counters, array-builders, etc.).

I've also closed/resolved a number of "wish list" PRs in Mantis. Rather than leaving these open until they are eventually dealt with, they are memorialized by the "suspended" resolution state. This makes it easy to search out the wish list (just search on resolution == suspended). These PRs can be re-opened when the time comes for work. Also, I'd like to see things bantered about in the community forums before they turn into wish-list PRs, if possible.

[toggledbits]

Build 21082:

• 0000140: Round-trip move of expression between rule and global and back doesn't manage state correctly

Work proceeding apace on the new alerts/trouble notification system.

[toggledbits]

Build 21084:

• 0000147 Issues (multiple) handling headers for request actions
• 0000146 Additional native capabilities for Hubitat (plus fixes to hs_color)

IMPORTANT: Please re-edit your Request action headers (if any) to get them into the new form, and correct at least one of the bugs related to the UI's contribution to the PR.

Additional:

• Timing animations are now removed from disabled rules.
• Expression value changes in the Rule status detail card are now properly animated.
• A data corruption when dragging conditions between groups has been found and fixed.
• Improved error handling in the Request action, including the storage function (could throw an unhandled promise rejection when the target variable was in a disabled rule).
• The display of "Last Updated" on the Entities list was stalling; this was reported by somebody some time ago but I couldn't find the PR for it, forgot about it, and then noticed it again earlier today.
• Improve error handling around the loading of Notifier subclasses, and modify the API slightly to move some additional repetitious implementation up to the base class.

A couple more notes on Request action headers: the RFC (2616, Section 2.4 specifically) says that header (field) names are not case-sensitive. In my experience, this is true for 99% of servers you run into. And Murphy is on board (i.e. the server you want to use is guaranteed to be in the 1% remaining). MSR makes no assumption about the case of the header/field name. However, the underlying node-fetch implementation does, effectively converting all headers to lower case. This may cause problems on some servers, but it's the server that is not RFC-compliant, not node-fetch or MSR, so you are likely not to see a fix for this if you run into a server where it's a problem (I'm guessing it will be the hackish HTTP stack on some cheap WiFi-based microcontroller that rears its ugly head here).

The other thing is that the RFC allows a header to be specified multiple times, the net effect being a comma-separated list of header/field values in order of appearance. That is, specifying an "Accepts" header twice, once with "application/json" and another with "text/xml" would be handled (in compliance with the RFC) as "Accepts: application/json, text/xml". Unfortunately, node-fetch uses a JavaScript Map object for its underlying implementation of headers, and that only permits one value per key -- you can't have multiple "Accepts" headers, each subsequent will simply replace any prior. Probably a rare circumstance, but nonetheless, I have the MSR Request action storing the headers you provide in its own format, not the implementation form, and MSR can handle repeated header/field names; if you use this feature, MSR will simply do the work that the server would normally do to concatenate the same-named fields on its end, before handing the headers to node-fetch.

So, while I can't work around node-fetch's corruption/overrule of the header case, I can provide a workaround for the unfortunate choice of data structure in node-fetch's header implementation. Onward.

[toggledbits]

Build 21085:

• 0000149 HTTP Request action now has an additional checkbox to allow SSL connections to endpoints with invalid/self-signed certificates.
• 0000148 Inconsistency in EACH expression results

Tomorrow (Saturday 3/27) is an house/family IT day for me, so barring any serious problems that need an urgent fix, there will be no build (21086) tomorrow.

[toggledbits]

Build 21087:

• 0000151 "Sustained for" condition restarts count early
• 0000152 Constraints may case reset reaction to run unexpectedly

[toggledbits]

Build 21089:

• 0000153 Problem with magnitude comparisons when one operand was a string not containing a number

The SystemController's reactorsystem>system entity now reports disk space. The default is to report for the volume(s) on which Reactor is installed, but it's configurable. Refer to the docs. Work is continuing on alerts and status.

You will need to run npm update in the reactor directory after unpacking the archive! (This does not apply to Docker users; your updates are included in the image.)

[toggledbits]

Build 21090:

• 0000155 Issue with second (and beyond) Vera controllers not getting x_vera_sys capability extended.
• 0000154 Improve dynamic updates of values in Rule editor, and extend to Expression and Rule conditions.

This version also previews the Status display still under development, and the Alerts system (which currently only provides alerts for controller online/offline and rule triggers and constraints; there are no alerts generated for problems with reactions or other subsystems yet). Since this is feature preview, discussion here is fine, but please don't open PRs for the Status page just yet.

[toggledbits]

Build 21091:

• 0000157 POST data not saved when editing reaction
• 0000156 Pulse mode output break has no countdown in rule status UI

Additional alerts added on the reaction side. Additional methods added to HTTP Request action. Minor UI tweaks and fixes ongoing.
Add Note

[toggledbits]

Build 21092:

No PRs addressed today; several remain open with fixes in 21091 or earlier, waiting for OPs to verify.

Ongoing work on status and chasing reports that didn't end up as PRs today. I think everything under discussion is understood, if not yet completely resolved. A new notifier type "Alert" has been added, so you can add your own alerts to the system Status from your reactions (see below). Work ongoing getting more alerts where needed.

---

To configure the "Alert" notification type, see the "Alert" configuration addition (one line) in dist-config/notification.yaml or just add this line (the notification type currently has no configuration options):

notification:
  Alert: {}           <--- add this line

  # SMTP (Simple Mail Transfer Protocol, for sending email)
  SMTP:
    # et cetera

[toggledbits]

Build 21093:

No PRs.

This release primarily addresses a failure on Hass caused by a breaking change in their WebSocket API between versions 2020.12.13 and 2021.3.4.

[toggledbits]

Build 21095:

No PRs.

But... this build addresses an issue on Hubitat that created a problem setting modes, updates lexpjs with some minor fixes. I was going to release a fix for 0000133 (need to save rule after creating expression to be able to use expression in condition or action) as well, but the changes snowballed into a bunch of other code cleanups and tweaks and I don't feel good about releasing them yet.

Also, with respect to the "Reset Latched" action (which is not implemented in MSR as an explicit action)... I'm opening that for debate, but if there are no compelling arguments for, I'm going to close the two related PRs (39 and 160) and move on.

[toggledbits]

Build 21096:

• 0000133 Must save rule to get newly-created rule expression to appear in "Expression Value" condition or "Set Variable" action pickers.

This version also has the first attempt at reaction status. You'd better have good eyes; most reactions that don't have delays or HTTP requests will finish in a few milliseconds, so they appear and disappear in a flash. The house mode entity on Vera now has a primary value.

EDIT: @LibraSun reminded me of something I forgot to mention. The rules editor (triggers and conditions) was not correctly storing the rule ID on Expression Value conditions and Set Variable expressions. If you had a global variable with the same name as a rule-based variable, and you had selected the global, a later edit would incorrectly load the selector with the rule-based name, rather than the global name. This is now fixed, but it has the effect of breaking existing conditions on local variables. You can fix this just by selecting the correct variable again and saving the rule, and it will be fixed forever more. As part of this fix, you can now also select variables in any rule (in addition to globals), where previously, you could only select those variables that were local to the rule you were editing. This should make some expression and rule forms much easier to write and maintain.