Moving MSR from a QNAP container to RP 5 - some issues

[toggledbits]

Build 21077:

• 0000123: Rename of rule sometimes doesn't stick
• 0000124: Request action URL check doesn't allow variable substitution

Notification now supports (hopefully) CallMeBot. The dist-config contains an updated notifications.yaml with a section for CMB. It looks like this:

notification:
  # other method configurations not shown

  # CallMeBot
  CallMeBot:
    profiles:
      default:
        # api_url - Access URL for CallMeBot API being used
        api_url: http://api.callmebot.com/start.php
        #
        # api_key - (if needed) API key (use for Facebook API, WhatsApp API, etc.).
        #           If the API you are using does not require it, leave it
        #           commented out or blank.
        #api_key: "place api key here"
        #
        # source - (if needed) Source field for some APIs. If yours does not require
        #          it, leave it commented out or blank.
        #source: "source-value"
        #
        # phone - (if needed) Phone field for some APIs. If yours does not require
        #         it, leave it commented out or blank.
        #phone: "phone-number"
        #
        # user - (if needed) User field for some APIs. If yours does not require
        #        it, leave it commented out or blank.
        #user: "username"
        #
        # lang - (optional) Language/voice; default "en-US-Standard-C".
        #        See CallMeBot docs for possible values.
        #lang: "en-US-Standard-C"
        #
        # rpt - (optional) Number of times to repeat message; default: 1.
        #rpt: 1

If you run into an issue, logs will help here.

[toggledbits]

Build 21078:

• 0000122: Prowl notifications not working
• 0000125: Add name search to Entities list (like Entity Picker)
• 0000129: Exit after creating rule leaves empty rule
• 0000130: Improve "last value" display for expressions

[toggledbits]

Build 21080:

• 0000135: Fix animation when rule SETs
• 0000127: Overly verbose error messages on expression syntax errors

Additional work:

• Rule status detail now shows expression values.
• Sun info now available in a "Sun Information" entity managed by the system controller.
• "Solar noon" option added to "Sunrise/Sunset" conditions.

[toggledbits]

Build 21081:

• 0000139: Unconfigured notification methods causing incomplete UI display/missing actions

Additional:

• In the system location, one can now set name to the name of the desired location for default weather (sometimes OpenWeatherMap comes up with some odd place names). The city name will be used by default.
• Work on expression substitutions in conditions is feature-complete but not thoroughly tested yet. Note that substitutions are only permitted in Entity Attribute and Expression Value condition operands, and in about half of the condition options. The latter is very much by design, as the proper operation of the condition options requires a deterministic (stable, unchanging) value between evaluations for proper operation of the option in many cases (e.g. pulse timing and counts cannot be set by expression values).
• The evaluation context for rules is now more persistent, which allows a global expression that is referred to twice or more in a rule or other global expressions to be only evaluated once (previously, the global variable was evaluated on every reference, which could have side-effects for expressions used as counters, array-builders, etc.).

I've also closed/resolved a number of "wish list" PRs in Mantis. Rather than leaving these open until they are eventually dealt with, they are memorialized by the "suspended" resolution state. This makes it easy to search out the wish list (just search on resolution == suspended). These PRs can be re-opened when the time comes for work. Also, I'd like to see things bantered about in the community forums before they turn into wish-list PRs, if possible.

[toggledbits]

Build 21082:

• 0000140: Round-trip move of expression between rule and global and back doesn't manage state correctly

Work proceeding apace on the new alerts/trouble notification system.

[toggledbits]

Build 21084:

• 0000147 Issues (multiple) handling headers for request actions
• 0000146 Additional native capabilities for Hubitat (plus fixes to hs_color)

IMPORTANT: Please re-edit your Request action headers (if any) to get them into the new form, and correct at least one of the bugs related to the UI's contribution to the PR.

Additional:

• Timing animations are now removed from disabled rules.
• Expression value changes in the Rule status detail card are now properly animated.
• A data corruption when dragging conditions between groups has been found and fixed.
• Improved error handling in the Request action, including the storage function (could throw an unhandled promise rejection when the target variable was in a disabled rule).
• The display of "Last Updated" on the Entities list was stalling; this was reported by somebody some time ago but I couldn't find the PR for it, forgot about it, and then noticed it again earlier today.
• Improve error handling around the loading of Notifier subclasses, and modify the API slightly to move some additional repetitious implementation up to the base class.

A couple more notes on Request action headers: the RFC (2616, Section 2.4 specifically) says that header (field) names are not case-sensitive. In my experience, this is true for 99% of servers you run into. And Murphy is on board (i.e. the server you want to use is guaranteed to be in the 1% remaining). MSR makes no assumption about the case of the header/field name. However, the underlying node-fetch implementation does, effectively converting all headers to lower case. This may cause problems on some servers, but it's the server that is not RFC-compliant, not node-fetch or MSR, so you are likely not to see a fix for this if you run into a server where it's a problem (I'm guessing it will be the hackish HTTP stack on some cheap WiFi-based microcontroller that rears its ugly head here).

The other thing is that the RFC allows a header to be specified multiple times, the net effect being a comma-separated list of header/field values in order of appearance. That is, specifying an "Accepts" header twice, once with "application/json" and another with "text/xml" would be handled (in compliance with the RFC) as "Accepts: application/json, text/xml". Unfortunately, node-fetch uses a JavaScript Map object for its underlying implementation of headers, and that only permits one value per key -- you can't have multiple "Accepts" headers, each subsequent will simply replace any prior. Probably a rare circumstance, but nonetheless, I have the MSR Request action storing the headers you provide in its own format, not the implementation form, and MSR can handle repeated header/field names; if you use this feature, MSR will simply do the work that the server would normally do to concatenate the same-named fields on its end, before handing the headers to node-fetch.

So, while I can't work around node-fetch's corruption/overrule of the header case, I can provide a workaround for the unfortunate choice of data structure in node-fetch's header implementation. Onward.

[toggledbits]

Build 21085:

• 0000149 HTTP Request action now has an additional checkbox to allow SSL connections to endpoints with invalid/self-signed certificates.
• 0000148 Inconsistency in EACH expression results

Tomorrow (Saturday 3/27) is an house/family IT day for me, so barring any serious problems that need an urgent fix, there will be no build (21086) tomorrow.

[toggledbits]

Build 21087:

• 0000151 "Sustained for" condition restarts count early
• 0000152 Constraints may case reset reaction to run unexpectedly

[toggledbits]

Build 21089:

• 0000153 Problem with magnitude comparisons when one operand was a string not containing a number

The SystemController's reactorsystem>system entity now reports disk space. The default is to report for the volume(s) on which Reactor is installed, but it's configurable. Refer to the docs. Work is continuing on alerts and status.

You will need to run npm update in the reactor directory after unpacking the archive! (This does not apply to Docker users; your updates are included in the image.)

[toggledbits]

Build 21090:

• 0000155 Issue with second (and beyond) Vera controllers not getting x_vera_sys capability extended.
• 0000154 Improve dynamic updates of values in Rule editor, and extend to Expression and Rule conditions.

This version also previews the Status display still under development, and the Alerts system (which currently only provides alerts for controller online/offline and rule triggers and constraints; there are no alerts generated for problems with reactions or other subsystems yet). Since this is feature preview, discussion here is fine, but please don't open PRs for the Status page just yet.

[toggledbits]

Build 21091:

• 0000157 POST data not saved when editing reaction
• 0000156 Pulse mode output break has no countdown in rule status UI

Additional alerts added on the reaction side. Additional methods added to HTTP Request action. Minor UI tweaks and fixes ongoing.
Add Note

[toggledbits]

Build 21092:

No PRs addressed today; several remain open with fixes in 21091 or earlier, waiting for OPs to verify.

Ongoing work on status and chasing reports that didn't end up as PRs today. I think everything under discussion is understood, if not yet completely resolved. A new notifier type "Alert" has been added, so you can add your own alerts to the system Status from your reactions (see below). Work ongoing getting more alerts where needed.

---

To configure the "Alert" notification type, see the "Alert" configuration addition (one line) in dist-config/notification.yaml or just add this line (the notification type currently has no configuration options):

notification:
  Alert: {}           <--- add this line

  # SMTP (Simple Mail Transfer Protocol, for sending email)
  SMTP:
    # et cetera

[toggledbits]

Build 21093:

No PRs.

This release primarily addresses a failure on Hass caused by a breaking change in their WebSocket API between versions 2020.12.13 and 2021.3.4.

[toggledbits]

Build 21095:

No PRs.

But... this build addresses an issue on Hubitat that created a problem setting modes, updates lexpjs with some minor fixes. I was going to release a fix for 0000133 (need to save rule after creating expression to be able to use expression in condition or action) as well, but the changes snowballed into a bunch of other code cleanups and tweaks and I don't feel good about releasing them yet.

Also, with respect to the "Reset Latched" action (which is not implemented in MSR as an explicit action)... I'm opening that for debate, but if there are no compelling arguments for, I'm going to close the two related PRs (39 and 160) and move on.

[toggledbits]

Build 21096:

• 0000133 Must save rule to get newly-created rule expression to appear in "Expression Value" condition or "Set Variable" action pickers.

This version also has the first attempt at reaction status. You'd better have good eyes; most reactions that don't have delays or HTTP requests will finish in a few milliseconds, so they appear and disappear in a flash. The house mode entity on Vera now has a primary value.

EDIT: @LibraSun reminded me of something I forgot to mention. The rules editor (triggers and conditions) was not correctly storing the rule ID on Expression Value conditions and Set Variable expressions. If you had a global variable with the same name as a rule-based variable, and you had selected the global, a later edit would incorrectly load the selector with the rule-based name, rather than the global name. This is now fixed, but it has the effect of breaking existing conditions on local variables. You can fix this just by selecting the correct variable again and saving the rule, and it will be fixed forever more. As part of this fix, you can now also select variables in any rule (in addition to globals), where previously, you could only select those variables that were local to the rule you were editing. This should make some expression and rule forms much easier to write and maintain.

[toggledbits]

Build 21101:

• 0000164: Rules watching variables in other Rules not triggering

A number of UI cleanups and fixes, as well as some internal cleanups. Lots of time on longer-term items, so the pace of new builds will be reduced to only when PR fixes or new features are ready for release (still not more frequently than daily, but also not less than weekly, going forward).

[toggledbits]

Build 21104

• 0000166 Hubitat: extra state for Tibber energy (example) not available
• 0000167 Interval with days not editable
• 0000168 Exiting edit of global reaction jumps to status display

Home Assistant is supported up to new version 2021.4.4. The "period" attribute for the Sun Information entity has been added ("day" or "night"; primary attribute). The expressions now support urlencode() and urldecode() functions. A UI error thrown when attempting to add a ruleset has been fixed (no PR). Ongoing code and documentation work.

[toggledbits]

Build 21105:

• 0000174 Reaction that contains only comment should stop counter-reaction
• 0000173 Importer crashes on trange condition "between" op
• 0000170 Exception when running Hubitat action (native, it turns out)
• 0000165 Provide support for Hubitat custom actions from drivers

NOTE: Just discovered an issue. I'm going to respin this build. Stay tuned. If you haven't upgraded yet, don't. False alarm! All OK. Upgrade when ready!

[toggledbits]

Build 21106:

• 0000177 Hubitat modes broken in 21105

Fixed an issue with (non-)presentation of the entity picker when clicked; fixed the bell click (broken by 0000168 fix); removed unused packages from configuration.

PLEASE RUN npm update --nosave AFTER UPDATING.

[toggledbits]

Build 21117

This build represents a significant update from previous builds. Principally, this build provides action groups in reactions, and action groups support constraints. So it now possible for the SET or RESET reactions in your rules to have conditionally-executed actions. This feature applies to rule-based reactions only; it is not a feature of global reactions (and for the removal of doubt, will not be in future). In addition, the following PRs are addressed:

• 0000186 Add quick-selects for weekends and weekdays to Week Day condition type.
• 0000184 Pushover now supports "device" parameter correctly.
• 0000180 Using left nav when active editor has no changes pending is now allowed.
• 0000179 Catch-all for a number of lexpjs (expression) fixes and enhancements.

This version also incorporates a lot of UI enhancements and cleanups; still much to do, though.

Also new, there are now three "official" Docker image builds available on DockerHub:

• toggledbits/reactor:latest-generic-amd64 -- Generic Intel/AMD Linux
• toggledbits/reactor:latest-synology-amd64 -- Synology-specific (see more below)
• toggledbuts/reactor:latest-raspbian-armv7l -- Raspian Buster (Raspberry Pi)

These Docker images can be downloaded using the usual docker pull <imagename> syntax.

For all images, when creating the container you must map /var/reactor to a local directory in which your Reactor configuration and storage files are located. You should also bind /etc/localtime to the same-path file on your local system to get the correct timezone in the container. If this is not possible or does not correctly set the timezone for your container, the TZ environment variable should be set.

Typical to run:

# For generic:
docker run -d -p 8111:8111 -v /my/reactor/data:/var/reactor --mount type=bind,src=/etc/localtime,target=/etc/localtime toggledbits/reactor:latest-generic-amd64

# For Raspian:
docker run -d -p 8111:8111 -v /my/reactor/data:/var/reactor -e TZ=America/New_York toggledbits/reactor:latest-raspbian-armv7l

---

SPECIAL INSTRUCTIONS FOR SYNOLOGY DOCKER IMAGE UPGRADE (ONLY):

I will no longer be making the manual-download image available for Synology. Upgrading from the previous manual-download image is not a straight line, but can be done pretty quickly and without losing your configuration and other data. It's basically a redo of the original install. The steps are as follows:

1. Stop the existing container (under Containers in the Synology Docker app).
2. Make a note of where your current config/storage data is stored on the NAS. You can find this by clicking on the container row, then clicking the "Details" button. Click the "Volume" tab on the details dialog, and note the path associated with /var/reactor. Copy-paste this to a safe location or write it down. Close the dialog.
3. Click on the container row and choose "Delete" from the Actions dropdown menu.
4. Go to the Image tab and delete the toggledbits/reactor:latest image.
5. Go to the Repository tab and search for "toggledbits". The toggledbits/reactor entry should be listed in the results.
6. Click the entry row and then click "Download".
7. Once the image has downloaded, select the image in the Image tab and click "Launch".
8. Leave the defaults on the "General Settings" dialog, and click the "Advanced Settings" button.
9. On the Advanced Settings dialog, click "Enable auto-restart".
10. Click the "Volume" tab, and then "Add Folder". Select the directory you saved in step 2 above (where your data and config files are stored); for the mount path, enter exactly /var/reactor.
11. Click the "Port Settings" tab, and change the "Auto" to 8111 under "Local Port".
12. Click the "Environment" tab, and then "+" to add an environment variable. Name it "TZ" (caps), and then set the value to the name of your local time zone. Typically these are "Area/Locale", like "America/New_York". A full list is here: https://en.wikipedia.org/wiki/List_of_tz_database_time_zones
13. Hit the Apply button and it will put you back on the General Settings dialog. Hit "Next" and then "Apply" (the checkbox to start the container should be checked already), and the container should start.
14. Access the container via your NAS' IP address on port 8111.

---

USING DOCKER-COMPOSE TO RUN CONTAINERS

Here's a sample docker-compose file. You'll need to modify it to have the correct image tag and directory where your data are stored.

# Multi-System Reactor template docker-compose.yml
version: '3'

services:
  web:
    container_name: reactor
    environment:
      REACTOR_DATA_PREFIX: /var/reactor

    # Change the image below to the one you are using, if necessary.
    image: toggledbits/reactor:latest-generic-amd64
    restart: always
    network_mode: "bridge"
    expose:
      - 8111
    ports:
      - 8111:8111
    # Modify the first entry below (only before the colon; do not modify the rest) 
    # to the path where you want config/data stored. Make sure the directory
    # exists before starting the container.
    volumes:
      - /my/reactor/data:/var/reactor
      - type: bind
        source: /etc/localtime
        target: /etc/localtime
        read_only: true
    tmpfs: /tmp
    logging:
      driver: "json-file"
      options:
        max-file: 5
        max-size: 2m