[Reactor] Variables not updating correctly in latest-25201-2aa18550

[toggledbits]

It's probably moot at this point... I've just spent the last two weeks completely revamping the evaluation of Rule-based variables, to make them perform more like global variables, respect dependencies between them better, etc. The last week has just been a deep dive through the logs watching the performance of my own house on the updated approach.

I'll shortly be releasing a blind build with these changes, just for you (@tunnus) and @therealdb to try. I expect this to be disruptive, because any approaches in the logic that rely on specific behaviors and side-effects of the old approach will now work differently, maybe fail, and adapting to the new approach will be necessary. But, I think the new approach will produce a result that is much more in keeping with how you would expect it to work. So, sit tight... I'll be ready to build later today or early tomorrow.

[toggledbits]

OK. Interim build latest-25248 for 64-bit docker systems only is up for testing/trials.

NOTE TO ALL USERS NOT CONTRIBUTING TO THIS THREAD — this is not an official build and you are advised not to upgrade to it just yet. This build is specifically for participants in this thread as we work to improve the behavior of Rule-based variables. Wait for a build advertised in the official announcement thread before upgrading your system.

Behavior of Rule-based variables now:

• Will still be re-evaluated when the Rule is started, and any time the Rule is subject to trigger condition check (i.e. behavior not changed);
• When the trigger condition check occurs, Rule-based variables are always evaluated/updated before trigger conditions (i.e. also same as before);
• Rule-based variables may be modified by Set Variable action and assignment statements in Script Actions in Rule Reactions (unreliable before);
• If a Rule-based variable references another Rule-based variable, and the latter changes, the former will be re-evaluated (now same as Global Variables).
• The order of Rule-based variables doesn't matter, it's just for your convenience. Dependencies between variables are established at startup, and evaluation takes place in dependency tree order, not the order shown in the UI (now same as Global Variables).

If you run into an issue you can't resolve, here's what I ask:

1. Make sure your expectations are aligned with the changes above; make sure you are not relying on any prior behavior or side-effect of the prior behavior.

2. Enable logging for the rule in question. Put a snippet like below in your logging.yaml file (replace the part after # with your rule ID):

    "Rule#rule-l3hj06aa":
      level: 5
      streams:
      - type: file  # filename is derived from rule ID
        keep: 2
        level: 999
        recycle: true
   

3. Restart Reactor and get the rule in question to demonstrate whatever the issue is. Make sure to then capture the log file called Logger#Rule#XXX.log... I will be asking for it.

4. Report the issue here on this thread, with all of the usual attention to posting requirements: good, detailed description; expectations of behavior; clear description of what's not meeting expectations; etc.

5. I will PM you a URL where you can upload the log file and your rule JSON file (from storage/rules) if necessary.

The locally-installed version of the documentation is also updated.

[gwp1]

@toggledbits whoops... guess this is one time having the system auto-update containers with new images is bad....

So I'm on the test version. Found this:

This is a Dynamic Group I'd created forever ago:

My ruleset:

I added the snippet as noted into my logging.yaml file:

When it didn't work I checked my yaml and found a spacing issue. That's been corrected, I now have a proper logger* log. Will upload to our pre-x Dropbox in folder 25248 logs along with the rule JSON file.

[toggledbits]

Is there a global variable called dead (it's not shown above)?

Or are you looking for the word dead in status_text? Because if it's that, your expression is incorrectly formed.

[gwp1]

@toggledbits kicker is: this has been working for a long while. Dead iblind nodes (far too common, btw) get caught and pinged and I get the alerts on my phone.

I don't see a Global Expression called dead and, yes, I believe when I wrote this I was looking for either a status of 3 or status_text of dead.

[toggledbits]

OK. Yes, the new build also has some new error reporting. That message was probably being logged, but since you weren't looking at the logs, you didn't see it.

You can also combine those into one status test, which will be a good bit more efficient:

- id: groups
  enabled: true
  implementation: DynamicGroupController
  name: Dynamic Group Controller
  config:
    groups:
      zwavejs_dead:
        select:
          - include_group: zwavejs
        filter_expression: <
          entity?.attributes?.zwave_device?.status === 3 ||
          entity?.attributes?.zwave_device?.status_text === "dead"

Also, please... Please... PLEASE do not screen shot pure text. Use the fenced code block formatting here for text like expressions, configs, logs, etc. It may be expeditious for you to just paste the screen shot, but when I try to give you suggestions or corrections like the above, I have to retype the entire block (I can't copy-paste the text and update it). That not only takes a bunch of time, but also introduces the possibility that I will make a typo in the part we don't care about and leave you with a non-working example we then need to fix in yet another series of post replies.

Don't forget to remove the Rule logging when it isn't needed.

[gwp1]

@toggledbits now I'm feeling hypocritical as I'm the guy at work chastising the Support folks for giving me screenshots for the exact reasons you noted.

Change deployed.

Got this upon restart of MSR:

    at _ClientAPI._commandTimeout (http://192.168.1.100:8111/client/ClientAPI.js:594:125)

Cleared it and refreshed the tab and all seems fine. Removing logging snippet.

Thank you, sir!

[therealdb]

@toggledbits I've updated my main system. I'll observe and report back. Thanks

[therealdb]

Ok, I got a bunch of errors at restart because of failed references in almost all variables - now it seems to be OK and variables are updated. I'm observing a couple of rules I haven't migrate yet - and, at this point, I'll not migrate - where I used to old trick that @gwp1 surfaced (and that I used without making it a rule). always seems to be OK.

[therealdb]

yep, it seems to be related (in general) to variables not being evaluated at startup. Here's a screenshot from a daily rule that it's setting a couple of things, including a virtual binary sensor for Christmas from a local variable:

After re-setting the rule, the expression got re-evaluated and the error went away. @toggledbits

EDIT. a similar one + logs

This one, instead, every time is evaluated gives the error.

[latest-25248]2025-09-11T07:19:37.602Z <Rule:ERR> Lights-Christmas-Off (rule-kwv6wdpw in Lights-Outdoor): error evaluating expression specialdays: [Error] Object does not exist [-]
[latest-25248]2025-09-11T07:19:37.603Z <Rule:ERR> Full expression: t = time(), m = dateparts(t).month, d = dateparts(t).day,
(m == 12 && d == 24) ||
(m == 12 && d == 25) ||
(m == 12 && d == 31) ||
(m == 1 && d == 1) ||
(m == 1 && d == 5) ||
(m == 1 && d == 6)
[latest-25248]2025-09-11T07:19:37.604Z <Rule:CRIT> Error: Object does not exist [-]
Error: Object does not exist
    at GlobalExpression.getInstance (/opt/reactor/server/lib/GlobalExpression.js:160:27)
    at _0x1ec9e7._func.<computed> [as _ref] (/opt/reactor/server/lib/Rule.js:848:69)
    at _resolve (/opt/reactor/common/lexp.js:1385:27)
    at _run (/opt/reactor/common/lexp.js:1429:31)
    at /opt/reactor/common/lexp.js:1697:33
    at Array.forEach (<anonymous>)
    at _run (/opt/reactor/common/lexp.js:1696:28)
    at _run (/opt/reactor/common/lexp.js:1604:33)
    at _run (/opt/reactor/common/lexp.js:1438:34)
    at /opt/reactor/common/lexp.js:1425:29
[latest-25248]2025-09-11T07:19:37.627Z <Rule:INFO> Lights-Christmas-Off (rule-kwv6wdpw in Lights-Outdoor) evaluated; rule state transition from NULL to RESET!
[latest-25248]2025-09-11T07:19:37.638Z <Rule:INFO> Lights-Christmas-Off (rule-kwv6wdpw in Lights-Outdoor) started

[toggledbits]

New build latest-25254 posted. This should address the local variable in rule-based expressions at startup issue. Docker 64-bit platforms only.

[toggledbits]

@therealdb said in [Reactor] Variables not updating correctly in latest-25201-2aa18550:

[latest-25248]2025-09-11T07:19:37.603Z Rule:ERR Full expression: t = time(), m = dateparts(t).month, d = dateparts(t).day,
(m == 12 && d == 24) ||
(m == 12 && d == 25) ||
(m == 12 && d == 31) ||
(m == 1 && d == 1) ||
(m == 1 && d == 5) ||
(m == 1 && d == 6)

Just a tip, a more efficient way to write this:

parts = dateparts(), m=parts.month, d=parts.day,
    (m == 12 && d == 24) ||
    (m == 12 && d == 25) ||
    (m == 12 && d == 31) ||
    (m == 1 && d == 1) ||
    (m == 1 && d == 5) ||
    (m == 1 && d == 6)

The default time for dateparts() is the current time, so you don't need d, and by saving dateparts() to parts, the function is only invoked once. We old-timers love to save every cycle we squeeze out of the processor!

[therealdb]

@toggledbits said in [Reactor] Variables not updating correctly in latest-25201-2aa18550:

Just a tip, a more efficient way to write this:

OMG, very old rules, isn't it? Thanks, I fixed it.

[toggledbits]

@therealdb said in [Reactor] Variables not updating correctly in latest-25201-2aa18550:

OMG, very old rules, isn't it? Thanks, I fixed it.

This particular effort has made me look at rules that I haven't touched since the earliest days as well, many of them almost untouched copies from the Vera importer. It's a trip down memory lane, for sure, and a follow-on "wow, there's such a better way to do that now".

How's the latest working out so far? @tunnus have you had a chance to play with it as well?

[therealdb]

So far, so good. Everything seems to be OK atm. I'll report in case of problems.

[toggledbits]

OK. More testing done. I've hit a condition where the dependency scan in the last build still isn't hitting absolutely everything. Another build coming soon...

[toggledbits]

New build 25260. Also included fixes and enhancements for the expression editor.

Again, this build is intended only for the limited audience of this thread to test this specific issue. While the build at this point has had considerable testing, it's still not quite ready for consumption by all (but will be soon, if this goes well).

[3rdStng]

I don't know if my issue is related or not here. It seems like it should, but I also understand that local expression handling has changed within the last couple of builds. I don't know exactly when my issue started, nor what build I was on. My guess would be Sept 7th as my rule was last "SET" on that date.

I am running latest-25254-f6a45e41 in a docker on unRAID. The following rule has been in place for years and has been working pretty reliably, until recently.

The Set Reaction is just to turn on the shower exhaust fan. I collapsed it for screenshot space.
If I change the Local Expressions to: (and in this order)

MBath_HumityRise = "CurrentHumidity - MBath_prevHumidity"
MBath_prevHumidity = "CurrentHumidity"
CurrentHumidity = (some # value like 48)

And then manually change the CurrentHumidity value to 54 (something >=3), the rule triggers. The MBath_HumidityRise value changes to 6 too. Setting the CurrentHumidity to 50, changes the HumidityRise value to -4.

When I set these expressions back to pull from the device, I see the prevHumidity value change, but the HumidityRise value always stays 0. I set up global expression under different names and watched the values too. In the same order with the Rise expression above the Prev expression, I never see the Rise value change.

I will see if I have 25260 available for my docker update and test that, but for the life of me I cannot figure out why this rule stopped working.

EDIT: Updated to 25260 and I'm still not seeing the HumidityRise expression change. I kept my global expressions since they update and refresh on the page, where the local expressons don't appear to do so within the rule. Watching these two expressions, as soon as the prevHumidity sensor updates, the expression turns green. But the Rise expression didn't and still showed 0. In this screen grab, the humidity went from 46 to 45. I would have expected the top expression to show a -1.

[toggledbits]

Build 25261 now available for 64-bit docker installs only. Fixes an issue with Rule expressions being evaluated too frequently (but the pendulum seems to be settling toward the middle).

[toggledbits]

@3rdStng said in [Reactor] Variables not updating correctly in latest-25201-2aa18550:

EDIT: Updated to 25260 and I'm still not seeing the HumidityRise expression change. I kept my global expressions since they update and refresh on the page, where the local expressons don't appear to do so within the rule. Watching these two expressions, as soon as the prevHumidity sensor updates, the expression turns green. But the Rise expression didn't and still showed 0. In this screen grab, the humidity went from 46 to 45. I would have expected the top expression to show a -1.

OK. So let's start with the Global Expressions. The rise is always going to be zero, because it will evaluate prevHumidity to the current value of the sensor whenever it changes, and then cause HumidityRise to be re-evaluated subtracting the current value of the sensor taken from the sensor via getEntity() with the current value of the sensor stored in prevHumidity, and that will always be zero. This is as it ever was.

Rule-based expressions have changed and now operate the same way on the recent builds, so the result is going to be the same. The difference will always be zero. And when I look at your "old" way, I'm kind of surprised that worked on the older builds, too. The fact that it may have is, to me, an indicator of the kind of problems reported on this thread, a problem which you just happened to use to your advantage previously.

In working on a new approach for you, I also found another use case not handled correctly, so you'll need to update to 25261 to continue with the instructions below.

Here's a new approach going forward (requires build 25261) :

First, tie the Rule's trigger condition to any change in the device entity's humidity_sensor.value attribute. Like this:

Tip: because we're using the changes operator, when the entity's humidity value changes, the trigger condition (and therefore the Rule) will issue a very quick pulse that activates the SET reaction. That pulse is often too quick to see in the UI (its milliseconds), so setting the condition (in condition options) for pulse output with a delay of a few seconds (e.g. 5) will delay the reset so you can more clearly see the distinct phases of handling in the UI. In this case, that delay makes no impact operationally, it's just for your convenience visually.

Then, the new set of local expressions:

Only humidity_rise has an expression. In text (for copy-paste), it is:

    getEntity( "mqtt>shelly_handt3" ).attributes.humidity_sensor.value - humidity_last

The humidity_last and rise_last variables are expressionless, used as storage for the Rule's logic.

First operating principle: when an entity referred to by a Rule-based expression or trigger condition is modified, the rule is re-evaluated, and that process begins with three steps: (1) re-evaluate its Rule-based expressions (recompute them); (2) evaluate and update the state of all trigger conditions; (3) set the Rule state from trigger conditions result. This is how Rules have always worked, and they still work this way.

Finally, the SET reaction for this rule would look like this:

That script in text for copy-paste is (note the image shows this incorrectly — use the text below):

rise_last = humidity_rise,
humidity_last = getEntity( "mqtt>shelly_handt3" ).attributes.humidity_sensor.value

The conditional group in that Reaction is where you turn on your dehumidifer/exhaust fan, etc. The Script Action updates the local humidity_last and rise_last variables. The rise_last variable is new, and I'll explain why I chose to have two rise variables below.

Theory of operation: whenever the device reports a new humidity value, the difference variable humidity_rise is calculated and then the trigger condition is evaluated and will be true (because the change in humidity caused the Rule to self-evaluate), so the SET reaction runs. In the SET reaction, if the rise is >= 3%, the actions inside the conditional group will be run. To finish, the Script action stores the last humidity and rise in their respective variables. Because it does this unconditionally (outside the conditional group), the last humidity is always recorded and the rule will therefore handle rising and falling humidity values correctly.

Why add rise_last? When the trigger condition's pulse time expires (either from your condition options or the native timing of the changes pulse), the Rule is re-evaluated, and again, the first step of that is to evaluate its rule-based variables (i.e. first operating principle in effect). This is going to cause humidity_rise to go to 0, because the SET reaction made humidity_last match the entity's current humidity value (therefore the difference is now zero). I added the rise_last variable, the value of which is set only during the SET reaction, to store the humidity rise from that phase of evaluation, so you can see what it was. It is normal and you should expect humidity_rise to be 0 except when the SET reaction is running; it's now a scratch variable. The rise_last variable is your go-to reference for the last difference that was or was not acted upon by the Rule.

Also... your original expressions wrapped attribute values in int(), which truncates them (e.g. int(2.99) results in 2, but it's really almost 3). You might want to use round() instead (e.g. round(2.2) is 2, and round(2.99) is 3).

And finally, you might consider looking at the time series capabilities of VirtualEntityController. That can directly produce a change rate value (using the rate aggregator) that makes it pretty easy to determine when there's a sudden upward change in humidity.