[Reactor] Variables not updating correctly in latest-25201-2aa18550

[toggledbits]

New build 25260. Also included fixes and enhancements for the expression editor.

Again, this build is intended only for the limited audience of this thread to test this specific issue. While the build at this point has had considerable testing, it's still not quite ready for consumption by all (but will be soon, if this goes well).

[3rdStng]

I don't know if my issue is related or not here. It seems like it should, but I also understand that local expression handling has changed within the last couple of builds. I don't know exactly when my issue started, nor what build I was on. My guess would be Sept 7th as my rule was last "SET" on that date.

I am running latest-25254-f6a45e41 in a docker on unRAID. The following rule has been in place for years and has been working pretty reliably, until recently.

The Set Reaction is just to turn on the shower exhaust fan. I collapsed it for screenshot space.
If I change the Local Expressions to: (and in this order)

MBath_HumityRise = "CurrentHumidity - MBath_prevHumidity"
MBath_prevHumidity = "CurrentHumidity"
CurrentHumidity = (some # value like 48)

And then manually change the CurrentHumidity value to 54 (something >=3), the rule triggers. The MBath_HumidityRise value changes to 6 too. Setting the CurrentHumidity to 50, changes the HumidityRise value to -4.

When I set these expressions back to pull from the device, I see the prevHumidity value change, but the HumidityRise value always stays 0. I set up global expression under different names and watched the values too. In the same order with the Rise expression above the Prev expression, I never see the Rise value change.

I will see if I have 25260 available for my docker update and test that, but for the life of me I cannot figure out why this rule stopped working.

EDIT: Updated to 25260 and I'm still not seeing the HumidityRise expression change. I kept my global expressions since they update and refresh on the page, where the local expressons don't appear to do so within the rule. Watching these two expressions, as soon as the prevHumidity sensor updates, the expression turns green. But the Rise expression didn't and still showed 0. In this screen grab, the humidity went from 46 to 45. I would have expected the top expression to show a -1.

[toggledbits]

Build 25261 now available for 64-bit docker installs only. Fixes an issue with Rule expressions being evaluated too frequently (but the pendulum seems to be settling toward the middle).

[toggledbits]

@3rdStng said in [Reactor] Variables not updating correctly in latest-25201-2aa18550:

EDIT: Updated to 25260 and I'm still not seeing the HumidityRise expression change. I kept my global expressions since they update and refresh on the page, where the local expressons don't appear to do so within the rule. Watching these two expressions, as soon as the prevHumidity sensor updates, the expression turns green. But the Rise expression didn't and still showed 0. In this screen grab, the humidity went from 46 to 45. I would have expected the top expression to show a -1.

OK. So let's start with the Global Expressions. The rise is always going to be zero, because it will evaluate prevHumidity to the current value of the sensor whenever it changes, and then cause HumidityRise to be re-evaluated subtracting the current value of the sensor taken from the sensor via getEntity() with the current value of the sensor stored in prevHumidity, and that will always be zero. This is as it ever was.

Rule-based expressions have changed and now operate the same way on the recent builds, so the result is going to be the same. The difference will always be zero. And when I look at your "old" way, I'm kind of surprised that worked on the older builds, too. The fact that it may have is, to me, an indicator of the kind of problems reported on this thread, a problem which you just happened to use to your advantage previously.

In working on a new approach for you, I also found another use case not handled correctly, so you'll need to update to 25261 to continue with the instructions below.

Here's a new approach going forward (requires build 25261) :

First, tie the Rule's trigger condition to any change in the device entity's humidity_sensor.value attribute. Like this:

Tip: because we're using the changes operator, when the entity's humidity value changes, the trigger condition (and therefore the Rule) will issue a very quick pulse that activates the SET reaction. That pulse is often too quick to see in the UI (its milliseconds), so setting the condition (in condition options) for pulse output with a delay of a few seconds (e.g. 5) will delay the reset so you can more clearly see the distinct phases of handling in the UI. In this case, that delay makes no impact operationally, it's just for your convenience visually.

Then, the new set of local expressions:

Only humidity_rise has an expression. In text (for copy-paste), it is:

    getEntity( "mqtt>shelly_handt3" ).attributes.humidity_sensor.value - humidity_last

The humidity_last and rise_last variables are expressionless, used as storage for the Rule's logic.

First operating principle: when an entity referred to by a Rule-based expression or trigger condition is modified, the rule is re-evaluated, and that process begins with three steps: (1) re-evaluate its Rule-based expressions (recompute them); (2) evaluate and update the state of all trigger conditions; (3) set the Rule state from trigger conditions result. This is how Rules have always worked, and they still work this way.

Finally, the SET reaction for this rule would look like this:

That script in text for copy-paste is (note the image shows this incorrectly — use the text below):

rise_last = humidity_rise,
humidity_last = getEntity( "mqtt>shelly_handt3" ).attributes.humidity_sensor.value

The conditional group in that Reaction is where you turn on your dehumidifer/exhaust fan, etc. The Script Action updates the local humidity_last and rise_last variables. The rise_last variable is new, and I'll explain why I chose to have two rise variables below.

Theory of operation: whenever the device reports a new humidity value, the difference variable humidity_rise is calculated and then the trigger condition is evaluated and will be true (because the change in humidity caused the Rule to self-evaluate), so the SET reaction runs. In the SET reaction, if the rise is >= 3%, the actions inside the conditional group will be run. To finish, the Script action stores the last humidity and rise in their respective variables. Because it does this unconditionally (outside the conditional group), the last humidity is always recorded and the rule will therefore handle rising and falling humidity values correctly.

Why add rise_last? When the trigger condition's pulse time expires (either from your condition options or the native timing of the changes pulse), the Rule is re-evaluated, and again, the first step of that is to evaluate its rule-based variables (i.e. first operating principle in effect). This is going to cause humidity_rise to go to 0, because the SET reaction made humidity_last match the entity's current humidity value (therefore the difference is now zero). I added the rise_last variable, the value of which is set only during the SET reaction, to store the humidity rise from that phase of evaluation, so you can see what it was. It is normal and you should expect humidity_rise to be 0 except when the SET reaction is running; it's now a scratch variable. The rise_last variable is your go-to reference for the last difference that was or was not acted upon by the Rule.

Also... your original expressions wrapped attribute values in int(), which truncates them (e.g. int(2.99) results in 2, but it's really almost 3). You might want to use round() instead (e.g. round(2.2) is 2, and round(2.99) is 3).

And finally, you might consider looking at the time series capabilities of VirtualEntityController. That can directly produce a change rate value (using the rate aggregator) that makes it pretty easy to determine when there's a sudden upward change in humidity.

[tunnus]

@toggledbits, since I don't have a separate testing environment, I've been hesitant to update to this build. Still trying to build up the nerve to take the plunge

[CatmanV2]

Fabulous effort here by @toggledbits We owe a great deal!

C

[toggledbits]

@tunnus I don't blame you for hesitating at all. It's a potential upheaval. I think it will mostly affect people that have unwittingly taken advantage of specific quirks of the old behavior that now are gone, and hopefully those are few. I have a test environment, but my home is always the final boss of QA.

@CatmanV2 said in [Reactor] Variables not updating correctly in latest-25201-2aa18550:

Fabulous effort here by @toggledbits We owe a great deal!

I can't take any bows for this. This is fixing past mistakes, decisions I've made (or side-effects that I just accepted) that I wish I hadn't, and now fixing it means potentially putting many in the community through some pain. Nothing about that feels good, but if I don't fix this now, it will continue to be thorn in everyone's side and I find that even less acceptable. This, too, shall pass.

[CatmanV2]

I'd not agree. You were not obliged to build it in the first place!

C

[toggledbits]

Build 25264 now out, again, docker 64-bit only, for the limited audience of this topic right now. A bunch of edge cases figured out and a lot of code cleanup.

[tunnus]

@toggledbits, I bit the bullet and tried this build, but getting the following errors on UI:

Unable to configure time series "temperature_sensor.value" on "virtual7b": either 'depth' or 'retention' should be specified (but not both)
Controller: VirtualEntityController#virtual
Last 12:35:54


Unable to configure time series "value_sensor.value" on "virtual7": either 'depth' or 'retention' should be specified (but not both)
Controller: VirtualEntityController#virtual
Last 12:35:54


Unable to configure time series "temperature_sensor.value" on "virtual6": either 'depth' or 'retention' should be specified (but not both)
Controller: VirtualEntityController#virtual
Last 12:35:54


Unable to configure time series "temperature_sensor.value" on "virtual5": either 'depth' or 'retention' should be specified (but not both)
Controller: VirtualEntityController#virtual
Last 12:35:54

Didn't change my configuration as manual still had examples where both are used. So is this a bug or has time series functionality changed?

Below a snippet from my reactor.yaml:

 id: virtual5
          name: "Nibe out temp"
          capabilities:
            temperature_sensor:
              attributes:
                value:
                  model: time series
                  entity: "hass>sensor_bt1_outdoor_temperature_40004"
                  attribute: "temperature_sensor.value"
                  interval: 5  # minutes
                  retention: 20  # minutes
                  aggregate: wa
                  depth: 4
                  precision: 1
                  weight:
                    - 0.50  # most recent sample, 50% weight
                    - 0.35  # 35%
                    - 0.10  # 10%
                    - 0.05  # 5% oldest sample of the four considered
          primary_attribute: temperature_sensor.value
          type: ValueSensor

[toggledbits]

This is a coming functionality change. I recommend commenting retention out. I am considering allowing both as long as they make sense together (i.e. retention is equal to or greater than that required by the configured interval and depth), and that may be done before the final release.

[tunnus]

@toggledbits configuration changed, no more errors. The bug I was experiencing (described in post #23) is also gone

Will continue testing...

[tunnus]

@toggledbits, now getting the following error which I haven't seen with earlier (official) builds. To be clear, build in use is 25264.

The global expression "VirtualTemp" isnull(getEntity( "weather>xxxxx" ).attributes.wx.temperature) ? VirtualTemp : round(getEntity( "weather>xxxxx" ).attributes.wx.temperature, 1)) could not be evaluated
Invalid scope in reference to member temperature of (object)null
Last 20:24:18; this alert has repeated 42 times.

(location redacted)

The above is actually an attempt to fix the situation (which did not help, as I'm getting these "invalid scope" errors), originally I had the following global expression:

VirtualTemp = round(getEntity( "weather>xxxx" ).attributes.wx.temperature, 1)

I'm wondering if this situation has been active already earlier, but maybe some change in this (experimental) build has made it visible?

Note that I'm not seeing these null values in the UI, e.g. as of now, in entities, wx.temperature has a value of "8.560000000000002" (rounded to 8.6 in global expressions)

[toggledbits]

If your intention is to guard against request/response failures from the weather API, that would be better written something more like this:

isnull(getEntity( "weather>xxxxx" )?.attributes?.wx?.temperature) ?
    VirtualTemp : 
    round(getEntity( "weather>xxxxx" ).attributes.wx.temperature, 1)) 

The ?. operator guards against its left side operand resolving to null.

Shorter version:

local d=getEntity( "weather>xxxxx" )?.attributes?.wx?.temperature,
isnull(d) ? VirtualTemp : round(d, 1)

And yes, the change in the variable approach exposes something that OWMWeatherController should be doing that it's not (and I've spotted and fixed that for a future build). If you're using another weather source, likely same/similar.

[therealdb]

@toggledbits I have a couple of routines (previously working OK) that are checking for various things based on the hour we're in.

The variable code is like this one:

d = dateparts(),
(d.hour >= 20 && d.hour <= 23)

I remember we've been told that this would not work in previous builds (but it did), but it's not really working now. It was false at 20 (the rule is triggered every 30 minutes when dark), and after I pressed play it went to true. I changed it with a script action and it's working now. But I'm reporting as an edge case, because I'm not sure it is supposed to work with the new behavior. Nothing else to report ATM. Thanks for the effort!

[toggledbits]

Build 25272 now up for docker 64-bit only once again. A good bit of cleanup again.

@therealdb wrote:

But I'm reporting as an edge case, because I'm not sure it is supposed to work with the new behavior.

I'm keeping it this way (not working) for the moment, because it keeps it completely consistent with how Global variables work. But I see why you'd want to do what you did, and I think I can make it work... give me a day or two...

[tunnus]

@toggledbits also using similar variables as @therealdb, so nice that there will be a "fix"

prevHour = dateparts(time()).hour - 1
currentHour = dateparts(time()).hour

[tunnus]

Testing build 25272, and although the following findings may not be specific to this build, I'm reporting via this thread.

First, I noticed that if you have a long array in local expressions, UI is horizontally (over)stretched when viewing the rule (does not happen in edit mode). See below.

Secondly (and this can be user error), I was experimenting with failure handling and got the following:

The expression "isnull(blob?.data?.PriceWithTax) ? \"ERROR\" : blob.data.PriceWithTax" in variable "hourPriceBackup" of rule "Get prices" (rule-m0bb80xn) failed evaluation
ReferenceError: Invalid scope in reference to member PriceWithTax of (number)0.10595
Last 20:55:07; this alert has repeated 2 times.

(PriceWithTax was intentionally left out from the response to cause error)

And below the actual expression:

What would be correct way to guard against this?

[toggledbits]

The error here is that data isn't an object, so you can't dereference through it. If you look at your blob, you've set data to a number (0.10595), and the error is telling you that you can't dereference through a number like an object. The coalesce operator isn't going to apply here, because data is just the wrong type. I assert that your test case is broken and testing something you are never actually going to encounter in real life. Well-written APIs, for example, don't do things like this, at least, not without other clues (e.g. a type value in the response to tell you what kind of data is in data, so that you can easily know how to handle it).

[tunnus]

@toggledbits okay, maybe my test case wasn't the best possible... what about that UI issue (a long array)?