Moving MSR from a QNAP container to RP 5 - some issues

[therealdb]

Hi @toggledbits

I'm working on a controller to generate llm response from a prompt in reactor. I have http response coming thru an http request action at the moment, capturing the response inside a local variable. So, it's practically sync.

I want to create a controller, so I don't have to rely on a proxy (and have a simpler architecture), and duplicate absurd http actions, but AFAIK in the current implementation, actions are async only. But if I have multiple requests going on, I cannot be sure what it's really inside an attribute. I also thought that something like a correlation id when sending the request could be used to identity multiple responses, but I wanted to double check with you before starting with something too complicated. I also noticed that some actions in home assistant (ie forecast) are sync and I'm wondering if you have any plan or hint to address this situation. Thanks.

Thanks.

[toggledbits]

@therealdb said in [Reactor] Services with sync response:

I want to create a controller, so I don't have to rely on a proxy (and have a simpler architecture), and duplicate absurd http actions, but AFAIK in the current implementation, actions are async only. But if I have multiple requests going on, I cannot be sure what it's really inside an attribute.

Sorry, I'm not really clear on what you're asking here. When you say "actions are async only", in what context are you referring to? Do you mean an action running in a Reaction, or do you mean the implementation of an action in a Controller instance?

[therealdb]

sorry, I was on my phone. I was referring to the fact it's impossible to get the response from a service inside a reaction. so, let's says I need to do that in order to pass a prompt and generate a response:

HTTP action is able to capture the reply in a variable. I want to build a custom controller with an action that could do the same, so reply in line with an output that I could capture inside a local variable (because replicating this code multiple times is not really efficient nor smart). Is that clear now?

[toggledbits]

Got it. Let me back up a second and supply a bit of background.

When a Reaction is asked to perform an action on an entity, the Engine's executive calls Entity.perform() on the target entity with the requested action name and parameters. By default, the entity will then call its Controller's performOnEntity() with itself, the action name, and the action's parameters. The Controller is expected to provide whatever implementation is necessasry to cause the target object represented by the entity to perform the desired action. The performOnEntity() implementation is expected to return a Promise that resolves when the command has been issued successfully (or, when the command complete successfully, if that is something that can be determined). That Promise is passed back as the call stack unwinds until the Engine gets it. At that point, the Engine will suspend execution of the Reaction until the action Promise is fulfilled (successfully or not).

TL;DR: The Engine waits for a Controller to signal that an action in a Reaction has been requested (or completed) before moving on to the next action.

I think this is useful for you. Your implementation of the action must return a Promise that must resolve before the Engine moves on. So your action can launch its HTTP request (e.g. using Controller.fetchJSON() or .fetchText()), which itself is an asynchronous operation, and wait for that response, then do whatever processing it needs to on that response, store the result wherever it needs to go, and then and only then resolve the original Promise. The Engine will be waiting that entire time before moving on to the next action in the running Reaction.

A couple more things to know:

• If you don't supply an override of performOnEntity() in your Controller implementation, the default (Controller base class) implementation will try to call a method called action_<capability>_<actionname>. For example, your Controller could implement the action x_mycontroller.update_weather by providing a method action_myController_update_weather( entity, parameters ) (where entity is the target and parameters is an object of key/value pairs).
• If you decide to implement an override of performOnEntity(), make sure it calls super.performOnEntity() if it can't figure out what to do (i.e. it is asked to perform an action it doesn't understand).

Hopefully I've answered your question and maybe provided some additional information that will help.

[therealdb]

Ok, that's clear and I think I've already done that in a coupe of controllers.
What I'm saying here is that I need to do something similar to be sure that I could capture the response and be sure I'm not getting another one. I want to capture the output in a variable like HTTP actions, but using an entity action.

AFAIK I cannot do that in a reliable way.

[toggledbits]

I want to capture the output in a variable like HTTP actions

If you're building a Controller, it would (IMO) make more sense to store the query result on an entity (even if it's the system entity for the Controller) in an attribute rather than a global variable.

AFAIK I cannot do that in a reliable way.

I'm not really sure where you are going with the "capture the response and be sure I'm not getting another one". If you are using any of the fetch methods, for example, the response would be handled on the Promise that was created:

     this.fetchJSON( "http://example.com/myquery" ).then( jsondata => {
        // The data from the response is now in `jsondata`. It will contain
        // only the data for this query, no other.
        this.getSystemEntity().setAttribute( 'x_mycontroller.myquery_result', jsondata );
    });

There are many existing Controllers that can submit multiple commands to a hub/service and handle a truly asynchronous response (e.g. via a callback or a later message on the channel). In these instances, a unique (for the session) ID is passed as part of the request, and the hub's API returns the ID as part of its response, so we know what the hub is responding to. Both HassController and ZWaveJSController do this, for example.

Here's what that looks like for both of these Controllers. This method is used to send a request to the hub. It assigns a serial ID as part of its work:

    /** Returns a Promise that sends a request to ZWave JS and resolves when it gets
     *  the matching reply. The reply can time out, and the Promise rejects.
     */
    _send_tracked_message( req, timeout ) {
        const self = this;
        timeout = timeout || DEFAULT_TASK_TIMEOUT;
        let slot = {
            req_id: ++this.next_msg_id,
            expires: Date.now() + timeout,
            resolve: false,
            reject: false,
            req: req
        };
        slot.promise = new util.TimedPromise( (resolve,reject) => {
            // Now that we're in the Promise, save the resolvers for later use when we get the response.
            slot.resolve = resolve;
            slot.reject = reject;
            req.messageId = slot.req_id;
            self.log.debug( 5, "%1 sending #%2: %3", this, slot.req_id, req );
            self.ws_send( JSON.stringify( req ) );
        }, timeout ).catch( err => {
            self.log.err("%1 request %2 (%4) failed: %3\n%5", self, slot.req_id, err, slot.type, slot.req);
            slot.reject( err );
            delete self.pending[String(slot.req_id)];
        });
        this.pending[String(slot.req_id)] = slot;
        return slot.promise;
    }

The Controller has an instance variable (an object) called pending that keeps track of all pending requests; it's created empty ({}) in the Controller's constructor and when a connection to the hub is acquired. Likewise, the next_msg_id variable is initialized to 0 in the constructor. The method creates a slot, which is just an object with enough data to track the request, using a TimedPromise (in util library) so that we can have a timeout if the hub never answers. The slot is stored in this.pending so we can get back to it later, whenever/however the response arrives.

The hub later sends a response somehow. In Z-Wave JS that's a result type message on the WebSocket channel, and the handler looks like this:

    /** Called for "result" type messages. If associated with a tracked request, that Promise will
     *  resolve or reject based on the returned success flag. Otherwise, the message is logged (if
     *  level is high enough) and the message discarded.
     */
    _handle_result( event ) {
        this.log.debug( 9, "%1 full result reply %2", this, event );
        // Find the slot in pending
        let slot = this.pending[String(event.messageId)];
        if ( slot ) {
            delete this.pending[String(event.messageId)];
            if ( event.success ) {
                this.log.debug( 5, "%1 request %2 (%3) success notification", this, slot.req_id, slot.req.command);
                slot.resolve( event );
            } else {
                // Ref: https://github.com/zwave-js/zwave-js-server#errors
                this.log.debug( 5, "%1 command result error: %2", this, event );
                let err;
                if ( "zwave_error" === event.errorCode ) {
                    err = new Error( `Z-Wave error ${event.zwaveErrorCode}: ${event.zwaveErrorMessage}` );
                } else {
                    err = new Error( event.errorCode );
                }
                slot.reject( err );
            }
        } else {
            this.log.debug( 6, "%1 can't resolve request %2 in reply, slot not found, maybe already timed out", this,
                event.messageId );
        }
    }

In this case, the slot data is acquired from this.pending. The slot is removed from this.pending, and then the stored resolve() function is called (with the result message data) if the hub indicates success, or reject() is called (with the error data) otherwise.

Does this help?

[therealdb]

that's fair, but I think the problem is different. My flow is this one:

Prompt => OpenAI => response

It's not really different from what HTTP action is doing, after all.
Prompt is by definition dynamic, so I cannot "bind" it to an entity. I think having multiple entities just because I need multiple response is cumbersome (and, probably, not efficient). It will eventually lead to confusion and complicate the final setup, because every time I need to write a new prompt, a new entity will be needed as well.

Now, if I create multiple entities, your remarks are obviously right. I did a couple of controllers and the infrastructure is pretty clear to me. But if I want to create just the system entity, with an action like x_ai.prompt, I cannot store the response inside an entity attribute, because I could potentially run multiple x_ai.prompt requests and this could cause a lot of troubles. I understand that in a entity-first world this is an edge case, and in fact this is more a service-oriented Controller than the traditional ones, but nevertheless I thinks this use case will eventually surface more and more. It's like the perform_action returning an object as result using the Promise, instead of just being void.

In my comment, I was referring to a local variable. Imagine a prompt giving you a number as a result, because you're using the GenAI engine to make some prediction based on your logic. By capturing this value inside a variable, I could write, inside the same rule, the logic necessary to tie this value to some specific actions, like turning on some light, or send a notification if a threshold is met. I know I can assign it to the system entity, but this will need multiple rules to capture the result, so this will probably make things very complicated.

I'm asking for some guidance in order to implement this kind of architecture, where the response is directly tied (and captured, and used) in the same context of the rule where the request is generated. I'm not sure if this is clearer than my previous attempt to describe the question. Thanks!

[toggledbits]

What causes the request? Local variables live in rules, but rules are only evaluated (and local variables updated) when a dependency notifies a change. What will cause the query to the remote endpoint?

[therealdb]

It could be a door being opened, a time trigger - or something else. The prompt will be generated from a source (this could be some text coming from an e-mail, a web site, or status coming from sensors - sky's the limit), sent to the OpenAI (or something else), processed, and a response generated. The output could be a JSON, a string, or whatever you want. The GenAI model is capable of generating everything, based on the prompt. This is a basic example:

The prompt is composed inline, and the response is being generated (and, in this case, sent to Alexa for TTS).

Here's another variation, for a JSON payload:

It's crucial to send the request and get the response in the same logic, otherwise it will be impossible to easily build something with this. Thanks.

[toggledbits]

OK. Still not fully there for me. When you run the HTTP Request action, it waits for the response, and then stores the response in the specified local variable. That value is now available to the remainder of the Reaction, so you can send it to TTS.

What am I missing?

[therealdb]

I don't want to use an HTTP action, but a controller one! Right now it's impossible to save entity action response to a variable, if I'm not missing something. The question is this: how to build the same flow using a controller action?

[toggledbits]

But you can use the entity-based action response as if it was a variable. It is in many ways better, in fact, because there's a ton more code required in Reactor to make rule conditions responsive to variables than there is to make it responsive to attributes — attributes are very efficient.

I know you don't want to use an HTTP Request, but I want to know why. I guess I'm not understanding what's inadequate about the HTTP Request for your purpose in the first place.

[therealdb]

My /ai/prompt endpoint encapsulate the logic to query OpenAI. That's great for me personally (so, not really shareable with the community), but that's another moving part in the system that could potentially fail. This is a simple web API that is proxying OpenAI (in this case Azure Open AI, but that's the same). It is acting as my controller, but it's not reusable, it's prone to errors and I think an OpenAIController will be better than this.

All that said:

But you can use the entity-based action response as if it was a variable. It is in many ways better, in fact, because there's a ton more code required in Reactor to make rule conditions responsive to variables than there is to make it responsive to attributes — attributes are very efficient.

So, you think that it's better to have two rules, one triggering the request and another looking for the response and watching the attributes (ie: response and response_id, tied to a request_id from the original prompt)?

[toggledbits]

I'm not understanding why you think it takes two rules.

If you run an HTTP Request and store the response in a local variable, the next step in the same Reaction has access to that local variable. And the contents of that local variable will be exactly the response to the request, not any other request (so no ID tracking is necessary). The Rule that triggers the request can be the same rule that processes it.

Are you assuming that the HTTP Request action is asynchronous when storing the result? Because it's not. It will wait for the result and store it before moving on to the next step of the Reaction. That's documented.

[therealdb]

No, I'm quite experienced and I know how a promise works

Anyway, the point here is different but I'm not able to fully explain myself. I'm out for work for a couple of days, but I'll post a photoshopped solution, so I hope things will be more clear.

[toggledbits]

@therealdb said in [Reactor] Services with sync response:

No, I'm quite experienced and I know how a promise works

I'm not talking about a Promise, and of course I know well that you know how that works. I'm talking about the HTTP Request action in a Reaction. It waits for the response and stores it, so that the rest of the steps of the Reaction can work with the response received. If that doesn't address your need, I'm unclear how.

[therealdb]

ok, so, here's a photoshopped version of the ideal world I'm trying to describe:

The point here is that I don't want to do an HTTP request, I want to call an entity action and capture the result, if any, in a variable. Keep in mind that I'm describing an ideal world, and I'm OK to alternative solutions that still uses a Controller (and not an HTTP Action, that's temporary and I want to move to a custom Controller), wait for a response from a service, capture it and use in the same flow, if possible.

So, more like services than entity-oriented controller.

I hope this is better explained now. Thanks!

[toggledbits]

OK. I think I see where you are going. I was bogged down by the HTTP part, since that's existing functionality you can solve the problem with today. You're really asking for something much bigger. You just generally want entity actions to be able to return data, which they currently don't, so you can do things like:

• Return the next upcoming event from a calendar;
• Receive a value and look it up in a table, then return the found result;
• Given the name of a playlist on a media player, return the list of media it contains.
• Given the symbol for a security or crypto coin, return its current price.
• Run a script on Home Assistant and receive whatever response it returned.

And you want this to work in-line in Reactions, without necessarily binding the result value to an attribute on the entity performing the action, or having to preconfigure an input in YAML config (e.g. the symbol for a stock or crypto asset to be fetched/polled). The result value is, in fact, typically temporary and not useful outside the context of the Reaction (although it and values derived from it can be stored otherwise by other steps of the Reaction).

Do I have it now?

[therealdb]

@toggledbits yes sir. That why you're the master and I'm your humble follower

Because I know it's not trivial, I'm also asking for guidance on how to achieve something similar with what we have today. Thanks.

[toggledbits]

I'm on it. Sit tight.