Generic support for external plug-ins in different languages

[therealdb]

I was toying with the idea to write a simple openLuup plug-in to execute any external plug-in (ie written in nodejs, puthon or even bash or .NET) via an openLuup plug-in, with access to luup's variables and/or the ability to execute actions against an external executable.

The general idea is to get an existing library (ie: the one controlling LG TV, or SwitchBot) that's written in another language and write a simple (and generic) bridge to represent a device, getting variables from a command line bash script (or nodejs app or whatever) and leverage someone else's work, instead of playing catch-up and try to port it over to LUA.

I did it by myself in my own fork of the MQTT2Luup bridge I released and it's working very well for me. I was able to implement support for Alexa, busylights, lgtv, netgear routers, switchbot and much more, all using established or official node packages, with a simple library to send command to the node lib (simulating a console in) and getting response from them (reading the console out). So my logic is still in the luup/openLuup engine and I could mix and match (ie: getting my TV volume to know if I want to raise up/down the thermostat temperature based on occupancy, or treat switchbots as normal light switch - and control them via Alexa and so on).

So, instead of doing it all by myself, I'm wondering it this could interesting as a generic openLuup capability, or as a general and "standardized" effort. I know it sounds like MultiSystem Reactor, but it's not, since the primary purpose is to be specific to openLuup devices and leave the logic to the user (this could be of course Reactor, the upcoming MSReactor or whatever you want). Am I the only one mixing things together and with such a need?

[PerH]

No, you're not alone. I have (at least) two plugins i want to "steal" from other systems, Tibber integration and Xiaomi Vacuum controller.. Found them in .NET, java and python..

[akbooer]

The real question is what the API should look like and what the parameter/command passing mechanism actual is. I'm happy to try and advise how best to integrate such a bridge with openLuup.

[therealdb]

I don’t have a solution. I want a formal way to do it, so we could write adapters and execute the code externally. Maybe something to easily map in/out console to variables/actions. I don’t want to do http requests and virtual devices, even if I’m doing this route and it’s somewhat ok, but not easily packaged nor user friendly (a lot of setup is needed). Using this route, instead, discoverability is possible.

[akbooer]

How about making them all MQTT clients?

[rafale77]

@akbooer said in Generic support for external plug-ins in different languages:

How about making them all MQTT clients?

I thought indeed that this was the very goal of MQTT... a universal and efficient framework to support multiple platform/languages. It just hasn't been super universal due to the diversity of implementations/lack of standards. It would be very interesting to see how all of this evolves into and I like the idea. I just don't know how to get there. MQTT is the closest I can think of.

[therealdb]

Well, as you know I have MQTT in place for a lot of things, but you'll need a lot of manual steps. I think MQTT support inside openLuup is better than anything else, but I'm not sure auto-discovery could be easy. The fact that any device has its own commands could be easily resolved, as I did in my own bridge. It will require manual steps for sure, but it's doable.

[akbooer]

@rafale77 said in Generic support for external plug-ins in different languages:

It just hasn't been super universal due to the diversity of implementations/lack of standards.

Really? I've been reading the OASIS standards document, and it seems very well defined.

MQTT Version 3.1.1

@therealdb said in Generic support for external plug-ins in different languages:

I think MQTT support inside openLuup is better than anything else

Are you talking as a philosophical point of view, or from a practical one? If the latter, then which client implementation (your own?)

I ask because I'm well into implementation of a QoS 0 MQTT server (aka broker.) It seems to me that this is the preferable approach when hosted by openLuup itself. It means that message travels directly between controller and device, rather than requiring two hops through a third-party (albeit local) server.

I already have my bare-bones implementation up and running for my Shelly devices. It's very fast and very easy to use from a plugin / Lua Test because you can subscribe directly to MQTT topics using

luup.request_handler ("MyMQTThandler", "mqtt:topc/name/goes/here")

The Shelly bridge requires no configuration, auto-installs when the first Shelly device announces its presence over MQTT, and fully configures child devices.

Hopefully available for wider testing in the not too distant future.

[therealdb]

@akbooer said in Generic support for external plug-ins in different languages:

Are you talking as a philosophical point of view, or from a practical one? If the latter, then which client implementation (your own?)
I ask because I'm well into implementation of a QoS 0 MQTT server (aka broker.) It seems to me that this is the preferable approach when hosted by openLuup itself. It means that message travels directly between controller and device, rather than requiring two hops through a third-party (albeit local) server.

nope, I was referring to yours Living inside openLuup is better from a performance point of view, no doubt here.

The Shelly bridge requires no configuration, auto-installs when the first Shelly device announces its presence over MQTT, and fully configures child devices.

sounds great, and this could be of use for future plug-ins as well. As I've already said, I have all kind of devices in place, using virtual devices and MQTT (temp, humidity, light sensors, distance, pressure, fertilization level, owntracks and many more).

Are you talking as a philosophical point of view

I think @rafale77 was referring to the lack of standardization of topics/messages. I think part of the work could be to just let this configurable for the users. As I said, I've now moved to MQTT and I'm currently dispatching events from my luup engine via my bridge/broker, and I'm using some of them to instruct my PC via a special Windows app. This way, I have notifications, but also events (ie: turn on lights via my PC if I'm at work when it's dark in the office, turn them off when the screensaver kiks in, show the cams automatically when there's motion, etc). All I did is to write a client and subscribe to my own topics. Very powerful.

[rafale77]

@therealdb

@therealdb said in Generic support for external plug-ins in different languages:

I think @rafale77 was referring to the lack of standardization of topics/messages. I think part of the work could be to just let this configurable for the users. As I said, I've now moved to MQTT and I'm currently dispatching events from my luup engine via my bridge/broker, and I'm using some of them to instruct my PC via a special Windows app. This way, I have notifications, but also events (ie: turn on lights via my PC if I'm at work when it's dark in the office, turn them off when the screensaver kiks in, show the cams automatically when there's motion, etc). All I did is to write a client and subscribe to my own topics. Very powerful.

Indeed, that's exactly what I was referring to.
I was inspired by your comment on z-way supporting WebSockets and indeed... it does it through a plugin. It also supports MQTT. Would be a great idea to move the z-way bridge to MQTT? I was also looking at ALTUI and how heavy it is on openLuup when an ALTUI page is open as it polls the openLuup API at a high frequency. Wouldn't it be better if it connected through WebSockets or MQTT?

[therealdb]

with mqtt inside openLuup, you'll completely remove many, many HTTP calls, and you'll not need polling anymore. so yes, MQTT is better from this point of view.

[akbooer]

@therealdb
@rafale77

Indeed, it would make HTTP almost redundant for this application!

My development of openLuup’s own MQTT server is nearing the stage where it’s ready for Beta testing. It’s really nice to use with the Shelly devices (currently I only have i3 and sw2.5 implemented, but easily extended.). The bridge is by far the simplest that I have implemented, and so fast.

[rafale77]

I am definitely looking forward to the embedded MQTT integration. I just revived my mosquitto broken and will be tinkering with it, time permitting.

[PerH]

Just barely grasping the concept of MQTT here, but when you say embedded MQTT server, is that the same as a broker? or would one need that as a independent program?
If I understand this correctly, it seems to make many bridges redundant as well? Z-way, Domoticz, mabye not vera, but probably ezlo ..?

Sound interesting either way!

[rafale77]

@perh

Let me try to explain it in more prosaic/layman terms:

The way the z-way bridge, the vera bridge and domoticz bridge work today is the host controller (i.e openLuup) is polling the server at regular intervals to get updates on every device over http. This is the concept of the REST API polling. The host can also send singular commands to that API to control devices on the secondary controller. The polling scheme generates a lot wasted traffic obviously because there is traffic even when nothing is happening/changing.

The idea is the same as for the zwave network management and the regular interval polling there which I advocated to eliminate back on the vera forum as z-wave devices don't need this anymore except for old light switches which did not support "instant status".

Another way to do this is to trigger updates only upon events. That's much more complicated to do because it requires setup on both the host and secondary controllers. The secondary controller needs to know where to send the messages to when an event occurs. That's what I did for home assistant to openLuup using purely HTTP APIs. It is tedious and every new device requires manually creating a virtual device on openLuup and a set of scenes/automation on both controllers to update one another and keep their status in sync over HTTP calls but it is a lot more efficient.

Now comes MQTT: It is a light weight protocol requiring a broker to manage all that traffic. It has all the various "clients" which are our controllers for example home assistant, z-way, openLuup etc... publish messages into topics. The broker then sends updates to all the controllers which subscribed to that topic. It is centralized event based communication and somewhat standardized as long as the different clients know how to interpret the messages and topics from one another (this is the part which is not standardized). It eliminates a ton of unnecessary polling and enables many clients to get updates instead of doing one to one communications.

A server is not a broker. I think AK is working on a client/server for openLuup to subscribe and publish to MQTT. There has been a variety of plugin attempt to do this but doing it inside of openLuup is new and make a lot of things simpler.

[PerH]

Thanks!

Have to correct you on the Domo bridge though, it includes a script in Domoticz that sends event updates immediately, polling is only needed if you havent installed that script. All that's needed in terms of domo setup is a user variable with an IP to openluup.

I guess that the broker would be even another container in my setup then.

[therealdb]

@rafale77 said in Generic support for external plug-ins in different languages:

It is centralized event based communication

MQTT is distributed, to be completely right

So you could publish messages to n brokers, subscribe to topics from m clients, and have k servers, all working together.

Let put it in context.

Shelly => Broker => openLuup
                 => Home Assistant
                 => MS Reactor

or

MS Reactor => Broker => openLuup
                     => InfluxDB
                     => your own client

Possibilities are limitless. Bonus point is that you could integrate a lot of things (ZigBee2MQTT, ZWaveJS2MQTT, Hue2MQTT) with virtual no native plug-ins required in openLuup, beside a bunch of (easy) virtual device, to route the implementation of commands.

I've asked the eZlo guys to do this for luup/ezlo fw because it's really powerfull, when you realize a single box is not enough in order to truly mix and match various techonologies.

[rafale77]

@PerH
Thanks for the precision on the domo bridge. I missed that.

I am a bit allergic to containers... my entire setup (habridge/homekitbridge/z-way/home-assistant/mosquitto/openLuup/grafana) runs within a single VM on my NAS as I didn't see enough significant benefits of containerizing to justify the added complexity and overhead. It just takes 2 command line entries and a 2 line edit of a config file to install the broker. It is also much easier to upgrade and modify. Maybe I will come to it one day but I really nowhere close to it now.

[akbooer]

@perh said in Generic support for external plug-ins in different languages:

MQTT server, is that the same as a broker?

@rafale77 said in Generic support for external plug-ins in different languages:

A server is not a broker.

To be sure 'server' is an overloaded term. However, within the scope of MQTT, a server is a broker. The OASIS MQTT specification (version 3.1.1, which is what I'm implementing) uses the term exclusively, and defines it (on lines 31-32) thus:

"A program or device that acts as an intermediary between Clients which publish Application Messages and Clients which have made Subscriptions."

I have, indeed, written a server, so you will not need Mosquitto, or any other server, installed anywhere, although, as @therealdb points out, you can have multiple servers (on different ports.)

[a-lurker]

@akbooer said in Generic support for external plug-ins in different languages:

I have, indeed, written a server, so you will not need Mosquitto, or any other server, installed anywhere

Fantastic. Thanks for your efforts on this one ak. It should be a really good addition to openLuup.