openLuup email server

[a-lurker]

Re: openLuup: MQTT server

@ronluna

Hoping you could tell us a bit about your experiences with ZWaveJS and MQTT.

[ronluna]

I was unable to get MQTT to work correctly in openluup... I had to give up... It's a real shame ... as ZwaveJS is by far the best thing I've seen to manage zwave networks.

[a-lurker]

@akbooer @ronLuna

I've been investigating using Z-Wave-JS with openLuup. I get these subscriptions reported in the console in openLuup.

zwave/+/+/+/+/+/set
zwave/+/+/+/+/set
zwave/_CLIENTS/ZWAVE_GATEWAY-zwave-js-ui/api/#
zwave/_CLIENTS/ZWAVE_GATEWAY-zwave-js-ui/broadcast/#
zwave/_CLIENTS/ZWAVE_GATEWAY-zwave-js-ui/multicast/#

I can run test code that publishes to this topic:

zwave/_CLIENTS/ZWAVE_GATEWAY-zwave-js-ui/api/#

and it all works. But this is not the way the developers envisage how you should use MQTT with their app, as they are more low level. These topics need to be used instead:

zwave/+/+/+/+/set
zwave/+/+/+/+/+/set

Currently the openLuup broker does not handle the + wild card, so these examples below do not work. The examples represent how Z-Wave-JS should/could be commanded:

zwave/4/38/0/targetValue/set
zwave/Test_dimmer/switch_multilevel/endpoint_0/targetValue/set
zwave/Lounge_downstairs/Test_dimmer/switch_multilevel/endpoint_0/targetValue/set

So my questions are: is it a big task to get the + wild card working? And in the interim can I force a publish of the topics listed immediately above for test purposes?

[akbooer]

Yes, this definitely needs to be addressed.

There is (or was) a performance reason for not implementing the '+' wildcard... every topic published need to have the match tested. But I believe, now, that there is a way around this.

The MQTT server was written carefully, especially the low-level implementation, but as a total system was never fully completed, since it was really written for basic built-in support of Shelly devices, rather than a more general service. But now its uses have grown, and I think it needs revisiting.

[akbooer]

I am currently testing a version of the server which correctly handles all wildcards…

[a-lurker]

@akbooer Excellent - it would certainly help for those who want to use ZWaveJS.

[akbooer]

Is this something you plan on testing?

[a-lurker]

If I remember correctly; you said you had removed all your ZWave gear. I've still got a Vera Edge acting as a ZWave radio with about thirty devices. Works fine but I don't want to be locked into to that long term. Ultimately I would like to see the ZWay plugin MQTTized. I believe that's possible from looking at the code. So I can have a go at testing; with the + wildcards certainly as a starting point. Might need to do a few backups regardless!!

[akbooer]

@a-lurker said in ZWaveJS and MQTT:

Ultimately I would like to see the ZWay plugin MQTTized.

What would that look like? All the ZWay devices reflected in openLuup should be accessible by MQTT? I can turn switches and lights on and off using MQTT commands already.

[a-lurker]

"How would that look like?"

In my case I have single relays, double relays, shade controllers and minimotes. That's it. At a basic level, I could set up some virtual devices and sensors with associated plugins. However this would take a while to set up.

openLuup could contain some code that just handles these items but then someone else comes along and wants a security sensor or a worst still say a thermostat.

I'm not overly familiar with ZWay or the openLuup ZWay plugin implementation, so please excuse any displayed ignorance.

I see the ZWay plugin as the code basis for a new completely separate plugin for Z-Wave JS.

ZWay has an "Expert User Interface" that does low level stuff such as include and exclude. Z-Wave JS also has a similar low level interface. None of that's required, as it can all be done using the "Z-Wave JS UI" (using their exact naming). Likewise, I don't see any point in trying to support ZWave scenes or associations.

Comparing ZWay with Z-Wave JS gettors & setttors looks like this:

ZWay:

Set:  http://openLuup-ip-address:8083/ZWaveAPI/Run/devices[4].instances[0].commandClasses[38].Set(100)

Get:  http://openLuup-ip-address:8083/ZWaveAPI/Run/devices[4].instances[0].commandClasses[38].Get

ZWave JS: see example setting a value

Set:  zwave/4/38/0/targetValue/set
with json payload:
{
    "value": 100,
    "options": {
        "transitionDuration": "5s"
    }
}

Get:  zwave/4/38/0/targetValue  that is: exact same command without the "set" suffix.

Where:
- devices[4] becomes 4      (device id)
- instances[0] becomes 0    (endpoint id)
- 38 is the command class

The openLuup ZWay plugin has all the command classes already defined and hooked into openLuup.

The command classes as seen by Z-Wave JS are detailed here.

and MQTT here.

Authentication would just be the usual MQTT broker authentication. No login is required.

Topics: Z-Wave JS allows a variety of topics and payloads to be formed, based on the user settings. See here.

This format seems the simplest, as it excludes room names and device names:

zwave/4/38/0/targetValue

CloneRooms and NameDevices booleans: Initially it would appear easier to place all ZWave devices in the "ZWave room" (as would be seen in say AltUI) and then reassign the various devices to their rooms with their names using say AltUI.

You could then use setNodeName and setNodeLocation to push the info back to Z-Wave JS?

Note: where it says "➤Mqtt usage" click on the little arrow head "➤" immediately to the left to see the info (I'm a bit slow - took me awhile to realise this).

zway_cgi.lua - not sure how this is used (configs ZWave devices via a web page?) but it looks like it's not needed?

I see the ZWay plugin uses async and non async HTTP requests - not sure why this is? Is it just about advantage of "http_aysnc.lua" in openLuup when its available?

getNodes, getInfo, pollValue and pingNode are potentially useful?

Looking at it somewhat naively, it looks like some minimal functionality could be got going by changing the http stuff in L_ZWay2.lua --> function ZWayAPI to the equivalent MQTT commands.

With this openLuup function:

-- send a data packet --> /ZWaveAPI/Run/SendData
local function zwsend (id, data)

I think you can use this: Send a custom command to a node

Here is an example. From another post: "to get the payload, you can snoop on it using the Debug option in the ZWave JS to MQTT interface" UI.

Also the outgoing & incoming json payloads would have to be reinterpreted to suit the identifiers used, etc.

I'm not sure how events, such as button pushes etc, are handled. Needs further investigation.

Very minor point: many of the ZWay plugin files have trailing whitespace - would be good to strip them off. eg Notepad ++, Edit --> Blank operations.

Clearly I need to learn more but regardless of the above I can experiment further as needed.

[akbooer]

Um, OK, well… looks like you’ve written most of it already.

[a-lurker]

OK the "plus" wildcards are, as tested here; operational. Great stuff.

I can do this with Z-Wave JS with the correct outcome:

  local topic   = "zwave/4/38/0/targetValue/set"
  local message = "0"
  mqtt.publish (topic, message)

or

  local topic   = "zwave/4/38/0/targetValue/set"
  local message = '{"value":99}'
  mqtt.publish (topic, message)

These also work OK:

-- setNodeName
local topic = "zwave/_CLIENTS/ZWAVE_GATEWAY-zwave-js-ui/api/setNodeName/set"
local message = '{ "args": [4, "New device Chandelier"] }'
mqtt.publish (topic, message)

Reply: message received on
zwave/_CLIENTS/ZWAVE_GATEWAY-zwave-js-ui/api/setNodeName
{"success":true,"message":"Success zwave api call","result":true,"args":[4,"New device Chandelier"],"origin":{"args":[4,"New device Chandelier"]}}

-- setNodeLocation
local topic = "zwave/_CLIENTS/ZWAVE_GATEWAY-zwave-js-ui/api/setNodeLocation/set"
local message = '{ "args": [4, "New room Laundry"] }'
mqtt.publish (topic, message)

Reply: message received on
zwave/_CLIENTS/ZWAVE_GATEWAY-zwave-js-ui/api/setNodeLocation
{"success":true,"message":"Success zwave api call","result":true,"args":[4,"New room Laundry"],"origin":{"args":[4,"New room Laundry"]}}

getNodes and getInfo are not particularly helpful:

-- getNodes
local topic = "zwave/_CLIENTS/ZWAVE_GATEWAY-zwave-js-ui/api/getNodes/set"
local message = '{ "args": [] }'

REPLIES with a torrent of info.

-- getInfo
local topic = "zwave/_CLIENTS/ZWAVE_GATEWAY-zwave-js-ui/api/getInfo/set"
local message = '{ "args": [] }'

REPLIES with info about the server.

If you want info about a node, you need to subscribe to it, along the lines of :

zwave/4/# ???

ie it's event driven. You send a command and get a response outcome. You don't request node info.

Not sure about sensors ( I have none). However, as I understand it, you request a report?

Good steps in the right direction.

[toggledbits]

ZWave centers around values. In effect, everything is a value within a command class and endpoint on a node. The paradigm is thinly visible in the old Vera firmware, and more clearly visible in the new Ezlo stuff. On a simple switch, for example, the Binary Switch (37) command class has a targetValue boolean; you set it to non-zero and it commands the switch on; set it zero and it commands the switch off. The currentValue value/property in the same CC should reflect the state of the switch as the device updates itself. When you set the targetValue, you should expect to receive events for value changed for targetValue and currentValue, and possibly other values in other CCs depending on the device type and behaviors.

Sensors behave in a similar way, it's just that their state is generally read-only (i.e. they tell you current value, but there's no target value to modify). Where things get hard for sensors in particular, but also for all ZWave devices generally, is that the command classes have evolved over time, and there's a lot of "flexibility" in how manufacturers implement them and which they choose to use. For example, a sensor may use the Basic (32) command class to show its tripped/untripped state, while a different sensor or even the same brand/model but later firmware/generation of the same sensor will instead use the Binary Sensor (48) command class. Many devices will advertise both, but only one works. Some devices will even have configuration to let you choose which one you want. Within the command classes, there are different versions evolved over time, having different capabilities and behaviors. I think this is the real struggle with implementing ZWave, and it is certainly the reason Vera/Ezlo and others (myself included) have had to chase individual support/exceptions for many models of devices. The ZWave standard isn't absolute, definitive, or deterministic despite many assertions and hopes to contrary.

Oh, additionally, there's also an Alarm command class, and a Notification command class, which are frequently used to send asynchronous events for, for example, door open/close, battery low, keypad unlock, and every other type of event and changing condition you can imagine. These are often transient, and in order to get the event, you have to receive the event the moment it's delivered -- unlike values on other command classes, you can't always poll them later to see what event you may have missed.

The upshot is, every device is going to be a little different when you are working at this (low) level.

[a-lurker]

@toggledbits Thanks for that. Zensys had it as their proprietary stuff ie under their control for ages and it's still a constellation of confusion. For example - how hard should it be to copy a set up from one USB stick to another? They really shot themselves in the foot a long time ago. Meanwhile Zigbee seems to have flourished?