Migrating Zwave & ZigBee stuff to MQTT - guidance needed.
-
” Looks like api.lua handles this somehow in the meta verse:”
Yes, FYI the new openLuup API maps device services and variables directly into the plug-in context. There are shorthand names for the otherwise unwieldy serviceIds:
API[devNo][serviceId][varname] = value -- sets the value value = API[devNo][serviceId][varname] -- gets the value
These are all standard Lua (metatable) structures, so you can write:
D = API[devNo] -- the device S = D[serviceId] -- the service S[varname] = value -- sets the value value = S[varname] -- gets the value kWh = D.energy.KWH -- get device energy usage
It’s still evolving, but this much is stable. It’s so neat and readable compared to the usual luup.variable_set/get() calls, and I’m now using it all the time in new code development (since I’ll never be writing another Vera-compatible plug-in.)
-
@akbooer
This is all going nicely - but time will tell how reliable the system as whole would be. Have added in one of these Aqara buttons and it work fine.Zigbee2mqtt "interviews" a device while it's being paired. On occasion the interview is partially completed ie it fails and some variables are void or invalid but "Interview completed" is still set to true.
With such a faulty pairing and with openLuup being subjected to same, more often or not, the function "infer_device_type" will default to "D_GenericZigbeeDevice.xml", of which the actual file doesn't seem to exist. So one occasion I observed this error, which appears to be related:
openLuup.context_switch:: ERROR: [dev #0] ./openLuup/L_Zigbee2MQTTBridge.lua:438: attempt to call field '?' (a nil value) openLuup.mqtt:: ERROR publishing application message for mqtt:zigbee2mqtt/0x00158d000706951c : ./openLuup/L_Zigbee2MQTTBridge.lua:438: attempt to call field '?' (a nil value)
You have to repair the device, delete the "Generic" device in AltUI and then restart the zigbee2mqtt server to get the correct device to show in AltUI.
Inference: Looking at the function "infer_device_type: the code IMO should be split, so "switch" and "action" produce different devices ie an on/off switch or a scene controller:
In GitHub: elseif exp.switch or exp.action then
The nomenclature is bit tricky: switch is often conflated with power relay switches and say a scene controller switch. In Zigbee2MQTT speak, a "switch" is like a relay (mechanical or solid state) and a scene controller switch is considered something that has an "action".
Composite devices will become an issue and hopefully that can be handled in the future (yes can of worms), especially dual outlet power sockets or multi gang switches, which are ubiquitous.
Here is an example of sometime that doesn't fit the mold. These Aqara devices logically act as both a switch and a scene controller. Question arises if you do say a "double" click, what happens to the switch status. Likewise for say "hold_release". There are very few of these devices and they are nearly all made by Xiaomi.
-
akbooerreplied to a-lurker on Dec 6, 2022, 11:14 AM last edited by akbooer Dec 6, 2022, 6:18 AM
@a-lurker said in Migrating Zwave & ZigBee stuff to MQTT - guidance needed.:
time will tell how reliable the system as whole would be
Is there some symptomatic problem I should know about?
@a-lurker said in Migrating Zwave & ZigBee stuff to MQTT - guidance needed.:
"D_GenericZigbeeDevice.xml", of which the actual file doesn't seem to exist
It does, in openLuup's virtual world. If you try
http://openLuupIP:3480/D_GenericZigbeeDevice.xml
you should see this in the browser:<?xml version="1.0"?> <root xmlns="urn:schemas-upnp-org:device-1-0"> <specVersion> <major>1</major> <minor>0</minor> <minimus>auto-generated</minimus></specVersion> <device> <staticJson>D_GenericZigbeeDevice.json</staticJson> <deviceType>GenericZigbeeDevice</deviceType></device></root>
I'll fix the error and review the infer_device_type() function. It was always going to evolve anyway as different device needs arise.
I have tended to represent any sort of multi-action switch as a scene controller. There will always be difficult cases (sometimes resolved by using vendor/manufacturer) but I have shied away from an extensive configuration database. Plausible, I suppose, to pull this from the web...?
I rely on Philips Hue for the few Zigbee devices that I have, so probably won't get to road test this for myself. You're going to want to do the whole thing again for Zwave JS ?
-
@akbooer said in Migrating Zwave & ZigBee stuff to MQTT - guidance needed.:
Is there some symptomatic problem I should know about?
I don't think so but who wants to set up heaps of devices in a new set up when everything has been going well for years with just openLuup and an old Vera war horse!
Plausible, I suppose, to pull this from the web...?
Don't need to get to complicated too soon, as you just about have it all covered but I'm still thinking of one of these Hue Smart plugs. It's just a switch, as opposed to a scene controller.
I rely on Philips Hue for the few Zigbee devices that I have
Arrhh you know you want to ditch that hub.
You're going to want to do the whole thing again for Zwave JS ?
Maybe after a cup of tea and good lie down. And yourself?
-
@a-lurker said in Migrating Zwave & ZigBee stuff to MQTT - guidance needed.:
Maybe after a cup of tea and good lie down. And yourself?
Definitely the tea. Possibly the lie down.
You'll have to educate me about the protocol, yet again, but it should almost be a dead-ringer for the Zigbee ??
-
Release development v22.12.6 separating switch and controller devices, and fixing nil function pointer.
-
@akbooer said in Migrating Zwave & ZigBee stuff to MQTT - guidance needed.:
Release development v22.12.6 separating switch and controller devices, and fixing nil function pointer
Thanks for that - looks OK now.
Have now got all my Hue stuff running with no Hue hub - that's good. A few complications here and there.
Main complication:
Conditions:
z2m is running and openLuup is running.
Then pair new device: in this case this Philips light bulb with z2m.
z2m logs this to the terminal. I've prettied the json section so you can read it:Zigbee2MQTT:info 2022-12-10 10:40:28: Device '0x001788010363fcbf' joined Zigbee2MQTT:info 2022-12-10 10:40:28: MQTT publish: topic 'zigbee2mqtt/bridge/event', payload '{"data":{"friendly_name":"0x001788010363fcbf","ieee_address":"0x001788010363fcbf"},"type":"device_joined"}' Zigbee2MQTT:info 2022-12-10 10:40:28: Starting interview of '0x001788010363fcbf' Zigbee2MQTT:info 2022-12-10 10:40:28: MQTT publish: topic 'zigbee2mqtt/bridge/event', payload '{"data":{"friendly_name":"0x001788010363fcbf","ieee_address":"0x001788010363fcbf","status":"started"},"type":"device_interview"}' Zigbee2MQTT:info 2022-12-10 10:40:29: Successfully interviewed '0x001788010363fcbf', device has successfully been paired Zigbee2MQTT:info 2022-12-10 10:40:29: Device '0x001788010363fcbf' is supported, identified as: Philips Hue White A60 Single bulb E27/B22 (8718696449691) Zigbee2MQTT:info 2022-12-10 10:40:29: MQTT publish: topic 'zigbee2mqtt/bridge/event', payload ' { "data": { "definition": { "description": "Hue White A60 Single bulb E27/B22", "exposes": [{ "features": [{ "access": 7, "description": "On/off state of this light", "name": "state", "property": "state", "type": "binary", "value_off": "OFF", "value_on": "ON", "value_toggle": "TOGGLE" }, { "access": 7, "description": "Brightness of this light", "name": "brightness", "property": "brightness", "type": "numeric", "value_max": 254, "value_min": 0 }], "type": "light" }, { "access": 2, "description": "Triggers an effect on the light (e.g. make light blink for a few seconds)", "name": "effect", "property": "effect", "type": "enum", "values": ["blink", "breathe", "okay", "channel_change", "finish_effect", "stop_effect"] }, { "access": 1, "description": "Link quality (signal strength)", "name": "linkquality", "property": "linkquality", "type": "numeric", "unit": "lqi", "value_max": 255, "value_min": 0 }], "model": "8718696449691", "options": [{ "access": 2, "description": "Controls the transition time (in seconds) of on/off, brightness, color temperature (if applicable) and color (if applicable) changes. Defaults to `0` (no transition).", "name": "transition", "property": "transition", "type": "numeric", "value_min": 0 }], "supports_ota": true, "vendor": "Philips" }, "friendly_name": "0x001788010363fcbf", "ieee_address": "0x001788010363fcbf", "status": "successful", "supported": true }, "type": "device_interview" } ' Zigbee2MQTT:info 2022-12-10 10:40:29: Configuring '0x001788010363fcbf' Zigbee2MQTT:info 2022-12-10 10:40:29: Successfully configured '0x001788010363fcbf'
Looks like openLuup then responds to the topic zigbee2mqtt/bridge/event ?? and creates a Generic device. The openLuup log doesn't make it clear what topic it's responding to when it creates a device but you can see in the log that it has found a new device:
2022-12-10 10:40:28.568 openLuup.mqtt:: PUBLISH tcp{client}: 0x3a7b670 2022-12-10 10:40:28.571 luup.zigbee2mqtt:0: Topic ignored : zigbee2mqtt : / : bridge/logging 2022-12-10 10:40:28.574 openLuup.mqtt:: PUBLISH tcp{client}: 0x3a7b670 2022-12-10 10:40:28.591 luup.zigbee2mqtt:0: New Zigbee detected: 0x001788010363fcbf 2022-12-10 10:40:28.594 luup.create_device:: [20010] D_GenericZigbeeDevice.xml / / D_GenericZigbeeDevice.json (GenericZigbeeDevice) 2022-12-10 10:40:28.602 openLuup.mqtt:: PUBLISH tcp{client}: 0x3a7b670 2022-12-10 10:40:28.603 luup.zigbee2mqtt:0: Topic ignored : zigbee2mqtt : / : bridge/logging 2022-12-10 10:40:28.605 openLuup.mqtt:: PUBLISH tcp{client}: 0x3a7b670 2022-12-10 10:40:28.605 luup.zigbee2mqtt:0: Topic ignored : zigbee2mqtt : / : bridge/event 2022-12-10 10:40:28.607 openLuup.mqtt:: PUBLISH tcp{client}: 0x3a7b670 2022-12-10 10:40:28.607 luup.zigbee2mqtt:0: Topic ignored : zigbee2mqtt : / : bridge/logging 2022-12-10 10:40:28.609 openLuup.mqtt:: PUBLISH tcp{client}: 0x3a7b670 2022-12-10 10:40:28.626 openLuup.mqtt:: PUBLISH tcp{client}: 0x3a7b670 2022-12-10 10:40:28.626 luup.zigbee2mqtt:0: Topic ignored : zigbee2mqtt : / : bridge/logging 2022-12-10 10:40:28.628 openLuup.mqtt:: PUBLISH tcp{client}: 0x3a7b670 2022-12-10 10:40:28.628 luup.zigbee2mqtt:0: Topic ignored : zigbee2mqtt : / : bridge/event 2022-12-10 10:40:28.754 openLuup.server:: request completed (11333 bytes, 1 chunks, 2261 ms) tcp{client}: 0x43d0070 2022-12-10 10:40:28.766 openLuup.server:: GET /data_request?id=user_data&output_format=json&DataVersion=564556066&_=1670559272961 HTTP/1.1 tcp{client}: 0x43d0070 2022-12-10 10:40:29.987 openLuup.server:: request completed (1723523 bytes, 108 chunks, 1220 ms) tcp{client}: 0x43d0070 2022-12-10 10:40:30.017 openLuup.mqtt:: PUBLISH tcp{client}: 0x3a7b670 2022-12-10 10:40:30.017 luup.zigbee2mqtt:0: Topic ignored : zigbee2mqtt : / : bridge/logging 2022-12-10 10:40:30.018 openLuup.mqtt:: PUBLISH tcp{client}: 0x3a7b670 2022-12-10 10:40:30.018 luup.zigbee2mqtt:0: Topic ignored : zigbee2mqtt : / : bridge/logging 2022-12-10 10:40:30.020 openLuup.mqtt:: PUBLISH tcp{client}: 0x3a7b670 2022-12-10 10:40:30.032 openLuup.mqtt:: PUBLISH tcp{client}: 0x3a7b670 2022-12-10 10:40:30.033 luup.zigbee2mqtt:0: Topic ignored : zigbee2mqtt : / : bridge/logging 2022-12-10 10:40:30.034 openLuup.mqtt:: PUBLISH tcp{client}: 0x3a7b670 2022-12-10 10:40:30.035 luup.zigbee2mqtt:0: Topic ignored : zigbee2mqtt : / : bridge/event 2022-12-10 10:40:30.036 openLuup.mqtt:: PUBLISH tcp{client}: 0x3a7b670 2022-12-10 10:40:30.036 luup.zigbee2mqtt:0: Topic ignored : zigbee2mqtt : / : bridge/logging 2022-12-10 10:40:30.037 openLuup.mqtt:: PUBLISH tcp{client}: 0x3a7b670 2022-12-10 10:40:30.037 luup.zigbee2mqtt:0: Topic ignored : zigbee2mqtt : / : bridge/logging 2022-12-10 10:40:30.039 openLuup.mqtt:: PUBLISH tcp{client}: 0x3a7b670 2022-12-10 10:40:30.048 openLuup.mqtt:: PUBLISH tcp{client}: 0x3a7b670 2022-12-10 10:40:30.052 luup.zigbee2mqtt:0: Topic ignored : zigbee2mqtt : / : bridge/info
So it looks as the event returns the same data as "zigbee2mqtt/bridge/devices" does, as each new device is paired but it's parent json is a little different. The device definition is below "data" in the json:
"data":
With type set as follows:
type="device_interview",
and this must indicate success:
data.status ="successful"
Hope that's clear - bit hard to explain.
Suffice to say I added a pile of stuff and ended up with a whole pile of generic devices, which I had to remove one by one with a openLuup restart for each. Got a bit ambitious!
However; it looks like the ability to handle the addition of new devices on the fly without having to restart thez2m server each time to get the device definitions, would certainly be possible and a welcome addition.
Looks like you just look for the event topic with {"type":"device_interview" and {"data": status":"successful".
The json contains definition.expose.type = light so the function infer_device_type() should still do its work OK as it currently does.
I can't test any new code on this as I have now added all my devices. Although I could unpair just a light bulb and repair it and see what happens.
-
a-lurkerreplied to a-lurker on Dec 11, 2022, 8:06 AM last edited by a-lurker Dec 11, 2022, 3:11 AM
A slight in convenience:
Conditions:
Having just added a scene controller to z2m and restarted the z2m server and openLuup has dutifully added it in and here it is:Great; so first thing to do next, is to link the scene controller to a scene: So create a new scene, dial up the scene controller and put a watch on to create a trigger:
LastSceneTime (new~=old)
Problem is LastSceneTime is nowhere to be found - bit disconcerting at this point. In order to proceed, you have to push a button on the scene controller to get the variable to be created. Suggest the variable is created when the scene controller is first added to openLuup and set its time to Unix year zero.
-
A slight in convenience: no way to set color on the light bulbs. X,Y color space seems better than RGB as it allows you to specify white color temperature.
-
This vibration detection device and other similar devices should be "inferred" as a motion sensor. They expose "contact".
Perhaps infer_device_type() could use some look up table or look at json string that could perhaps be later saved to a file that the user could fiddle with? eg some vague pesudo code. Whatever you figure would work.
local deviceType = nil if (lookup.exposes.contact) then deviceType = lookup.exposes.contact end
-
akbooerreplied to a-lurker on Dec 11, 2022, 10:41 AM last edited by akbooer Dec 11, 2022, 5:58 AM
@a-lurker said in Migrating Zwave & ZigBee stuff to MQTT - guidance needed.:
Looks like openLuup then responds to the topic zigbee2mqtt/bridge/event ?? and creates a Generic device.
Can't see how that would happen. The bridge code is clear that anything other than bridge/devices is ignored:
local function handle_bridge_topics (topic, message) local subtopic = topic: match "^bridge/(.+)" if subtopic: match "^devices" then create_devices (message) else ignore_topic (topic) end end
and it should write Topic ignored... in the log. If it creates a device, it must be receiving the actual /devices topics. I take it that your "info" log messages are your own additions to the code. I could certainly make it respond to a /event topic. The Shelly and Tasmota bridges already create new devices on the fly, because they send their own configurations when they connect to MQTT.
-
@a-lurker said in Migrating Zwave & ZigBee stuff to MQTT - guidance needed.:
Problem is LastSceneTime is nowhere to be found - bit disconcerting at this point. In order to proceed, you have to push a button on the scene controller to get the variable to be created.
Yes, that's true. The root cause is actually that the service file for a scene controller S_SceneController1.xml does not contain a definition of that variable, or a default vaule for it. I could force this at device creation, but you have found a work-around alreay.
-
@a-lurker said in Migrating Zwave & ZigBee stuff to MQTT - guidance needed.:
A slight in convenience: no way to set color on the light bulbs. X,Y color space seems better than RGB as it allows you to specify white color temperature.
Some investigation required here. The existing Color service definition defines three actions of interest:
- setColor
- setColorRGB
- setColorTemperature
It's not clear from the service file definition what the possible value/syntax is for any of these commands, so that would mean delving into the UPnP documentation, which, previously, I've found to be an absolute nightmare (as are most international standard definitions.) Perhaps csomeone here can aleady tell us the answer?
-
@a-lurker said in Migrating Zwave & ZigBee stuff to MQTT - guidance needed.:
This vibration detection device and other similar devices should be "inferred" as a motion sensor. They expose "contact".
Do you have the bridge/devices JSON for that device?
-
@akbooer said in Migrating Zwave & ZigBee stuff to MQTT - guidance needed.:
The root cause is actually that the service file for a scene controller S_SceneController1.xml does not contain a definition of that variable, or a default vaule for it. I could force this at device creation
...and I have just done so in development v22.12.11.
-
@akbooer said in Migrating Zwave & ZigBee stuff to MQTT - guidance needed.:
Do you have the bridge/devices JSON for that device?
No but I would see it as no different to checking "occupancy" in the motion sensor device.
-
@akbooer said in Migrating Zwave & ZigBee stuff to MQTT - guidance needed.:
I take it that your "info" log messages are your own additions to the code.
The two logs are directly as they were spat out, except for uncollapsing of the json as mentioned above. I made no changes.
Yes - this is very odd - I can't see how the code would respond to anything else but the bridges/devices topic either. But certainly it appears to have done same; but unsuccessfully as all devices ended up being considered Generic. I may have to try repairing a bulb with more debug code.
Moving forward - responding to an /event topic would be good.
-
These messages:
Zigbee2MQTT:info 2022-12-10 10:40:28: MQTT publish: topic 'zigbee2mqtt/bridge/event', payload '{"data":{"friendly_name":"0x001788010363fcbf","ieee_address":"0x001788010363fcbf"},"type":"device_joined"}' Zigbee2MQTT:info 2022-12-10 10:40:28: Starting interview of '0x001788010363fcbf' Zigbee2MQTT:info 2022-12-10 10:40:28: MQTT publish: topic 'zigbee2mqtt/bridge/event', payload '{"data":{"friendly_name":"0x001788010363fcbf","ieee_address":"0x001788010363fcbf","status":"started"},"type":"device_interview"}' Zigbee2MQTT:info 2022-12-10 10:40:29: Successfully interviewed '0x001788010363fcbf', device has successfully been paired Zigbee2MQTT:info 2022-12-10 10:40:29: Device '0x001788010363fcbf' is supported, identified as: Philips Hue White A60 Single bulb E27/B22 (8718696449691)
…don’t come from my code. You mentioned that this was printed out to the console… something else must be doing this too.
-
@akbooer said in Migrating Zwave & ZigBee stuff to MQTT - guidance needed.:
Some investigation required here. The existing Color service definition defines three actions of interest:
setColor
setColorRGB
setColorTemperatureIt's not clear from the service file definition what the possible value/syntax is for any of these commands, so that would mean delving into the UPnP documentation, which, previously, I've found to be an absolute nightmare (as are most international standard definitions.) Perhaps csomeone here can aleady tell us the answer?
I've done a lot of work on the color side of thing in the Vera, and my code used to work on openluup as well. Feel free to take inspiration from https://github.com/dbochicchio/vera-VirtualDevices/blob/master/L_VirtualRGBW1.lua
From line 134 you'll find all the formats that luup is supporting, at least on Vera's side. There's also W/D, for warn and cool white.
56/73