Migrating Zwave & ZigBee stuff to MQTT - guidance needed.
-
Currently I have a Vera and Hue hub all reliably controlled by openLuup with AltUI, plus any number of plugins. Been working really well for a few years now. However would like to head for a more MQTT based set up. Eliminate the Hue hub and hopefully eliminate Vera by using ZWAVE JS UI. Noting that Zwavejs2mqtt has been renamed to Z-Wave JS UI. Probably also run the stuff using Docker. Just because. Everything would end up on the one computer for easier management. Erhhh that's the hope.
Some of the new Zigbee Aqara stuff is very good and inexpensive plus it fits in with HomeKit. Also the Aqara battery powered stuff looks to have a good battery lifetime: ie suggested up to five years. The battery operated Hue buttons I have; have lasted for ages. Would like to use zigbee2mqtt with a SonOff dongle, which would allow access to the over two and half thousand devices zigbee2mqtt now supports:
AK has the MQTT stuff working in openLuup. Have played around with it and it works well, as one would expect. Love the UDP to MQTT code.
Shellys are great and also very inexpensive and they spit out & accept MQTT but I would prefer to stay away from WiFi. Not meshed and higher power consumption. Horses for courses.
Now here's the query:
Got about forty or more ZWave twin light switches, plus a few other ZWave bits & pieces such as blind controllers. Then there are the Hue devices on top of that. That's a lot of virtual devices to set up in openLuup. What's an appropriate way to do this?It seems there is no "auto magic bridge set up". Do I need to use say @therealdb's Virtual Devices plugin that supports MQTT or is there some other approach?
I have to confess I still don't understand the master child approach in that plugin. Seems one light switch would have all the other light switches hanging off it? Helps Vera but not a problem with openLuup - why is that? Suspect AK's good coding beats Vera's?
GitHub - dbochicchio/vera-VirtualDevices: Virtual HTTP Devices plug-in for Vera and openLuup
Virtual HTTP Devices plug-in for Vera and openLuup - dbochicchio/vera-VirtualDevices
Setting up manually say 100 virtual devices is a bit much to ask. I had a look at hacking the user_data.json file. Good approach till you see all the UIDs and the individually numbered ControlURL and EventURLs that need to be set up.
I need some way of say of creating about 80 light switches in "No room" or in say the "ZWave upgrade" room. Or say some sort of code that could go through all my existing bridged ZWAVE devices in openLuup and create virtual devices for each one. I caould then use the openLuup console to name them and place them in their rooms:
openLuup_IP_address:3480/console?page=devices_table
At that point I could hack the user_data.json file to insert the MQTT topics fairly easily for each? Plus any other fine tuning needed.
Then the old ZWave stuff could be swapped over to ZWAVE JS UI and all the virtual MQTT devices would be ready to go or am I dreaming? Then delete all the old Vera bridged stuff. I'm not too fussed about scene code and the like, as a I have all my code in one block, that is set up in the openLuup start up.
It seems that with ZWay you can create all the ZWave device by doing some sort of interrogation of ZWay's API? Seems also to be the case with the Shelleys?
So any ideas, suggestions or code snippets are welcome on how to move towards MQTT and in particular ZWAVE JS UI and zigbee2mqtt.
I'm in no hurry as openLuup is performing nicely, with the old Vera handling all my ZWave devices.
-
@a-lurker said in Migrating Zwave & ZigBee stuff to MQTT - guidance needed.:
I have to confess I still don't understand the master child approach in that plugin. Seems one light switch would have all the other light switches hanging off it? Helps Vera but not a problem with openLuup - why is that? Suspect AK's good coding beats Vera's?
It's to better manage memory under Vera: each plug-in is allocating its own memory, so with master/child I could re-use some for the very same basic things. in openLuup this isn't needed, since memory is managed differently.
I'm in no hurry as openLuup is performing nicely, with the old Vera handling all my ZWave devices.
I still have a couple of virtual devices in Vera, but mainly because of dashboard/the native app. Right now, many of them are simply calling Reactor's endpoint, where I've mapped the real devices (thru HASS or MQTT or whatever virtual device I have mapped).
I have no plans to dismiss my Vera, since (as a glorified Zwave controller) it is performing well for me.The migration you've in mind is not easy, since you'll get completely different device ID/types under zwavejs2mqtt. I know this is not the answer you're looking for, but I'll start over again, with a blank system and code/logic ported to new IDs.
-
@a-lurker said in Migrating Zwave & ZigBee stuff to MQTT - guidance needed.:
It seems there is no "auto magic bridge set up".
I need to spend a bit of time digesting your post, but for both Shellies and Tasmota devices I did create a "magic bridge setup", wherein there is no install or configuration required. The bridge self-installs if openLuup sees one of those devices (over MQTT) and creates a corresponding virtual device.
I could do the same for other types of device, but I'd need to understand the protocol that your environment was providing.
-
@akbooer said in Migrating Zwave & ZigBee stuff to MQTT - guidance needed.:
I did create a "magic bridge setup"
Is that available to mortals?
C
-
For Shellies and Tasmota, that's built in already...
-
Oooh now there's interesting.....
<dumb question> How is OpenLuup communicating with MQTT? I can't see anything in my config
C
-
@catmanv2 said in Migrating Zwave & ZigBee stuff to MQTT - guidance needed.:
How is OpenLuup communicating with MQTT? I can't see anything in my config
openLuup has its own built-in MQTT server:
https://smarthome.community/topic/20/openluup-version-log/13
-
Ooooooooooh!
Is there any reason I shouldn't run both this and Mosquitto? Apart from the inelegance?
Is there any reason not to kill mosquitto and just point everything at this?
Cheers
C
-
You can run both. You could even bridge to brokers. The openLuup broker only supports QoS 0, but this is fine for this purpose.
Give it a try. Doesn’t yet support Shelly Plus (NG) devices.
-
I intend to! As soon as I get some time. Last dumb q: Upgrading is simply downloading the openLuup_install.lua file and running it in the correct location?
Nothing else need to worry about? I'm currently on
2021.04.26
Would seem a sensible time to get up to 2021.07/25?Cheers!
C
-
Just go to this URL and hit the upgrade (blue arrows) upgrade button. If feeling brave, type the word development into the box beforehand and get the latest and greatest:
http://openLuup_IP_address:3480/console?page=plugins_table
I think my thread has been hijacked!
-
@a-lurker said in Migrating Zwave & ZigBee stuff to MQTT - guidance needed.:
I think my thread has been hijacked!
I am SO sorry
I will shut up now, I promise!
C
-
a-lurkerreplied to akbooer on Nov 26, 2022, 8:35 AM last edited by a-lurker Nov 26, 2022, 9:41 PM
@akbooer said in Migrating Zwave & ZigBee stuff to MQTT - guidance needed.:
The bridge self-installs if openLuup sees one of those devices (over MQTT) and creates a corresponding virtual device.
I can see the complexity of doing this for zigbee2mqtt as there are so many devices, many of which are composites. So for example these motion detectors:
which are tiny and also have high claimed battery life, have all sorts of properties in their payload - however it's fundamentally a motion sensor:
{ "battery": 100, "detection_interval": 30, "device_temperature": 22, "illuminance": 64, "linkquality": 255, "motion_sensitivity": "medium", "occupancy": false, "power_outage_count": 0, "trigger_indicator": false, "voltage": 3215 }Home Assistant has similar problems, where there are so many properties to perhaps make use of and/or handle.
I'm thinking that with zigbee2mqtt and also ZWAVE JS UI you could possibly do OK with just handling a few types of devices. I would suggest just:
- Relays
- Dimmable and colored lights
- Blinds?
Other stuff could probably be handled with code and the Virtual HTTP Devices plug-in.
Out of interest; the zigbee2mqtt "Bridge" churns out stuff like this for each zigbee device:
[ { "definition": null, "endpoints": { "1": { "bindings": [], "clusters": { "input": [ "genBasic", "genIdentify", "genOnOff", "genTime", "genOta", "26", "lightingColorCtrl" ], "output": [ "genBasic", "genIdentify", "genGroups", "genScenes", "genOnOff", "genLevelCtrl", "genPollCtrl", "lightingColorCtrl", "msIlluminanceMeasurement", "msTemperatureMeasurement", "msRelativeHumidity", "msOccupancySensing", "ssIasZone", "haMeterIdentification", "haApplianceStatistics", "haElectricalMeasurement", "seMetering", "touchlink", "manuSpecificUbisysDimmerSetup", "manuSpecificSamsungAccelerometer" ] }, "configured_reportings": [], "scenes": [] }, "242": { "bindings": [], "clusters": { "input": [], "output": [ "greenPower" ] }, "configured_reportings": [], "scenes": [] } }, "friendly_name": "Coordinator", "ieee_address": "0xredacted", "interview_completed": true, "interviewing": false, "network_address": 0, "supported": false, "type": "Coordinator" }, { "date_code": "Dec 7 2021", "definition": { "description": "Aqara P1 human body movement and illuminance sensor", "exposes": [ { "access": 1, "description": "Indicates whether the device detected occupancy", "name": "occupancy", "property": "occupancy", "type": "binary", "value_off": false, "value_on": true }, { "access": 1, "description": "Measured illuminance in lux", "name": "illuminance_lux", "property": "illuminance", "type": "numeric", "unit": "lx" }, { "access": 1, "description": "Measured illuminance in lux", "name": "illuminance", "property": "illuminance", "type": "numeric", "unit": "lx" }, { "access": 7, "description": ". Press pairing button right before changing this otherwise it will fail.", "name": "motion_sensitivity", "property": "motion_sensitivity", "type": "enum", "values": [ "low", "medium", "high" ] }, { "access": 7, "description": "Time interval for detecting actions. Press pairing button right before changing this otherwise it will fail.", "name": "detection_interval", "property": "detection_interval", "type": "numeric", "unit": "s", "value_max": 65535, "value_min": 2 }, { "access": 7, "description": "When this option is enabled then blue LED will blink once when motion is detected. Press pairing button right before changing this otherwise it will fail.", "name": "trigger_indicator", "property": "trigger_indicator", "type": "binary", "value_off": false, "value_on": true }, { "access": 1, "description": "Temperature of the device", "name": "device_temperature", "property": "device_temperature", "type": "numeric", "unit": "°C" }, { "access": 1, "description": "Remaining battery in %", "name": "battery", "property": "battery", "type": "numeric", "unit": "%", "value_max": 100, "value_min": 0 }, { "access": 1, "description": "Voltage of the battery in millivolts", "name": "voltage", "property": "voltage", "type": "numeric", "unit": "mV" }, { "access": 1, "description": "Link quality (signal strength)", "name": "linkquality", "property": "linkquality", "type": "numeric", "unit": "lqi", "value_max": 255, "value_min": 0 } ], "model": "RTCGQ14LM", "options": [ { "access": 2, "description": "Time in seconds after which occupancy is cleared after detecting it (default is \"detection_interval\" + 2 seconds). The value must be equal to or greater than \"detection_interval\", and it can also be a fraction.", "name": "occupancy_timeout", "property": "occupancy_timeout", "type": "numeric", "unit": "s", "value_min": 0, "value_step": 0.1 }, { "access": 2, "description": "Sends a message the last time occupancy (occupancy: true) was detected. When setting this for example to [10, 60] a `{\"no_occupancy_since\": 10}` will be send after 10 seconds and a `{\"no_occupancy_since\": 60}` after 60 seconds.", "item_type": "number", "name": "no_occupancy_since", "property": "no_occupancy_since", "type": "list" }, { "access": 2, "description": "Calibrates the illuminance value (percentual offset), takes into effect on next report of device.", "name": "illuminance_calibration", "property": "illuminance_calibration", "type": "numeric" }, { "access": 2, "description": "Calibrates the device_temperature value (absolute offset), takes into effect on next report of device.", "name": "device_temperature_calibration", "property": "device_temperature_calibration", "type": "numeric" } ], "supports_ota": true, "vendor": "Xiaomi" }, "endpoints": { "1": { "bindings": [], "clusters": { "input": [ "genBasic", "genPowerCfg", "genIdentify", "aqaraOpple" ], "output": [ "genIdentify", "genOta", "aqaraOpple" ] }, "configured_reportings": [], "scenes": [] } }, "friendly_name": "hello/testing", "ieee_address": "0x01234567890abcde", "interview_completed": true, "interviewing": false, "manufacturer": "LUMI", "model_id": "lumi.motion.ac02", "network_address": 32549, "power_source": "Battery", "software_build_id": "0.0.0_0005", "supported": true, "type": "EndDevice" } ]However getting back to the migration issue; I could probably make use of the Virtual HTTP Devices plug-in but as previously mentioned, I may need up to 80 devices to be created.
How do you "Create" a device programmatically? Looks like the MIOS documentation Wiki is being abandoned; otherwise I would scour through that for the answer. Assume you need to post D_zyz.xml and I_xyz.xml file names, etc to somewhere?
-
Look at this:
vera-OpenSprinkler/L_VeraOpenSprinkler1.lua at master · dbochicchio/vera-OpenSprinkler
Completely new and rewritten plug-in to interface an OpenSprinkler to a Vera or openLuup system. - dbochicchio/vera-OpenSprinkler
You'll need to append them and then sync
vera-OpenSprinkler/L_VeraOpenSprinkler1.lua at master · dbochicchio/vera-OpenSprinkler
Completely new and rewritten plug-in to interface an OpenSprinkler to a Vera or openLuup system. - dbochicchio/vera-OpenSprinkler
Honestly, I've dumped all my luup code from production, so my skills are definitely at the lower end and the wiki has gone.
-
@a-lurker said in Migrating Zwave & ZigBee stuff to MQTT - guidance needed.:
I'm thinking that with zigbee2mqtt and also ZWAVE JS UI you could possibly do OK with just handling a few types of devices. I would suggest just:
Relays
Dimmable and colored lights
Blinds?Other stuff could probably be handled with code and the Virtual HTTP Devices plug-in.
Yes, that's the sort of thing I had in mind. Both Shelly and Tasmota bridges are similarly constrained.
What topics do zigbee2mqtt and ZWAVE JS UI use?
Shelly currently subsribes to "shellies/#", and Tasmota to "tele/#", and " tasmota/tele/#".
-
a-lurkerreplied to akbooer on Nov 27, 2022, 2:40 AM last edited by a-lurker Nov 27, 2022, 2:42 AM
I'm still delving into this, so my current knowledge is a little on the short side and the following is a little generalized. However:
It seems that battery operated devices have nothing that can be controlled and their status cannot be read on demand ie no set or get possible (as indicated by an "access" sub property - see below). Presumably because they are asleep most of the time. So a battery powered Aqara motion detector publishes this only when it's triggered by motion:
"zigbee2mqtt/0x01234567890abcde" and supplying a flat json payload containing all the properties - see previous post above.
"zigbee2mqtt" begins all the topics. It can be set to something else but not sure why you would; unless you had multiple instances of zigbee2mqtt running.
0x01234567890abcde is the "ieee_address" of the device and it can be associated with a "friendly_name" of your own choice, such as "bedroom1/left_corner/motion"
So you could have topics such as:
"zigbee2mqttInstance2/bedroom1/left_corner/motion" instead of "zigbee2mqtt/0x01234567890abcde"
The bridge defines all the devices and the devices' topics:
"zigbee2mqtt/bridge/devices" with a payload, which is an array of json objects - one for each device. ieee_address and friendly_name are top level properties of each device in the array. See previous post above.
The "zigbee2mqtt/bridge/devices" topic is published when you first connect to the zigbee2mqtt server. ie you can't do a GET on it - not sure why that is. It's republished if a device joins or leaves. You can force a republish by doing: "zigbee2mqtt/bridge/request/restart" but that's a bit radical, as it does a full server restart.
Each property in the device's status report is also further described by the bridge. So a status property such as temperature has sub properties eg "access", which indicates if the property can be set or get. ie battery operated devices can't be set or get. That is to say each device has an array of property description objects.
So to set up: you would load up the array of devices from "zigbee2mqtt/bridge/devices". Then wizz through them looking for ones of interest under "definition.exposes[array]". Trying to identify what the device type is, appears to be a bit tricky. Looks like you have check the device for the presence of various property combinations or omissions.
Aqara motion detector: has "occupancy". Also has device_temperature & illuminance, which in this case is not much use, as you only get the values when motion triggers the unit.
Hue light bulb: has "light"
Hue light switch: has "action"
Gets tricky with say composite devices. But it would be good to allow some of them eg:
Ikuü Dual outlet: has "switch" with "state_left" & "state_right" but no "brightness"
Tuya five gang light switch: has: "switch" with "state_l1" to "state_l5" (but no "moving_*" properties see below).
Lots of the devices also have temperature sensors and some have energy consumption sensors.
But then you have this that helps confound identification of a device:
Dual curtain/blind controller: It has "moving_left" & "moving_right" but it also has "state_left" & "state_right" like the Ikuü Dual outlet above!
Controlling things - eg a light bulb. There are two ways:
- send a block of json as the payload
- or pure text as the payload, with words such as ON or OFF or TOGGLE
From the following link that talks about these two SET & GET:
In case you don't want to use JSON, publishing to zigbee2mqtt/[FRIENDLY_NAME]/set/state with payload ON is the same as publishing to zigbee2mqtt/[FRIENDLY_NAME]/set payload {"state": "ON"}"
Hope this is of some help.
-
Log of switch operation:
luup.call_action:: 209.urn:upnp-org:serviceId:SwitchPower1.SetTarget luup_log:209: VirtualDevices[3.0-beta7@209](actionPower@94):actionPower(209,"1") luup_log:209: VirtualDevices[3.0-beta7@209](actionPowerInternal@47):actionPowerInternal(209,true,true) luup_log:209: VirtualDevices[3.0-beta7@209](setVar@104):setVar("urn:upnp-org:serviceId:SwitchPower1","Target","1",209) old value "0" luup.variable_set:: 209.urn:upnp-org:serviceId:SwitchPower1.Target was: 0 now: 1 #hooks:0 luup_log:209: VirtualDevices[3.0-beta7@209](sendDeviceCommand@308):sendDeviceCommand("SetPowerURL","on",209)These are all empty:
MQTT_BrightnessValue
MQTT_PowerStatusOff
MQTT_PowerStatusOnThese are all set to: "http://"
SetPowerOffURL
SetPowerURL
SetToggleURL
SetUpdateMetersURL -
Zigbee supported devices page shows three column headings:
- Vendor
- Exposes
- Search
The "Exposes" pull down list is very useful as you can filter devices by what properties they expose. Useful for working on how to identify the device type. The filter allows multiple properties to be selected - allowing devices to be narrowed down further.
-
Should be able to run this code to get some idea how the type of the zigbee devices could potentially be identified:
local zigbee2mqtt_bridge_devices = [[ [{ "definition": null, "endpoints": { "1": { "bindings": [], "clusters": { "input": ["genBasic", "genIdentify", "genOnOff", "genTime", "genOta", "26", "lightingColorCtrl"], "output": ["genBasic", "genIdentify", "genGroups", "genScenes", "genOnOff", "genLevelCtrl", "genPollCtrl", "lightingColorCtrl", "msIlluminanceMeasurement", "msTemperatureMeasurement", "msRelativeHumidity", "msOccupancySensing", "ssIasZone", "haMeterIdentification", "haApplianceStatistics", "haElectricalMeasurement", "seMetering", "touchlink", "manuSpecificUbisysDimmerSetup", "manuSpecificSamsungAccelerometer"] }, "configured_reportings": [], "scenes": [] }, "242": { "bindings": [], "clusters": { "input": [], "output": ["greenPower"] }, "configured_reportings": [], "scenes": [] } }, "friendly_name": "Coordinator", "ieee_address": "0xredacted", "interview_completed": true, "interviewing": false, "network_address": 0, "supported": false, "type": "Coordinator" }, { "date_code": "Dec7 2021", "definition": { "description": "Aqara P1 human body movement and illuminance sensor", "exposes": [{ "access": 1, "description": "Indicates whether the device detected occupancy", "name": "occupancy", "property": "occupancy", "type": "binary", "value_off": false, "value_on": true }, { "access": 1, "description": "Measured illuminance in lux", "name": "illuminance_lux", "property": "illuminance", "type": "numeric", "unit": "lx" }, { "access": 1, "description": "Measured illuminance in lux", "name": "illuminance", "property": "illuminance", "type": "numeric", "unit": "lx" }, { "access": 7, "description": ". Press pairing button right before changing this otherwise it will fail.", "name": "motion_sensitivity", "property": "motion_sensitivity", "type": "enum", "values": ["low", "medium", "high"] }, { "access": 7, "description": "Time interval for detecting actions. Press pairing button right before changing this otherwise it will fail.", "name": "detection_interval", "property": "detection_interval", "type": "numeric", "unit": "s", "value_max": 65535, "value_min": 2 }, { "access": 7, "description": "When this option is enabled then blue LED will blink once when motion is detected. Press pairing button right before changing this otherwise it will fail.", "name": "trigger_indicator", "property": "trigger_indicator", "type": "binary", "value_off": false, "value_on": true }, { "access": 1, "description": "Temperature of the device", "name": "device_temperature", "property": "device_temperature", "type": "numeric", "unit": "°C" }, { "access": 1, "description": "Remaining battery in %", "name": "battery", "property": "battery", "type": "numeric", "unit": "%", "value_max": 100, "value_min": 0 }, { "access": 1, "description": "Voltage of the battery in millivolts", "name": "voltage", "property": "voltage", "type": "numeric", "unit": "mV" }, { "access": 1, "description": "Link quality (signal strength)", "name": "linkquality", "property": "linkquality", "type": "numeric", "unit": "lqi", "value_max": 255, "value_min": 0 }], "model": "RTCGQ14LM", "options": [{ "access": 2, "description": "Time in seconds after which occupancy is cleared after detecting it (default is \"detection_interval\" + 2 seconds). The value must be equal to or greater than \"detection_interval\", and it can also be a fraction.", "name": "occupancy_timeout", "property": "occupancy_timeout", "type": "numeric", "unit": "s", "value_min": 0, "value_step": 0.1 }, { "access": 2, "description": "Sends a message the last time occupancy (occupancy: true) was detected. When setting this for example to [10, 60] a `{\"no_occupancy_since\": 10}` will be send after 10 seconds and a `{\"no_occupancy_since\": 60}` after 60 seconds.", "item_type": "number", "name": "no_occupancy_since", "property": "no_occupancy_since", "type": "list" }, { "access": 2, "description": "Calibrates the illuminance value (percentual offset), takes into effect on next report of device.", "name": "illuminance_calibration", "property": "illuminance_calibration", "type": "numeric" }, { "access": 2, "description": "Calibrates the device_temperature value (absolute offset), takes into effect on next report of device.", "name": "device_temperature_calibration", "property": "device_temperature_calibration", "type": "numeric" }], "supports_ota": true, "vendor": "Xiaomi" }, "endpoints": { "1": { "bindings": [], "clusters": { "input": ["genBasic", "genPowerCfg", "genIdentify", "aqaraOpple"], "output": ["genIdentify", "genOta", "aqaraOpple"] }, "configured_reportings": [], "scenes": [] } }, "friendly_name": "hello/testing", "ieee_address": "0x01234567890abcde", "interview_completed": true, "interviewing": false, "manufacturer": "LUMI", "model_id": "lumi.motion.ac02", "network_address": 32549, "power_source": "Battery", "software_build_id": "0.0.0_0005", "supported": true, "type": "EndDevice" }] ]] local json = require ('dkjson') local ZBdevices = json.decode(zigbee2mqtt_bridge_devices) print('Number of devices = '..#ZBdevices..'\n') for idx=1, #ZBdevices do print ('friendly_name is '..ZBdevices[idx].friendly_name) print ('ieee_address is '..ZBdevices[idx].ieee_address..'\n') -- if not the coordinator if ZBdevices[idx].definition then print('Properties follow:') local foundMotion = false for idx2=1, #ZBdevices[idx].definition.exposes do devProperty = ZBdevices[idx].definition.exposes[idx2].property print (devProperty) if devProperty == 'occupancy' then foundMotion = true end end if foundMotion then print("\nDevice is a motion sensor") end end end return true
Zigbee2MQTT
8/73
