Migrating Zwave & ZigBee stuff to MQTT - guidance needed.

akbooer

@a-lurker said in Migrating Zwave & ZigBee stuff to MQTT - guidance needed.:

Here is one observed issue: I have in my own code with a subscription to the occupancy sensor and using that to set a switch in VirtualHTTP. Has been working and has now stopped working. You see the code here - line 415.. Looks like something in openLuup has stopped that functioning? See error in log below:

This is clearly a nil topic value in the call. Where that is supposed to come from, I don't know.

@a-lurker said in Migrating Zwave & ZigBee stuff to MQTT - guidance needed.:

Here is the light being turned on & off noting that it actually does go on & off but I'm unclear about all the "Topic ignored". eg "0x_redacted_light_bulb/set/state"

The ignored topics are the reason that the openLuup device state is not changing – I simply haven't coded the monitoring side of things yet. The next step.

I'm unclear because of your log edits whether the devices are using their ieee_address or their friendly_name as MQTT topics. AFAIK, it's legal to use either? I need to make some extra arrangements if so.

Which devices are currently not being correctly identified?

a-lurker

"The ignored topics are the reason that the openLuup device state is not changing – I simply haven't coded the monitoring side of things yet" - makes sense. Yes next step.
"your log edits": - sorry about this. Being a bit paranoid. All the ieee_address(s) and their associated friendly_name(s) are equal to each other. I haven't changed them. They are still the hex "mac" addresses of the devices.
"AFAIK, it's legal to use either? " It would certainly appear so. The use of the friendly_name(s) would certainly be needed to do topic filtering. ie plus & hash. But I've made no attempt so far to change the friendly_name(s). Trying to keep the change in all the variables to a minimum.
"This is clearly a nil topic value in the call" - this is my separate code. I subscribe to the occupancy sensor and the openLuup handler routine is coded to set a virtual switch. Seems the topic is getting screwed up in the handler but may be red herring, so please don't take too much notice. However as far as I can see @thereadldb has not made any recent code changes.
"Which devices are currently not being correctly identified?" - errhh none! Working very well so far.

If I need to provide more info please let me know.

akbooer

@a-lurker said in Migrating Zwave & ZigBee stuff to MQTT - guidance needed.:

"Which devices are currently not being correctly identified?" - errhh none! Working very well so far.

Oh! ... wasn't expecting that!

Currently working on the device variable status updates.

akbooer

@a-lurker said in Migrating Zwave & ZigBee stuff to MQTT - guidance needed.:

Being a bit paranoid.

I did, at least, replace username and password in the MQTT diagnostics with "****"!

These wouldn't anyway appear in the log at all if you had not enabled DEBUG in that module.

akbooer

New testing version v22.11.30 should now mirror the device status (on/off and brightness) and also enable the openLuup device on/off control to work. Brightness level not yet implemented.

Battery levels should be shown too (once the device has woken up.)

Does the occupancy sensor work too?

a-lurker

@akbooer
Looking good - using AltUI for the following:

Light bulb:

Works:

on/off does control the bulb
variable LoadLevelStatus reflects any brightness changes
virtually all variables are loaded with a few exceptions - see below

Doesn't work:

brightness slider (as you noted)
light bulb icon does not change state in UI
variable LoadLevelTarget is always zero
variable linkquality is not loaded from the mqtt messages
variable effect is not loaded from the mqtt messages

Motion detector:

Works:

red dagger shows in UI when triggered
last trigger time shows in the UI (30 secs later as per its set up parms)
battery icon shows with percentage
virtually all variables are loaded with a few exceptions - see below

Not loaded from the mqtt messages:

detection_interval
device_temperature
illuminance
illuminance_lux
linkquality
motion_sensitivity
occupancy
trigger_indicator
voltage

Other:

variable "BatteryLevel" (and UI icon) shows "100" and variable "battery" shows "numeric". mqtt exposes "battery", so maybe some mix up or duplication here. They're probably the same thing - one mios & one zigbee.

Start up problem:
Conditions:

devices have all been previously detected and built and are available in AltUI.
the zigbee2mqtt server is running.

OK now restart openLupp. Status of all devices are not updated by any incoming mqtt messages from then on. Log says: "Topic Ignored : 0x_ieee_address".

Referring to file L_Zigbee2MQTTBridge.lua: function handle_friendly_names line 407 . Table "devices" is empty, so "dno" and "D" are nil.

Reason: it appears the "devices" table is loaded by "function init_device" and this is only called when the devices are first created in "function create_devices".

The devices table needs to be loaded if the devices have already been created. The work around is to restart the zigbee2mqtt server, so it publishes the topic "zigbee2mqtt/devices".

Logging:
Another slight issue with logging: The log showed for example on occasion : "bridge/logging" when it's actually "zigbee2mqtt/bridge/logging". The prefix zigbee2mqtt is missing and it makes a little unclear what topic is actually being ignored. Just makes it a bit harder to debug.

Console -> openLuup -> About:
There is item labelled "Debug" - how do you set this to true?

Everything is coming together!

a-lurker

Next device I'm hoping to try out: a Xiaomi button.

The above has similarities with the Hue dimmer switch

And then there is the magic cube - that's a very interesting one. eg you can just rotate it to dim/undim lights. Flip it and control something else.

akbooer

Thanks for the report… all exactly as I would expect. Indeed I had not done anything about initialisation apart from the bridge/device message. I now need to use the variables from existing devices to build the initial table.

I see what you mean about the error message. Will fix that. Will post separately on the debug switch. It may make sense to have console page for that.

akbooer

@a-lurker

Latest testing version v22.12.01 should fix all reported issues.

I'm working on the switches. These will appear as scene controllers.

akbooer

@a-lurker

Just pushed v22.12.01b with initial go at supporting those switches.

a-lurker

@akbooer

Light bulb slider now working:

Issue with variable updating. Looks like api.lua handles this somehow in the meta verse:

openLuup.context_switch::  ERROR: [dev #0] ./openLuup/L_Zigbee2MQTTBridge.lua:166: ERROR - READONLY: attempt to create index illuminance

So added some debug code to this section - not sure how variable "S" is used:

  -- update exposed values
  local S = D.exposes
  for n,v in pairs (message) do
    D[n] = tostring(v)
  end

Motion sensor:

luup_log:0: D: table: 0x1b25f70
luup_log:0: S: table: 0x1680730
luup_log:0: n: illuminance
luup_log:0: v: 75
luup_log:0: n: device_temperature
luup_log:0: v: 20
luup_log:0: n: battery
luup_log:0: v: 100
luup_log:0: n: occupancy
luup_log:0: v: false
luup_log:0: n: detection_interval
luup_log:0: v: 30
luup_log:0: n: voltage
luup_log:0: v: 3177
luup_log:0: n: linkquality
luup_log:0: v: 255
luup_log:0: n: trigger_indicator
luup_log:0: v: false
luup_log:0: n: power_outage_count
luup_log:0: v: 0
luup_log:0: n: motion_sensitivity
luup_log:0: v: medium

Light bulb. Note that "color" and "update" are table ptrs:

luup_log:0: D: table: 0x1b25a30
luup_log:0: S: table: 0xed39b0
luup_log:0: n: state
luup_log:0: v: ON
luup_log:0: n: color_temp
luup_log:0: v: 153
luup_log:0: n: color_mode
luup_log:0: v: xy
luup_log:0: n: color
luup_log:0: v: table: 0x8ec4f8
luup_log:0: n: brightness
luup_log:0: v: 53
luup_log:0: n: linkquality
luup_log:0: v: 212
luup_log:0: n: update
luup_log:0: v: table: 0x3e0268

akbooer

Ooops!

Looks like this should be:

    S[n] = tostring(v)

akbooer

@a-lurker

” Looks like api.lua handles this somehow in the meta verse:”

Yes, FYI the new openLuup API maps device services and variables directly into the plug-in context. There are shorthand names for the otherwise unwieldy serviceIds:

API[devNo][serviceId][varname] = value    -- sets the value
value = API[devNo][serviceId][varname]    -- gets the value

These are all standard Lua (metatable) structures, so you can write:

D = API[devNo]    -- the device
S = D[serviceId]    -- the service
S[varname] = value    -- sets the value
value = S[varname]    -- gets the value

kWh = D.energy.KWH    -- get device energy usage

It’s still evolving, but this much is stable. It’s so neat and readable compared to the usual luup.variable_set/get() calls, and I’m now using it all the time in new code development (since I’ll never be writing another Vera-compatible plug-in.)

akbooer

@a-lurker

Switching to the development branch for updates...

new v22.12.5 should update those table variables now (and fix the typo.)

a-lurker

@akbooer
This is all going nicely - but time will tell how reliable the system as whole would be. Have added in one of these Aqara buttons and it work fine.

Zigbee2mqtt "interviews" a device while it's being paired. On occasion the interview is partially completed ie it fails and some variables are void or invalid but "Interview completed" is still set to true.

With such a faulty pairing and with openLuup being subjected to same, more often or not, the function "infer_device_type" will default to "D_GenericZigbeeDevice.xml", of which the actual file doesn't seem to exist. So one occasion I observed this error, which appears to be related:

openLuup.context_switch::  ERROR: [dev #0] ./openLuup/L_Zigbee2MQTTBridge.lua:438: attempt to call field '?' (a nil value)
openLuup.mqtt:: ERROR publishing application message for mqtt:zigbee2mqtt/0x00158d000706951c : ./openLuup/L_Zigbee2MQTTBridge.lua:438: attempt to call field '?' (a nil value)

You have to repair the device, delete the "Generic" device in AltUI and then restart the zigbee2mqtt server to get the correct device to show in AltUI.

Inference: Looking at the function "infer_device_type: the code IMO should be split, so "switch" and "action" produce different devices ie an on/off switch or a scene controller:

In GitHub: elseif exp.switch or exp.action then

The nomenclature is bit tricky: switch is often conflated with power relay switches and say a scene controller switch. In Zigbee2MQTT speak, a "switch" is like a relay (mechanical or solid state) and a scene controller switch is considered something that has an "action".

Composite devices will become an issue and hopefully that can be handled in the future (yes can of worms), especially dual outlet power sockets or multi gang switches, which are ubiquitous.

Here is an example of sometime that doesn't fit the mold. These Aqara devices logically act as both a switch and a scene controller. Question arises if you do say a "double" click, what happens to the switch status. Likewise for say "hold_release". There are very few of these devices and they are nearly all made by Xiaomi.

akbooer

@a-lurker said in Migrating Zwave & ZigBee stuff to MQTT - guidance needed.:

time will tell how reliable the system as whole would be

Is there some symptomatic problem I should know about?

@a-lurker said in Migrating Zwave & ZigBee stuff to MQTT - guidance needed.:

"D_GenericZigbeeDevice.xml", of which the actual file doesn't seem to exist

It does, in openLuup's virtual world. If you try http://openLuupIP:3480/D_GenericZigbeeDevice.xml you should see this in the browser:

<?xml version="1.0"?>
<root xmlns="urn:schemas-upnp-org:device-1-0">
 <specVersion>
  <major>1</major>
  <minor>0</minor>
  <minimus>auto-generated</minimus></specVersion>
 <device>
  <staticJson>D_GenericZigbeeDevice.json</staticJson>
  <deviceType>GenericZigbeeDevice</deviceType></device></root>

I'll fix the error and review the infer_device_type() function. It was always going to evolve anyway as different device needs arise.

I have tended to represent any sort of multi-action switch as a scene controller. There will always be difficult cases (sometimes resolved by using vendor/manufacturer) but I have shied away from an extensive configuration database. Plausible, I suppose, to pull this from the web...?

I rely on Philips Hue for the few Zigbee devices that I have, so probably won't get to road test this for myself. You're going to want to do the whole thing again for Zwave JS ?

a-lurker

@akbooer said in Migrating Zwave & ZigBee stuff to MQTT - guidance needed.:

Is there some symptomatic problem I should know about?

I don't think so but who wants to set up heaps of devices in a new set up when everything has been going well for years with just openLuup and an old Vera war horse!

Plausible, I suppose, to pull this from the web...?

Don't need to get to complicated too soon, as you just about have it all covered but I'm still thinking of one of these Hue Smart plugs. It's just a switch, as opposed to a scene controller.

I rely on Philips Hue for the few Zigbee devices that I have

Arrhh you know you want to ditch that hub.

You're going to want to do the whole thing again for Zwave JS ?

Maybe after a cup of tea and good lie down. And yourself?

akbooer

@a-lurker said in Migrating Zwave & ZigBee stuff to MQTT - guidance needed.:

Maybe after a cup of tea and good lie down. And yourself?

Definitely the tea. Possibly the lie down.

You'll have to educate me about the protocol, yet again, but it should almost be a dead-ringer for the Zigbee ??

akbooer

Release development v22.12.6 separating switch and controller devices, and fixing nil function pointer.

a-lurker

@akbooer said in Migrating Zwave & ZigBee stuff to MQTT - guidance needed.:

Release development v22.12.6 separating switch and controller devices, and fixing nil function pointer

Thanks for that - looks OK now.

Have now got all my Hue stuff running with no Hue hub - that's good. A few complications here and there.

Main complication:
Conditions:
z2m is running and openLuup is running.
Then pair new device: in this case this Philips light bulb with z2m.
z2m logs this to the terminal. I've prettied the json section so you can read it:

Zigbee2MQTT:info  2022-12-10 10:40:28: Device '0x001788010363fcbf' joined
Zigbee2MQTT:info  2022-12-10 10:40:28: MQTT publish: topic 'zigbee2mqtt/bridge/event', payload '{"data":{"friendly_name":"0x001788010363fcbf","ieee_address":"0x001788010363fcbf"},"type":"device_joined"}'
Zigbee2MQTT:info  2022-12-10 10:40:28: Starting interview of '0x001788010363fcbf'
Zigbee2MQTT:info  2022-12-10 10:40:28: MQTT publish: topic 'zigbee2mqtt/bridge/event', payload '{"data":{"friendly_name":"0x001788010363fcbf","ieee_address":"0x001788010363fcbf","status":"started"},"type":"device_interview"}'
Zigbee2MQTT:info  2022-12-10 10:40:29: Successfully interviewed '0x001788010363fcbf', device has successfully been paired
Zigbee2MQTT:info  2022-12-10 10:40:29: Device '0x001788010363fcbf' is supported, identified as: Philips Hue White A60 Single bulb E27/B22 (8718696449691)
Zigbee2MQTT:info  2022-12-10 10:40:29: MQTT publish: topic 'zigbee2mqtt/bridge/event', payload '
{
	"data": {
		"definition": {
			"description": "Hue White A60 Single bulb E27/B22",
			"exposes": [{
				"features": [{
					"access": 7,
					"description": "On/off state of this light",
					"name": "state",
					"property": "state",
					"type": "binary",
					"value_off": "OFF",
					"value_on": "ON",
					"value_toggle": "TOGGLE"
				}, {
					"access": 7,
					"description": "Brightness of this light",
					"name": "brightness",
					"property": "brightness",
					"type": "numeric",
					"value_max": 254,
					"value_min": 0
				}],
				"type": "light"
			}, {
				"access": 2,
				"description": "Triggers an effect on the light (e.g. make light blink for a few seconds)",
				"name": "effect",
				"property": "effect",
				"type": "enum",
				"values": ["blink", "breathe", "okay", "channel_change", "finish_effect", "stop_effect"]
			}, {
				"access": 1,
				"description": "Link quality (signal strength)",
				"name": "linkquality",
				"property": "linkquality",
				"type": "numeric",
				"unit": "lqi",
				"value_max": 255,
				"value_min": 0
			}],
			"model": "8718696449691",
			"options": [{
				"access": 2,
				"description": "Controls the transition time (in seconds) of on/off, brightness, color temperature (if applicable) and color (if applicable) changes. Defaults to `0` (no transition).",
				"name": "transition",
				"property": "transition",
				"type": "numeric",
				"value_min": 0
			}],
			"supports_ota": true,
			"vendor": "Philips"
		},
		"friendly_name": "0x001788010363fcbf",
		"ieee_address": "0x001788010363fcbf",
		"status": "successful",
		"supported": true
	},
	"type": "device_interview"
}
'
Zigbee2MQTT:info  2022-12-10 10:40:29: Configuring '0x001788010363fcbf'
Zigbee2MQTT:info  2022-12-10 10:40:29: Successfully configured '0x001788010363fcbf'

Looks like openLuup then responds to the topic zigbee2mqtt/bridge/event ?? and creates a Generic device. The openLuup log doesn't make it clear what topic it's responding to when it creates a device but you can see in the log that it has found a new device:

2022-12-10 10:40:28.568   openLuup.mqtt:: PUBLISH tcp{client}: 0x3a7b670
2022-12-10 10:40:28.571   luup.zigbee2mqtt:0: Topic ignored : zigbee2mqtt : / : bridge/logging
2022-12-10 10:40:28.574   openLuup.mqtt:: PUBLISH tcp{client}: 0x3a7b670
2022-12-10 10:40:28.591   luup.zigbee2mqtt:0: New Zigbee detected: 0x001788010363fcbf
2022-12-10 10:40:28.594   luup.create_device:: [20010] D_GenericZigbeeDevice.xml /  / D_GenericZigbeeDevice.json   (GenericZigbeeDevice)
2022-12-10 10:40:28.602   openLuup.mqtt:: PUBLISH tcp{client}: 0x3a7b670
2022-12-10 10:40:28.603   luup.zigbee2mqtt:0: Topic ignored : zigbee2mqtt : / : bridge/logging
2022-12-10 10:40:28.605   openLuup.mqtt:: PUBLISH tcp{client}: 0x3a7b670
2022-12-10 10:40:28.605   luup.zigbee2mqtt:0: Topic ignored : zigbee2mqtt : / : bridge/event
2022-12-10 10:40:28.607   openLuup.mqtt:: PUBLISH tcp{client}: 0x3a7b670
2022-12-10 10:40:28.607   luup.zigbee2mqtt:0: Topic ignored : zigbee2mqtt : / : bridge/logging
2022-12-10 10:40:28.609   openLuup.mqtt:: PUBLISH tcp{client}: 0x3a7b670
2022-12-10 10:40:28.626   openLuup.mqtt:: PUBLISH tcp{client}: 0x3a7b670
2022-12-10 10:40:28.626   luup.zigbee2mqtt:0: Topic ignored : zigbee2mqtt : / : bridge/logging
2022-12-10 10:40:28.628   openLuup.mqtt:: PUBLISH tcp{client}: 0x3a7b670
2022-12-10 10:40:28.628   luup.zigbee2mqtt:0: Topic ignored : zigbee2mqtt : / : bridge/event
2022-12-10 10:40:28.754   openLuup.server:: request completed (11333 bytes, 1 chunks, 2261 ms) tcp{client}: 0x43d0070
2022-12-10 10:40:28.766   openLuup.server:: GET /data_request?id=user_data&output_format=json&DataVersion=564556066&_=1670559272961 HTTP/1.1 tcp{client}: 0x43d0070
2022-12-10 10:40:29.987   openLuup.server:: request completed (1723523 bytes, 108 chunks, 1220 ms) tcp{client}: 0x43d0070
2022-12-10 10:40:30.017   openLuup.mqtt:: PUBLISH tcp{client}: 0x3a7b670
2022-12-10 10:40:30.017   luup.zigbee2mqtt:0: Topic ignored : zigbee2mqtt : / : bridge/logging
2022-12-10 10:40:30.018   openLuup.mqtt:: PUBLISH tcp{client}: 0x3a7b670
2022-12-10 10:40:30.018   luup.zigbee2mqtt:0: Topic ignored : zigbee2mqtt : / : bridge/logging
2022-12-10 10:40:30.020   openLuup.mqtt:: PUBLISH tcp{client}: 0x3a7b670
2022-12-10 10:40:30.032   openLuup.mqtt:: PUBLISH tcp{client}: 0x3a7b670
2022-12-10 10:40:30.033   luup.zigbee2mqtt:0: Topic ignored : zigbee2mqtt : / : bridge/logging
2022-12-10 10:40:30.034   openLuup.mqtt:: PUBLISH tcp{client}: 0x3a7b670
2022-12-10 10:40:30.035   luup.zigbee2mqtt:0: Topic ignored : zigbee2mqtt : / : bridge/event
2022-12-10 10:40:30.036   openLuup.mqtt:: PUBLISH tcp{client}: 0x3a7b670
2022-12-10 10:40:30.036   luup.zigbee2mqtt:0: Topic ignored : zigbee2mqtt : / : bridge/logging
2022-12-10 10:40:30.037   openLuup.mqtt:: PUBLISH tcp{client}: 0x3a7b670
2022-12-10 10:40:30.037   luup.zigbee2mqtt:0: Topic ignored : zigbee2mqtt : / : bridge/logging
2022-12-10 10:40:30.039   openLuup.mqtt:: PUBLISH tcp{client}: 0x3a7b670
2022-12-10 10:40:30.048   openLuup.mqtt:: PUBLISH tcp{client}: 0x3a7b670
2022-12-10 10:40:30.052   luup.zigbee2mqtt:0: Topic ignored : zigbee2mqtt : / : bridge/info

So it looks as the event returns the same data as "zigbee2mqtt/bridge/devices" does, as each new device is paired but it's parent json is a little different. The device definition is below "data" in the json:

"data":

With type set as follows:

type="device_interview",

and this must indicate success:

data.status ="successful"

Hope that's clear - bit hard to explain.

Suffice to say I added a pile of stuff and ended up with a whole pile of generic devices, which I had to remove one by one with a openLuup restart for each. Got a bit ambitious!

However; it looks like the ability to handle the addition of new devices on the fly without having to restart thez2m server each time to get the device definitions, would certainly be possible and a welcome addition.

Looks like you just look for the event topic with {"type":"device_interview" and {"data": status":"successful".

The json contains definition.expose.type = light so the function infer_device_type() should still do its work OK as it currently does.

I can't test any new code on this as I have now added all my devices. Although I could unpair just a light bulb and repair it and see what happens.

openLuup email server

Why different types of Triggers within openLuup & ALTUI

openLuup_install.lua - URL changed?

OpenLuup : Loading Global Modules

openLuup charting and forward slashes in variable names

openLuup log files - LuaUPnP.log and LuaUPnP_startup.log

openLuup: Version Log

ZWaveJS and MQTT

Awesome MQTT - GitHub list

GetSolarCoords - correct results?

Truncated web pages using wireguard over long distances

Console view: - No Live refreshes via :3480/console?page=devices

openLuup and room names

Need to get rid of RulesEngine

Migrating Zwave & ZigBee stuff to MQTT - guidance needed.

DisplayLine1 & 2 - Ability to align text?

AltUI access over VPN

Problem with luup.call_delay nexted in luup.call_timer

OpenLuup installation

Unexpected stop of openLuup

Migrating Zwave & ZigBee stuff to MQTT - guidance needed.

Recent Topics