openLuup email server

[akbooer]

@a-lurker said in Migrating Zwave & ZigBee stuff to MQTT - guidance needed.:

It seems there is no "auto magic bridge set up".

I need to spend a bit of time digesting your post, but for both Shellies and Tasmota devices I did create a "magic bridge setup", wherein there is no install or configuration required. The bridge self-installs if openLuup sees one of those devices (over MQTT) and creates a corresponding virtual device.

I could do the same for other types of device, but I'd need to understand the protocol that your environment was providing.

[CatmanV2]

@akbooer said in Migrating Zwave & ZigBee stuff to MQTT - guidance needed.:

I did create a "magic bridge setup"

Is that available to mortals?

C

[akbooer]

For Shellies and Tasmota, that's built in already...

[CatmanV2]

Oooh now there's interesting.....

<dumb question> How is OpenLuup communicating with MQTT? I can't see anything in my config

C

[akbooer]

@catmanv2 said in Migrating Zwave & ZigBee stuff to MQTT - guidance needed.:

How is OpenLuup communicating with MQTT? I can't see anything in my config

openLuup has its own built-in MQTT server:

https://smarthome.community/topic/20/openluup-version-log/13

[CatmanV2]

Ooooooooooh!

Is there any reason I shouldn't run both this and Mosquitto? Apart from the inelegance?

Is there any reason not to kill mosquitto and just point everything at this?

Cheers

C

[akbooer]

You can run both. You could even bridge to brokers. The openLuup broker only supports QoS 0, but this is fine for this purpose.

Give it a try. Doesn’t yet support Shelly Plus (NG) devices.

[CatmanV2]

I intend to! As soon as I get some time. Last dumb q: Upgrading is simply downloading the openLuup_install.lua file and running it in the correct location?

Nothing else need to worry about? I'm currently on
2021.04.26
Would seem a sensible time to get up to 2021.07/25?

Cheers!

C

[a-lurker]

Just go to this URL and hit the upgrade (blue arrows) upgrade button. If feeling brave, type the word development into the box beforehand and get the latest and greatest:

http://openLuup_IP_address:3480/console?page=plugins_table

I think my thread has been hijacked!

[CatmanV2]

@a-lurker said in Migrating Zwave & ZigBee stuff to MQTT - guidance needed.:

I think my thread has been hijacked!

I am SO sorry

I will shut up now, I promise!

C

[a-lurker]

@akbooer said in Migrating Zwave & ZigBee stuff to MQTT - guidance needed.:

The bridge self-installs if openLuup sees one of those devices (over MQTT) and creates a corresponding virtual device.

I can see the complexity of doing this for zigbee2mqtt as there are so many devices, many of which are composites. So for example these motion detectors:

Aqara P1

which are tiny and also have high claimed battery life, have all sorts of properties in their payload - however it's fundamentally a motion sensor:

{
  "battery": 100,
  "detection_interval": 30,
  "device_temperature": 22,
  "illuminance": 64,
  "linkquality": 255,
  "motion_sensitivity": "medium",
  "occupancy": false,
  "power_outage_count": 0,
  "trigger_indicator": false,
  "voltage": 3215
}

Home Assistant has similar problems, where there are so many properties to perhaps make use of and/or handle.

I'm thinking that with zigbee2mqtt and also ZWAVE JS UI you could possibly do OK with just handling a few types of devices. I would suggest just:

• Relays
• Dimmable and colored lights
• Blinds?

Other stuff could probably be handled with code and the Virtual HTTP Devices plug-in.

Out of interest; the zigbee2mqtt "Bridge" churns out stuff like this for each zigbee device:

[
  {
    "definition": null,
    "endpoints": {
      "1": {
        "bindings": [],
        "clusters": {
          "input": [
            "genBasic",
            "genIdentify",
            "genOnOff",
            "genTime",
            "genOta",
            "26",
            "lightingColorCtrl"
          ],
          "output": [
            "genBasic",
            "genIdentify",
            "genGroups",
            "genScenes",
            "genOnOff",
            "genLevelCtrl",
            "genPollCtrl",
            "lightingColorCtrl",
            "msIlluminanceMeasurement",
            "msTemperatureMeasurement",
            "msRelativeHumidity",
            "msOccupancySensing",
            "ssIasZone",
            "haMeterIdentification",
            "haApplianceStatistics",
            "haElectricalMeasurement",
            "seMetering",
            "touchlink",
            "manuSpecificUbisysDimmerSetup",
            "manuSpecificSamsungAccelerometer"
          ]
        },
        "configured_reportings": [],
        "scenes": []
      },
      "242": {
        "bindings": [],
        "clusters": {
          "input": [],
          "output": [
            "greenPower"
          ]
        },
        "configured_reportings": [],
        "scenes": []
      }
    },
    "friendly_name": "Coordinator",
    "ieee_address": "0xredacted",
    "interview_completed": true,
    "interviewing": false,
    "network_address": 0,
    "supported": false,
    "type": "Coordinator"
  },
  {
    "date_code": "Dec  7 2021",
    "definition": {
      "description": "Aqara P1 human body movement and illuminance sensor",
      "exposes": [
        {
          "access": 1,
          "description": "Indicates whether the device detected occupancy",
          "name": "occupancy",
          "property": "occupancy",
          "type": "binary",
          "value_off": false,
          "value_on": true
        },
        {
          "access": 1,
          "description": "Measured illuminance in lux",
          "name": "illuminance_lux",
          "property": "illuminance",
          "type": "numeric",
          "unit": "lx"
        },
        {
          "access": 1,
          "description": "Measured illuminance in lux",
          "name": "illuminance",
          "property": "illuminance",
          "type": "numeric",
          "unit": "lx"
        },
        {
          "access": 7,
          "description": ". Press pairing button right before changing this otherwise it will fail.",
          "name": "motion_sensitivity",
          "property": "motion_sensitivity",
          "type": "enum",
          "values": [
            "low",
            "medium",
            "high"
          ]
        },
        {
          "access": 7,
          "description": "Time interval for detecting actions. Press pairing button right before changing this otherwise it will fail.",
          "name": "detection_interval",
          "property": "detection_interval",
          "type": "numeric",
          "unit": "s",
          "value_max": 65535,
          "value_min": 2
        },
        {
          "access": 7,
          "description": "When this option is enabled then blue LED will blink once when motion is detected. Press pairing button right before changing this otherwise it will fail.",
          "name": "trigger_indicator",
          "property": "trigger_indicator",
          "type": "binary",
          "value_off": false,
          "value_on": true
        },
        {
          "access": 1,
          "description": "Temperature of the device",
          "name": "device_temperature",
          "property": "device_temperature",
          "type": "numeric",
          "unit": "°C"
        },
        {
          "access": 1,
          "description": "Remaining battery in %",
          "name": "battery",
          "property": "battery",
          "type": "numeric",
          "unit": "%",
          "value_max": 100,
          "value_min": 0
        },
        {
          "access": 1,
          "description": "Voltage of the battery in millivolts",
          "name": "voltage",
          "property": "voltage",
          "type": "numeric",
          "unit": "mV"
        },
        {
          "access": 1,
          "description": "Link quality (signal strength)",
          "name": "linkquality",
          "property": "linkquality",
          "type": "numeric",
          "unit": "lqi",
          "value_max": 255,
          "value_min": 0
        }
      ],
      "model": "RTCGQ14LM",
      "options": [
        {
          "access": 2,
          "description": "Time in seconds after which occupancy is cleared after detecting it (default is \"detection_interval\" + 2 seconds). The value must be equal to or greater than \"detection_interval\", and it can also be a fraction.",
          "name": "occupancy_timeout",
          "property": "occupancy_timeout",
          "type": "numeric",
          "unit": "s",
          "value_min": 0,
          "value_step": 0.1
        },
        {
          "access": 2,
          "description": "Sends a message the last time occupancy (occupancy: true) was detected. When setting this for example to [10, 60] a `{\"no_occupancy_since\": 10}` will be send after 10 seconds and a `{\"no_occupancy_since\": 60}` after 60 seconds.",
          "item_type": "number",
          "name": "no_occupancy_since",
          "property": "no_occupancy_since",
          "type": "list"
        },
        {
          "access": 2,
          "description": "Calibrates the illuminance value (percentual offset), takes into effect on next report of device.",
          "name": "illuminance_calibration",
          "property": "illuminance_calibration",
          "type": "numeric"
        },
        {
          "access": 2,
          "description": "Calibrates the device_temperature value (absolute offset), takes into effect on next report of device.",
          "name": "device_temperature_calibration",
          "property": "device_temperature_calibration",
          "type": "numeric"
        }
      ],
      "supports_ota": true,
      "vendor": "Xiaomi"
    },
    "endpoints": {
      "1": {
        "bindings": [],
        "clusters": {
          "input": [
            "genBasic",
            "genPowerCfg",
            "genIdentify",
            "aqaraOpple"
          ],
          "output": [
            "genIdentify",
            "genOta",
            "aqaraOpple"
          ]
        },
        "configured_reportings": [],
        "scenes": []
      }
    },
    "friendly_name": "hello/testing",
    "ieee_address": "0x01234567890abcde",
    "interview_completed": true,
    "interviewing": false,
    "manufacturer": "LUMI",
    "model_id": "lumi.motion.ac02",
    "network_address": 32549,
    "power_source": "Battery",
    "software_build_id": "0.0.0_0005",
    "supported": true,
    "type": "EndDevice"
  }
]

However getting back to the migration issue; I could probably make use of the Virtual HTTP Devices plug-in but as previously mentioned, I may need up to 80 devices to be created.

How do you "Create" a device programmatically? Looks like the MIOS documentation Wiki is being abandoned; otherwise I would scour through that for the answer. Assume you need to post D_zyz.xml and I_xyz.xml file names, etc to somewhere?

[therealdb]

Look at this:

vera-OpenSprinkler/L_VeraOpenSprinkler1.lua at master · dbochicchio/vera-OpenSprinkler

Completely new and rewritten plug-in to interface an OpenSprinkler to a Vera or openLuup system. - dbochicchio/vera-OpenSprinkler

You'll need to append them and then sync

vera-OpenSprinkler/L_VeraOpenSprinkler1.lua at master · dbochicchio/vera-OpenSprinkler

Completely new and rewritten plug-in to interface an OpenSprinkler to a Vera or openLuup system. - dbochicchio/vera-OpenSprinkler

Honestly, I've dumped all my luup code from production, so my skills are definitely at the lower end and the wiki has gone.

[akbooer]

@a-lurker said in Migrating Zwave & ZigBee stuff to MQTT - guidance needed.:

I'm thinking that with zigbee2mqtt and also ZWAVE JS UI you could possibly do OK with just handling a few types of devices. I would suggest just:

Relays
Dimmable and colored lights
Blinds?

Other stuff could probably be handled with code and the Virtual HTTP Devices plug-in.

Yes, that's the sort of thing I had in mind. Both Shelly and Tasmota bridges are similarly constrained.

What topics do zigbee2mqtt and ZWAVE JS UI use?

Shelly currently subsribes to "shellies/#", and Tasmota to "tele/#", and " tasmota/tele/#".

[a-lurker]

@akbooer

I'm still delving into this, so my current knowledge is a little on the short side and the following is a little generalized. However:

It seems that battery operated devices have nothing that can be controlled and their status cannot be read on demand ie no set or get possible (as indicated by an "access" sub property - see below). Presumably because they are asleep most of the time. So a battery powered Aqara motion detector publishes this only when it's triggered by motion:

"zigbee2mqtt/0x01234567890abcde" and supplying a flat json payload containing all the properties - see previous post above.

"zigbee2mqtt" begins all the topics. It can be set to something else but not sure why you would; unless you had multiple instances of zigbee2mqtt running.

0x01234567890abcde is the "ieee_address" of the device and it can be associated with a "friendly_name" of your own choice, such as "bedroom1/left_corner/motion"

So you could have topics such as:

"zigbee2mqttInstance2/bedroom1/left_corner/motion" instead of "zigbee2mqtt/0x01234567890abcde"

The bridge defines all the devices and the devices' topics:

"zigbee2mqtt/bridge/devices" with a payload, which is an array of json objects - one for each device. ieee_address and friendly_name are top level properties of each device in the array. See previous post above.

The "zigbee2mqtt/bridge/devices" topic is published when you first connect to the zigbee2mqtt server. ie you can't do a GET on it - not sure why that is. It's republished if a device joins or leaves. You can force a republish by doing: "zigbee2mqtt/bridge/request/restart" but that's a bit radical, as it does a full server restart.

Each property in the device's status report is also further described by the bridge. So a status property such as temperature has sub properties eg "access", which indicates if the property can be set or get. ie battery operated devices can't be set or get. That is to say each device has an array of property description objects.

So to set up: you would load up the array of devices from "zigbee2mqtt/bridge/devices". Then wizz through them looking for ones of interest under "definition.exposes[array]". Trying to identify what the device type is, appears to be a bit tricky. Looks like you have check the device for the presence of various property combinations or omissions.

Aqara motion detector: has "occupancy". Also has device_temperature & illuminance, which in this case is not much use, as you only get the values when motion triggers the unit.

Hue light bulb: has "light"

Hue light switch: has "action"

Gets tricky with say composite devices. But it would be good to allow some of them eg:

Ikuü Dual outlet: has "switch" with "state_left" & "state_right" but no "brightness"

Tuya five gang light switch: has: "switch" with "state_l1" to "state_l5" (but no "moving_*" properties see below).

Lots of the devices also have temperature sensors and some have energy consumption sensors.

But then you have this that helps confound identification of a device:

Dual curtain/blind controller: It has "moving_left" & "moving_right" but it also has "state_left" & "state_right" like the Ikuü Dual outlet above!

Controlling things - eg a light bulb. There are two ways:

• send a block of json as the payload
• or pure text as the payload, with words such as ON or OFF or TOGGLE

From the following link that talks about these two SET & GET:

In case you don't want to use JSON, publishing to zigbee2mqtt/[FRIENDLY_NAME]/set/state with payload ON is the same as publishing to zigbee2mqtt/[FRIENDLY_NAME]/set payload {"state": "ON"}"

MQTT Topics and Messages | Zigbee2MQTT

Hope this is of some help.

[a-lurker]

@akbooer
So far, I haven't looked at the ZWave bridge operation.

[a-lurker]

@therealdb

Log of switch operation:

luup.call_action:: 209.urn:upnp-org:serviceId:SwitchPower1.SetTarget 
luup_log:209: VirtualDevices[3.0-beta7@209](actionPower@94):actionPower(209,"1")
luup_log:209: VirtualDevices[3.0-beta7@209](actionPowerInternal@47):actionPowerInternal(209,true,true)
luup_log:209: VirtualDevices[3.0-beta7@209](setVar@104):setVar("urn:upnp-org:serviceId:SwitchPower1","Target","1",209) old value "0"
luup.variable_set:: 209.urn:upnp-org:serviceId:SwitchPower1.Target was: 0 now: 1 #hooks:0
luup_log:209: VirtualDevices[3.0-beta7@209](sendDeviceCommand@308):sendDeviceCommand("SetPowerURL","on",209)

These are all empty:
MQTT_BrightnessValue
MQTT_PowerStatusOff
MQTT_PowerStatusOn

These are all set to: "http://"
SetPowerOffURL
SetPowerURL
SetToggleURL
SetUpdateMetersURL

[a-lurker]

@akbooer

Zigbee supported devices page shows three column headings:

• Vendor
• Exposes
• Search

The "Exposes" pull down list is very useful as you can filter devices by what properties they expose. Useful for working on how to identify the device type. The filter allows multiple properties to be selected - allowing devices to be narrowed down further.

[a-lurker]

@akbooer

Should be able to run this code to get some idea how the type of the zigbee devices could potentially be identified:

local zigbee2mqtt_bridge_devices = [[

[{
	"definition": null,
	"endpoints": {
		"1": {
			"bindings": [],
			"clusters": {
				"input": ["genBasic", "genIdentify", "genOnOff", "genTime", "genOta", "26", "lightingColorCtrl"],
				"output": ["genBasic", "genIdentify", "genGroups", "genScenes", "genOnOff", "genLevelCtrl", "genPollCtrl", "lightingColorCtrl", "msIlluminanceMeasurement", "msTemperatureMeasurement", "msRelativeHumidity", "msOccupancySensing", "ssIasZone", "haMeterIdentification", "haApplianceStatistics", "haElectricalMeasurement", "seMetering", "touchlink", "manuSpecificUbisysDimmerSetup", "manuSpecificSamsungAccelerometer"]
			},
			"configured_reportings": [],
			"scenes": []
		},
		"242": {
			"bindings": [],
			"clusters": {
				"input": [],
				"output": ["greenPower"]
			},
			"configured_reportings": [],
			"scenes": []
		}
	},
	"friendly_name": "Coordinator",
	"ieee_address": "0xredacted",
	"interview_completed": true,
	"interviewing": false,
	"network_address": 0,
	"supported": false,
	"type": "Coordinator"
}, {
	"date_code": "Dec7 2021",
	"definition": {
		"description": "Aqara P1 human body movement and illuminance sensor",
		"exposes": [{
			"access": 1,
			"description": "Indicates whether the device detected occupancy",
			"name": "occupancy",
			"property": "occupancy",
			"type": "binary",
			"value_off": false,
			"value_on": true
		}, {
			"access": 1,
			"description": "Measured illuminance in lux",
			"name": "illuminance_lux",
			"property": "illuminance",
			"type": "numeric",
			"unit": "lx"
		}, {
			"access": 1,
			"description": "Measured illuminance in lux",
			"name": "illuminance",
			"property": "illuminance",
			"type": "numeric",
			"unit": "lx"
		}, {
			"access": 7,
			"description": ". Press pairing button right before changing this otherwise it will fail.",
			"name": "motion_sensitivity",
			"property": "motion_sensitivity",
			"type": "enum",
			"values": ["low", "medium", "high"]
		}, {
			"access": 7,
			"description": "Time interval for detecting actions. Press pairing button right before changing this otherwise it will fail.",
			"name": "detection_interval",
			"property": "detection_interval",
			"type": "numeric",
			"unit": "s",
			"value_max": 65535,
			"value_min": 2
		}, {
			"access": 7,
			"description": "When this option is enabled then blue LED will blink once when motion is detected. Press pairing button right before changing this otherwise it will fail.",
			"name": "trigger_indicator",
			"property": "trigger_indicator",
			"type": "binary",
			"value_off": false,
			"value_on": true
		}, {
			"access": 1,
			"description": "Temperature of the device",
			"name": "device_temperature",
			"property": "device_temperature",
			"type": "numeric",
			"unit": "°C"
		}, {
			"access": 1,
			"description": "Remaining battery in %",
			"name": "battery",
			"property": "battery",
			"type": "numeric",
			"unit": "%",
			"value_max": 100,
			"value_min": 0
		}, {
			"access": 1,
			"description": "Voltage of the battery in millivolts",
			"name": "voltage",
			"property": "voltage",
			"type": "numeric",
			"unit": "mV"
		}, {
			"access": 1,
			"description": "Link quality (signal strength)",
			"name": "linkquality",
			"property": "linkquality",
			"type": "numeric",
			"unit": "lqi",
			"value_max": 255,
			"value_min": 0
		}],
		"model": "RTCGQ14LM",
		"options": [{
			"access": 2,
			"description": "Time in seconds after which occupancy is cleared after detecting it (default is \"detection_interval\" + 2 seconds). The value must be equal to or greater than \"detection_interval\", and it can also be a fraction.",
			"name": "occupancy_timeout",
			"property": "occupancy_timeout",
			"type": "numeric",
			"unit": "s",
			"value_min": 0,
			"value_step": 0.1
		}, {
			"access": 2,
			"description": "Sends a message the last time occupancy (occupancy: true) was detected. When setting this for example to [10, 60] a `{\"no_occupancy_since\": 10}` will be send after 10 seconds and a `{\"no_occupancy_since\": 60}` after 60 seconds.",
			"item_type": "number",
			"name": "no_occupancy_since",
			"property": "no_occupancy_since",
			"type": "list"
		}, {
			"access": 2,
			"description": "Calibrates the illuminance value (percentual offset), takes into effect on next report of device.",
			"name": "illuminance_calibration",
			"property": "illuminance_calibration",
			"type": "numeric"
		}, {
			"access": 2,
			"description": "Calibrates the device_temperature value (absolute offset), takes into effect on next report of device.",
			"name": "device_temperature_calibration",
			"property": "device_temperature_calibration",
			"type": "numeric"
		}],
		"supports_ota": true,
		"vendor": "Xiaomi"
	},
	"endpoints": {
		"1": {
			"bindings": [],
			"clusters": {
				"input": ["genBasic", "genPowerCfg", "genIdentify", "aqaraOpple"],
				"output": ["genIdentify", "genOta", "aqaraOpple"]
			},
			"configured_reportings": [],
			"scenes": []
		}
	},
	"friendly_name": "hello/testing",
	"ieee_address": "0x01234567890abcde",
	"interview_completed": true,
	"interviewing": false,
	"manufacturer": "LUMI",
	"model_id": "lumi.motion.ac02",
	"network_address": 32549,
	"power_source": "Battery",
	"software_build_id": "0.0.0_0005",
	"supported": true,
	"type": "EndDevice"
}]

]]

local json = require ('dkjson')

local ZBdevices = json.decode(zigbee2mqtt_bridge_devices)

print('Number of devices = '..#ZBdevices..'\n')

for idx=1, #ZBdevices do
    print ('friendly_name is '..ZBdevices[idx].friendly_name)
    print ('ieee_address is '..ZBdevices[idx].ieee_address..'\n')

    -- if not the coordinator
    if ZBdevices[idx].definition then
        print('Properties follow:')
        local foundMotion = false
        for idx2=1, #ZBdevices[idx].definition.exposes do
            devProperty = ZBdevices[idx].definition.exposes[idx2].property
            print (devProperty)
            
            if devProperty == 'occupancy' then
                foundMotion = true
            end
            
        end
        if foundMotion then print("\nDevice is a motion sensor") end
    end
end
return true

[therealdb]

@a-lurker said in Migrating Zwave & ZigBee stuff to MQTT - guidance needed.:

These are all set to: "http://"
SetPowerOffURL
SetPowerURL
SetToggleURL
SetUpdateMetersURL

Set them to skip and try again.

[a-lurker]

@therealdb

OK it works with them all set to skip although the documentation says blank, skip or http:// is OK . Might be useful to initialize them to skip.