openLuup email server

[akbooer]

Development Branch: 2021 Release 3.2

• MQTT server now supports all forms of the '#' wildcard (but not yet '+')
• ACE editor updated to 1.4.12

[akbooer]

Development Branch: 2021 Release 3.5b

• openLuup_IP:3480/relay/NNN?turn=... allows device Id or name for NNN
• openLuup_IP:3480/scene/NNN? allows scene Id or name for NNN
• openLuup console control panel for Shellies links to their native configuration pages
• MQTT disconnect error fixed

Examples of Shelly device control panels:

with scheduled timer pages:

[akbooer]

Development Branch: 2021 Release 3.11b

%(#cc0000)[This release includes a significant update for the monitoring and control of devices, and represents an intention to move away from the need for any UPnP Vera-style requests or polling.]

Using the built-in MQTT QoS 0 server, two new sets of PUBLISH topics may be sent by openLuup:

• topics named openLuup/update/device_no/short_service_id/var_name (where short_service_id is the useful part of the serviceId – ie. the last alphanumeric bit.) These are sent as the named variable changes value (ie. not if it is set to the same value.) The message text is simply the new string content of the variable.
• a single topic named openLuup/status is sent every few seconds (user definable) cycling in a round-robin way through all of the devices. The JSON-formatted text message contains the current status of all the variables in that particular device. This, then, is the 'push' replacement for long-polling with a /data_request?id=status HTTP request.

In addition, existing functionality allows simple Shelly-style HTTP requests to control switches and lights. These will be extended to dimmers and other controls in due course. Since Release 2021 3.5b you can use the requests:

• openLuup_IP:3480/relay/NNN?turn=[on/off/toggle] allows device Id or name for NNN
• openLuup_IP:3480/scene/NNN? allows scene Id or name for NNN

The new MQTT published messages may be controlled by setting parameters in Startup Lua:

  luup.attr_set ("openLuup.MQTT.Port", 1883)
  luup.attr_set ("openLuup.MQTT.PublishVariableUpdates", true) -- publish every variable update
  luup.attr_set ("openLuup.MQTT.PublishDeviceStatus", 2)       -- publish a single device status every N seconds (0 = never)

The JSON format of a single device status report, which is very compact, is that of nested tables indexed by strings: device number / short ServiceId / variable name; such that flattening the table appropriately would give the MQTT topic used in openLuup/update messages (aside from that prefix.) For example, for openLuup (device #2):

{"2":{
    "HaDevice1":{
      "CommFailure":"0",
      "CommFailureTime":"0"
    },
    "altui1":{
      "DisplayLine1":"8Mb, 0.1%cpu, 0.1days",
      "DisplayLine2":"[Home]"
    },
    "openLuup":{
      "CpuLoad":"0.1",
      "HouseMode":"1",
      "Memory_Mb":"7.6",
      "StartTime":"2021-03-11T14:53:02",
      "Uptime_Days":"0.06",
      "Version":"v21.3.11",
      "Vnumber":"210311"
    }
  }}

With a round-robin interval of 2 seconds, a moderately-sized setup of 150 devices would be cycled through in five minutes, providing a sanity check that no transient device variable updates have been missed.

Additionally, the MQTT server now publishes some of the $SYS/broker/ statistics, as defined by Mosquitto

Also included in this release is the openLuup console dimmer control change suggested by @rafale77.

[akbooer]

Development Branch: 2021 Release 3.17

The openLuup MQTT QoS 0 server now incorporates a UDP -> MQTT bridge (currently, uni-directional.)

This means that any machine which can send a UDP datagram (and, frankly, that should be everything) can publish to subscribers of the openLuup MQTT server without the need for any MQTT client library. The use of UDP means that there is very little overhead to sending the data, and that there is absolutely no possibility of blocking the host process. The UDP 'fire and forget' concept also meshes very well with the MQTT QoS 0 service level.

The UDP -> MQTT bridge is enabled at openLuup startup with the additional configuration parameter, which defines the listening port number:

luup.attr_set ("openLuup.MQTT.Bridge_UDP", 2883) -- UDP bridge PORT number

The UDP datagram format for publishing is:

TopicName/=/ApplicationMessage

By way of example, here's a tiny bit of Lua Startup code, which I've tested on a Vera:

local socket = require "socket"
local sock = socket.udp()
sock: setpeername("172.16.42.121", "2883"). -- send to openLuup IP and port
local pk = (luup.attr_get "PK_AccessPoint")
pk = "Vera-" .. pk: match "%d+"

function test_watch (d,s,v,o,n)
  local msg = table.concat ({pk,'update',d,s,v,'=',n}, '/')
  sock: send (msg)
end

local services = {   -- list any services you want to watch here
  "urn:micasaverde-com:serviceId:EnergyMetering1",
  "urn:upnp-org:serviceId:SwitchPower1",
 }

for _,sid in ipairs(services) do
  luup.variable_watch ("test_watch", sid)
end

This sets up a device variable watch for all variables in specific services in all devices. When one of those variables changes, a UDP datagram of the form:

Vera-PK_AccessPoint/update/dev_number/serviceId/var_name/=/value

is sent to the UDP -> MQTT bridge and converted into a PUBLISH. For example:

• topic: Vera-35104005/update/203/urn:micasaverde-com:serviceId:EnergyMetering1/KWH
• message: 18900.9060

Here's an MQTT Explorer view of my test system:

The updates are effectively instantaneous, and if Vera was in my long-term future I might write a new VeraBridge plugin to take advantage of this. I doubt that will happen, but I have plenty of other basic devices which could benefit from this transport mechanism. I may take that approach for an updated ZWay bridge. Future developments might include a bi-directional bridge, so that clients can also receive commands over UDP.

[akbooer]

Development Branch: 2021 Release 3.25b

• yet more checks around socket.send()
• error log lines now highlighted in %(#ff2600)[red] (for @rafale77)

[akbooer]

Development Branch: 2021 Release 3.26

In the presence of errors, the console log pages now show a red %(#ff2600)[Messages] button which toggles the message window to show just the error lines of the log...

[akbooer]

Development Branch: 2021 Release 3.28

• correct console "Unistall" label
• retain last command in console page
• server.SelectWait default to 0.1s (was 1.0)
• add Shelly 1/1PM device support
• include @rafale77 update of http_async to LuaSec v1

[akbooer]

TESTING Branch: 2021 Release 4.30

A rare release to the testing branch to evaluate the effectiveness of RapidJSON as an enhancement to the openLuup.json module and a full replacement for the venerable dkjson module, per @rafale77's suggestion here:

https://smarthome.community/topic/151/rapidjson-instead-of-cjson-and-dkjson

If installed, RapidJSON will become the json encoder/decoder for any code requiring the openLuup.json module, using a default option table for encode of:

{empty_table_as_array = true, sort_keys=true, pretty=true}

This provides compatibilty with the formatting of the native openLuup json module (although, regrettably, with a few more white space elements.)

Additionally, any code requiring the dkjson module gets, instead, the raw RapidJSON module (with no encode defaults.) This provides a significant performance boost to many plugins, including AltUI, AltHue, etc...

I'm keen, of course, to know if this causes any detremental effects, or if, perhaps, there should be some default encode options for dkjson substitute.

[akbooer]

Development Branch: 2021 Release 5.23

Significant changes and additions to the Data Historian:

• Variable graphing pages enables plotting of any of the archives or the in-memory cache
• Database editor to view variable values and commit changes to the database
• Support for multiple Graphite Carbon databases (complete replacement for DataYours)
• New console page to show rules for which variables are cached and/or archived
• Historian archive rules now independent (decoupled from old Whisper database .conf files)

In addition, the openLuup plugin device now has variables for solar Right Ascension (RA), Declination (DEC), Altitude (ALT), and Azimuth (AZ), which are updated every two minutes. Also a GetSolarCoords action which gives these values for any given time (defaults to now) and latitude/longitude (defaults to current location.) These calculations have always been done by openLuup to support scene times of sunrise/sunset, but are now available for use (negating the need for a separate plugin, eg. Heliotrope.)

[akbooer]

Development Branch: 2021 Release 6.21

Summer Solstice Edition

• Support for Shelly H&T child devices and battery level
• Tasmota prefixes and topics configurable in Lua Startup

If you have existing Shelly H&T devices, you will need to delete them, and restart, in order for them to be recreated (automatically on next update.) The battery level is shown in the parent ShellyH&T device in the usual way.

For Tasmota devices, you can change the default prefixes and topics recognised by the Tasmota bridge in Lua Startup:

luup.attr_set ("openLuup.Tasmota.Prefix", "tele, tasmota/tele, stat")
luup.attr_set ("openLuup.Tasmota.Topic", "SENSOR, STATE, RESULT, LWT")

The default values are as shown above.

––––––––––

As proof of the solstice, here's openLuup's DEC variable, showing that the solar declination has reached its maximum of nearly 23.5 degrees...

[akbooer]

Master Branch: 2021 Release 7.25

Long-overdue roll-up of the last year's development updates!

including...

• MQTT QoS 0 server
• UDP -> MQTT bridge
• Shelly device support
• Tasmota device support
• Highlighting of errors on log pages
• Simplified Shelly-like HTTP commands for device control
• Automatic substitution of JSON module with RapidJson, if available
• Data Historian improvements including better archive data plotting
• Solar coordinates available (for shades / lighting / etc...)

As ever, enormous thanks to those who provide ideas and feedback, or even criticism!

AK

[akbooer]

Master Branch: 2022 Release 11.22

Long-overdue roll-up of the this year's development updates!

Including:

• extended Shelly device support (H&T)
• update ace editor to 1.12.5
• numerous internal updates

Thanks to any/all still using this!

openLuup is still at the heart of my HA system, providing management of Shelly devices, Hue devices (through the excellent ALTHUE plugin by @amg0), a platform for native plugins, automation, &tc.

With the recent demise of my last Vera, and failure of one of my remaining two Zwave power meter readers, I have now finally removed all Zwave devices from my HA ecosystem, so am not able to provide any direct support on any Zwave issues.

[akbooer]

Development Branch: 2022 Release 12.2

• Correct MQTT response to Qos 1 and QoS 2 PUBLISH requests (PUBACK / PUBREC)
• thanks @Crille & @toggledbits
• initial release of Zigbee2MQTT bridge (thanks @a-lurker )

[akbooer]

Master Branch: 2023 Release 1.6

• REQUIRED update for Netatmo plugin authorization to function correctly
• correct long-standing issue with some LuaSocket library versions when finding own IP address
• remove leading and/or trailing blanks from room names (thanks @a-lurker)

[akbooer]

Development Branch: 2024 Release 2.8

• Add support for Shelly Plus H&T
• some internal restructuring for TCP servers

The Shelly Plus H&T device currently generates an error message in the log every time it disconnects:

openLuup.mqtt:: RECEIVE ERROR: (Fixed Header byte) closed tcp{client}: 0x564bc05a8a88

...but this doesn't affect the operation of the device or the system, and doesn't generally occur in the log very often, since, when battery powered, the device spends most of its time in sleep mode.

[akbooer]

Development Branch: 2024 Release 2.9

Fixes MQTT Fixed Header Byte error message.

[akbooer]

Development Branch: 2024 Release 2.26

• improve console startup
• update ACE editor to 1.32.6
• basic support for Shelly Plus UNI counter input

[akbooer]

Development Branch: 2024 Release 3.20

Spring Equinox Edition *

• the built-in MQTT server has been refactored and now fully supports both '#' and '+' wildcards
• the Shelly Bridge has been updated to take advantage of this for Plus devices
• minor improvements to scheduler's memory usage

---

*Spring equinox as demonstrated by openLuup's solar.DEC variable moving through zero from negative to positive values at 3:06 this morning:

[akbooer]

Development Branch: 2024 Release 4.5

• Significant refactoring of ShellyBridge handling of Gen 2+ devices.
• ShellyBridge device lists child firmware on its console page (see below)
• pretty() function now accessible as luup.openLuup.pretty()
• added MQTT retained topic $SYS/broker/version

[akbooer]

Development Branch: 2024 Release 4.8

Eclipse Edition

• further refactoring of Shelly Gen2+ devices
• fix missing variables for Shelly H&T
• support for Shelly Gen1 RGBW2 controller