openLuup email server

[akbooer]

Development Branch: 2020 Release 7.4

INDEPENDENCE DAY Edition

...nothing to do with the USA, but the day in the UK when Covid-19 restrictions are reduced, and more shops and businesses can open!

• added Required Files sub-page to the Plugins page group, showing which files/modules are required by plugins (and how many times.)

This is in preparation for being able to specify replacement modules without altering the plugin code itself. Applications include replacing dkjson with RapidJSON, and the like.

Not fixed:

• the log file customisation in Lua Startup (sorry @CatmanV2, next time.)

---

[akbooer]

Development Branch: 2020 Release 10.1

Thanks to today's influx of Shelly devices here, I've implemented a shorter data request for running scenes.

So instead of:

http://openLuupIP:3480/data_request?id=action&serviceId=urn:micasaverde-com:serviceId:HomeAutomationGateway1&action=RunScene&SceneNum=1

or even:

http://openLuupIP:3480/data_request?id=action&DeviceNum=2&serviceId=openLuup&action=RunScene&SceneNum=1

You can simply write:

http://openLuupIP:3480/data_request?id=run_scene&SceneNum=1

...much easier to program into all those new switched and relays!

[akbooer]

Development Branch: 2020 Release 10.2

Shelly-itis has hit hard... here is the beginnings of a Shelly-like API for all openLuup devices:

Switches:

Turn on / off / toggle with:

http://openLuupIP:3480/relay/NNN?turn=on
http://openLuupIP:3480/relay/NNN?turn=off
http://openLuupIP:3480/relay/NNN?turn=toggle

where NNN is the device number.

Scenes:

Run scenes with:

http://openLuupIP:3480/scene/NNN

where NNN is the scene id.

This makes programming Shelly i3 switches for actioning any openLuup device particularly easy, and consistent with controlling Shelly devices from openLuup (or any HTTP command.)

[akbooer]

Development Branch: 2020 Release 12.10

Fix long-standing error with retrieving icon files from Vera using VeraBridge GetVeraFiles action.

[akbooer]

Development Branch: 2021 Release 2.11

Two significant new developments available in BETA form:

1. integral MQTT server (broker)
2. integral Shelly device bridge

MQTT server

This server provides Quality of Service (QoS) 0: At most once delivery, only. That is (according to the specification): "The message is delivered according to the capabilities of the underlying network. No response is sent by the receiver and no retry is performed by the sender. The message arrives at the receiver either once or not at all."

IMHO, QoS 0 is perfectly adequate for almost any home automation application operating over an internal LAN. Messages are hugely unlikely to get lost somewhere.

From openLuup Lua Startup, Scene Lua, or plugins, you can directly publish and subscribe.

To publish:

local mqtt = luup.openLuup.mqtt
mqtt.publish ("My/Topic/Name", "My Application message")

To subscribe, you use a standard Luup request handler:

function MyMQTThandler (topic, message)
  -- your handler code here
end

luup.register_handler ("MyMQTThandler", "mqtt:My/Topic/Name")

You can register any number of handlers to different topics, or a single handler to many topics. Currently only one wildcard topic is allowed, '#', which subscribes you to all user messages.

The MQTT server has to be enabled in Lua startup with the line:

luup.attr_set ("openLuup.MQTT.Port", 1883)    -- choose any free port, you might not want to use this MQTT default

Shelly Bridge

The Shelly bridge is implemented in an entirely new way for openLuup bridges, requiring no configuration at all, and installing itself automatically when the first Shelly device announces itself via MQTT. Shelly devices get their configured name (or ID, if no name was set.)

The bridge functionality is in three parts:

• MQTT client – as described above, it creates announced devices and updates their variables. All published states are reflected in device variables. Some devices (eg. Shelly Swich v2.5) have specific child devices created for them too.
• ShellyBridge plugin – provides the standard Luup device interface, handling all the child device action calls.
• Shelly CGI interface for all openLuup devices – as described in release v20.10.2 above, providing a Shelly-like HTTP command structure for any compatible openLuup device to turn on/off/toggle and retrieve status, etc.

Currently only i3 and sw2.5 devices are supported.

Here's a screen shot of one of each of those on a test system. The i3 is represented as a scene controller. The sw2.5 has two BinarySwitch child devices:

Here are the variables in a sw2.5 parent device:

Here's the variables for one of the child switches:

This is all BETA, so very interested in reports (and logs) good, or bad.

[akbooer]

Development Branch: 2021 Release 2.18

MQTT server now supports login credentials.

To set Username and Password for MQTT connections use the following commands in Lua Startup:

luup.attr_set ("openLuup.MQTT.Username", "---username---")
luup.attr_set ("openLuup.MQTT.Password", "---password---")

[akbooer]

Development Branch: 2021 Release 3.2

• MQTT server now supports all forms of the '#' wildcard (but not yet '+')
• ACE editor updated to 1.4.12

[akbooer]

Development Branch: 2021 Release 3.5b

• openLuup_IP:3480/relay/NNN?turn=... allows device Id or name for NNN
• openLuup_IP:3480/scene/NNN? allows scene Id or name for NNN
• openLuup console control panel for Shellies links to their native configuration pages
• MQTT disconnect error fixed

Examples of Shelly device control panels:

with scheduled timer pages:

[akbooer]

Development Branch: 2021 Release 3.11b

%(#cc0000)[This release includes a significant update for the monitoring and control of devices, and represents an intention to move away from the need for any UPnP Vera-style requests or polling.]

Using the built-in MQTT QoS 0 server, two new sets of PUBLISH topics may be sent by openLuup:

• topics named openLuup/update/device_no/short_service_id/var_name (where short_service_id is the useful part of the serviceId – ie. the last alphanumeric bit.) These are sent as the named variable changes value (ie. not if it is set to the same value.) The message text is simply the new string content of the variable.
• a single topic named openLuup/status is sent every few seconds (user definable) cycling in a round-robin way through all of the devices. The JSON-formatted text message contains the current status of all the variables in that particular device. This, then, is the 'push' replacement for long-polling with a /data_request?id=status HTTP request.

In addition, existing functionality allows simple Shelly-style HTTP requests to control switches and lights. These will be extended to dimmers and other controls in due course. Since Release 2021 3.5b you can use the requests:

• openLuup_IP:3480/relay/NNN?turn=[on/off/toggle] allows device Id or name for NNN
• openLuup_IP:3480/scene/NNN? allows scene Id or name for NNN

The new MQTT published messages may be controlled by setting parameters in Startup Lua:

  luup.attr_set ("openLuup.MQTT.Port", 1883)
  luup.attr_set ("openLuup.MQTT.PublishVariableUpdates", true) -- publish every variable update
  luup.attr_set ("openLuup.MQTT.PublishDeviceStatus", 2)       -- publish a single device status every N seconds (0 = never)

The JSON format of a single device status report, which is very compact, is that of nested tables indexed by strings: device number / short ServiceId / variable name; such that flattening the table appropriately would give the MQTT topic used in openLuup/update messages (aside from that prefix.) For example, for openLuup (device #2):

{"2":{
    "HaDevice1":{
      "CommFailure":"0",
      "CommFailureTime":"0"
    },
    "altui1":{
      "DisplayLine1":"8Mb, 0.1%cpu, 0.1days",
      "DisplayLine2":"[Home]"
    },
    "openLuup":{
      "CpuLoad":"0.1",
      "HouseMode":"1",
      "Memory_Mb":"7.6",
      "StartTime":"2021-03-11T14:53:02",
      "Uptime_Days":"0.06",
      "Version":"v21.3.11",
      "Vnumber":"210311"
    }
  }}

With a round-robin interval of 2 seconds, a moderately-sized setup of 150 devices would be cycled through in five minutes, providing a sanity check that no transient device variable updates have been missed.

Additionally, the MQTT server now publishes some of the $SYS/broker/ statistics, as defined by Mosquitto

Also included in this release is the openLuup console dimmer control change suggested by @rafale77.

[akbooer]

Development Branch: 2021 Release 3.17

The openLuup MQTT QoS 0 server now incorporates a UDP -> MQTT bridge (currently, uni-directional.)

This means that any machine which can send a UDP datagram (and, frankly, that should be everything) can publish to subscribers of the openLuup MQTT server without the need for any MQTT client library. The use of UDP means that there is very little overhead to sending the data, and that there is absolutely no possibility of blocking the host process. The UDP 'fire and forget' concept also meshes very well with the MQTT QoS 0 service level.

The UDP -> MQTT bridge is enabled at openLuup startup with the additional configuration parameter, which defines the listening port number:

luup.attr_set ("openLuup.MQTT.Bridge_UDP", 2883) -- UDP bridge PORT number

The UDP datagram format for publishing is:

TopicName/=/ApplicationMessage

By way of example, here's a tiny bit of Lua Startup code, which I've tested on a Vera:

local socket = require "socket"
local sock = socket.udp()
sock: setpeername("172.16.42.121", "2883"). -- send to openLuup IP and port
local pk = (luup.attr_get "PK_AccessPoint")
pk = "Vera-" .. pk: match "%d+"

function test_watch (d,s,v,o,n)
  local msg = table.concat ({pk,'update',d,s,v,'=',n}, '/')
  sock: send (msg)
end

local services = {   -- list any services you want to watch here
  "urn:micasaverde-com:serviceId:EnergyMetering1",
  "urn:upnp-org:serviceId:SwitchPower1",
 }

for _,sid in ipairs(services) do
  luup.variable_watch ("test_watch", sid)
end

This sets up a device variable watch for all variables in specific services in all devices. When one of those variables changes, a UDP datagram of the form:

Vera-PK_AccessPoint/update/dev_number/serviceId/var_name/=/value

is sent to the UDP -> MQTT bridge and converted into a PUBLISH. For example:

• topic: Vera-35104005/update/203/urn:micasaverde-com:serviceId:EnergyMetering1/KWH
• message: 18900.9060

Here's an MQTT Explorer view of my test system:

The updates are effectively instantaneous, and if Vera was in my long-term future I might write a new VeraBridge plugin to take advantage of this. I doubt that will happen, but I have plenty of other basic devices which could benefit from this transport mechanism. I may take that approach for an updated ZWay bridge. Future developments might include a bi-directional bridge, so that clients can also receive commands over UDP.

[akbooer]

Development Branch: 2021 Release 3.25b

• yet more checks around socket.send()
• error log lines now highlighted in %(#ff2600)[red] (for @rafale77)

[akbooer]

Development Branch: 2021 Release 3.26

In the presence of errors, the console log pages now show a red %(#ff2600)[Messages] button which toggles the message window to show just the error lines of the log...

[akbooer]

Development Branch: 2021 Release 3.28

• correct console "Unistall" label
• retain last command in console page
• server.SelectWait default to 0.1s (was 1.0)
• add Shelly 1/1PM device support
• include @rafale77 update of http_async to LuaSec v1

[akbooer]

TESTING Branch: 2021 Release 4.30

A rare release to the testing branch to evaluate the effectiveness of RapidJSON as an enhancement to the openLuup.json module and a full replacement for the venerable dkjson module, per @rafale77's suggestion here:

https://smarthome.community/topic/151/rapidjson-instead-of-cjson-and-dkjson

If installed, RapidJSON will become the json encoder/decoder for any code requiring the openLuup.json module, using a default option table for encode of:

{empty_table_as_array = true, sort_keys=true, pretty=true}

This provides compatibilty with the formatting of the native openLuup json module (although, regrettably, with a few more white space elements.)

Additionally, any code requiring the dkjson module gets, instead, the raw RapidJSON module (with no encode defaults.) This provides a significant performance boost to many plugins, including AltUI, AltHue, etc...

I'm keen, of course, to know if this causes any detremental effects, or if, perhaps, there should be some default encode options for dkjson substitute.

[akbooer]

Development Branch: 2021 Release 5.23

Significant changes and additions to the Data Historian:

• Variable graphing pages enables plotting of any of the archives or the in-memory cache
• Database editor to view variable values and commit changes to the database
• Support for multiple Graphite Carbon databases (complete replacement for DataYours)
• New console page to show rules for which variables are cached and/or archived
• Historian archive rules now independent (decoupled from old Whisper database .conf files)

In addition, the openLuup plugin device now has variables for solar Right Ascension (RA), Declination (DEC), Altitude (ALT), and Azimuth (AZ), which are updated every two minutes. Also a GetSolarCoords action which gives these values for any given time (defaults to now) and latitude/longitude (defaults to current location.) These calculations have always been done by openLuup to support scene times of sunrise/sunset, but are now available for use (negating the need for a separate plugin, eg. Heliotrope.)

[akbooer]

Development Branch: 2021 Release 6.21

Summer Solstice Edition

• Support for Shelly H&T child devices and battery level
• Tasmota prefixes and topics configurable in Lua Startup

If you have existing Shelly H&T devices, you will need to delete them, and restart, in order for them to be recreated (automatically on next update.) The battery level is shown in the parent ShellyH&T device in the usual way.

For Tasmota devices, you can change the default prefixes and topics recognised by the Tasmota bridge in Lua Startup:

luup.attr_set ("openLuup.Tasmota.Prefix", "tele, tasmota/tele, stat")
luup.attr_set ("openLuup.Tasmota.Topic", "SENSOR, STATE, RESULT, LWT")

The default values are as shown above.

––––––––––

As proof of the solstice, here's openLuup's DEC variable, showing that the solar declination has reached its maximum of nearly 23.5 degrees...

[akbooer]

Master Branch: 2021 Release 7.25

Long-overdue roll-up of the last year's development updates!

including...

• MQTT QoS 0 server
• UDP -> MQTT bridge
• Shelly device support
• Tasmota device support
• Highlighting of errors on log pages
• Simplified Shelly-like HTTP commands for device control
• Automatic substitution of JSON module with RapidJson, if available
• Data Historian improvements including better archive data plotting
• Solar coordinates available (for shades / lighting / etc...)

As ever, enormous thanks to those who provide ideas and feedback, or even criticism!

AK

[akbooer]

Master Branch: 2022 Release 11.22

Long-overdue roll-up of the this year's development updates!

Including:

• extended Shelly device support (H&T)
• update ace editor to 1.12.5
• numerous internal updates

Thanks to any/all still using this!

openLuup is still at the heart of my HA system, providing management of Shelly devices, Hue devices (through the excellent ALTHUE plugin by @amg0), a platform for native plugins, automation, &tc.

With the recent demise of my last Vera, and failure of one of my remaining two Zwave power meter readers, I have now finally removed all Zwave devices from my HA ecosystem, so am not able to provide any direct support on any Zwave issues.

[akbooer]

Development Branch: 2022 Release 12.2

• Correct MQTT response to Qos 1 and QoS 2 PUBLISH requests (PUBACK / PUBREC)
• thanks @Crille & @toggledbits
• initial release of Zigbee2MQTT bridge (thanks @a-lurker )

[akbooer]

Master Branch: 2023 Release 1.6

• REQUIRED update for Netatmo plugin authorization to function correctly
• correct long-standing issue with some LuaSocket library versions when finding own IP address
• remove leading and/or trailing blanks from room names (thanks @a-lurker)