Truncated web pages using wireguard over long distances

[akbooer]

A long while ago (May, 2015) I wrote my 2000-th post on another forum: openLuup - running unmodified plugins on any machine. Here’s the gist of it:

---

...I want to work in a more open and stable [Vera] environment...

...All would be solved if Luup was open source and could be run on the plethora of cheap and reliable hardware available today. But it’s not. But we could get something like that effect if we engineered a sufficient subset of Luup to run on such a platform. Could it be done? What would we need?

1. UI
2. scheduler
3. web server
4. Luup compatible API
5. Device and Implementation xml file reader
6. Zwave bridge to Vera
7. runs most plugins without modification

What we wouldn’t need is UPnP.

What have we (nearly) got already?

• We have, courtesy of @amg0, the most excellent AltUI: Alternate UI to UI7, and that, I think, is probably the hardest one to do in the above list.
• Items 2 - 5, and 7, I’ve prototyped, in pure Lua, and posted elsewhere: DataYours on Raspberry Pi, running selected plugins unmodified, including: DataYours, EventWatcher, Netatmo, RBLuaTest, altUI. See screenshot attached.

Is it worth the effort? Probably not. Will I pursue this quest? Yes.

---

openLuup was the result.

[akbooer]

Development Branch: 2020 Release 5.1

• Startup log shows JSON module version information (thanks @a-lurker)

If you have installed Cjson then the startup log should include:

2020-05-01 13:00:56.780 openLuup.json:: version 2020.04.16 @akbooer
2020-05-01 13:00:56.780 openLuup.init:: using Cjson 2.1.0 for fast decoding

If not, then:

2020-05-01 12:54:47.675 openLuup.json:: version 2020.04.16 @akbooer
2020-05-01 12:54:47.675 openLuup.init:: Cjson not installed - using openLuup.json.Lua.decode() instead

[akbooer]

Development Branch: 2020 Release 5.6

• I_openLuupCamera1 mods:
• add username attribute
• add password attribute
• add URL variable
• add DirectStreamingURL variable
• thanks @prophead

[akbooer]

Development Branch: 2020 Release 5.10

• native image panel for openLuupCamera1 device
• category 6 for ditto
• file counts in Luup files page menu

The openLuup console Luup Files page offers a very fast way to browse and view all the Luup device files that the system knows about. A click on the file name opens a Viewer on the file itself. The file filter menu now shows a count of each file type. I seem to have a total of 772 such files on my development system...

[akbooer]

Development Branch: 2020 Release 5.20

• fix for cjson.null in decoded structures
• thanks @reneboer (and @mrFarmer !)

As usual, an earlier update breaks something. This probably wouldn't have been spotted for a while, since it doesn't impact VeraBridge or the ZWave bridge, but thanks to @reneboer's vigilance, it's found and fixed.

A little more detail. The Cjson module decodes Json null as the private data structure cjson.null. The following code snippet shows the problem:

local json = require "openLuup.json"
local J = json.encode {1, nil, 3}
local L = json.Lua.decode(J)
local C = json.C.decode(J)

print("JSON", J)
print("Lua", pretty(L))
print("C", pretty(C))

writes the output:

JSON 	[1,null,3]
Lua 	{1,nil,3}
C 	{1,userdata: (nil),3}

However, the default openLuup json.decode() function has now been fixed, when it's using the Cjson module:

local D = json.decode(J)
print ("json.decode [using Cjson]", pretty(D))

now gives:

json.decode [using Cjson] 	{1,nil,3}

Kudos again to @reneboer and @mrFarmer !

[akbooer]

Master Branch: 2020 Release 5.22

%(#d93b50)[5th Aniiversary edition!!]

Yes, it's five years since openLuup was unveiled to the world in this post:

http://forum.micasaverde.com/index.php/topic,32315.465.html

...so I thought that today I'd roll up all the changes to date into the master branch.

I suppose I should also be setting out a roadmap. At the risk of it becoming a commitment, let's just say that, for the moment, it's just a list of incremental development ideas and some blue sky thinking:

• ZWay plugin integrated in openLuup distribution (as are AltAppStore and VeraBridge)
• native scene triggers – not the same as Vera, not the same as AltUI variable watches, but definable in the openLuup console.
• utilities for managing Data Historian archives
• better task & job messages

and then...

• fully asynchronous I/O
• enhanced HTTP server with data compression (perhaps using third-party web server?)
• Ajax calls for asynchronous updates of some console pages (Devices, Scenes, ...)
• ...

Ideas always welcomed!

[akbooer]

Development Branch: 2020 Release 6.28

Implemented some recent suggestions to assist code/plugin development in the openLuup environment.

• the console System > All Globals page now starts with an entry for the shared environment (Startup / Shutdown / Test / Scenes...)
• there is a link from the above item to the Lua Globals page within the Lua Code page group, which expands the content of non-standard (ie. user-created) global Lua tables in the shared environment.
• a new system variable luup.openLuup.cpu_table contains the running total of CPU times (with sub-millisecond resolution) for all plugins.

The last item above may be used to profile run times for all the plugins:

local T = luup.openLuup.cpu_table    -- yes, this is a table, not a function call
print (T)

produces an output like:

      (s.ms)      [#] device name
       0.010      [2] openLuup
       0.650      [3] Alternate UI
       0.010      [4] Alternate App Store
       8.300     [13] Vera Edge
       0.040     [16] BBB Sonos (Study)
       2.690     [17] Vera 0
       3.100     [18] Vera 1
       1.470     [20] Netatmo
       0.040     [42] DarkSky Weather
       7.140     [48] Philips hue
       4.140     [70] Studio hue

but it's also an object which you may use to calculate run-time deltas. Hence:

local T0 = luup.openLuup.cpu_table

--  wait a bit

local T1 = luup.openLuup.cpu_table 

print (T1 - T0)

will show you the elapsed CPU time for all the plugins.

      (s.ms)      [#] device name
       0.000      [2] openLuup
       0.020      [3] Alternate UI
       0.000      [4] Alternate App Store
       1.440     [13] Vera Edge
       0.000     [16] BBB Sonos (Study)
       0.080     [17] Vera 0
       0.070     [18] Vera 1
       0.560     [20] Netatmo
       0.000     [42] DarkSky Weather
       1.460     [48] Philips hue
       0.770     [70] Studio hue

My thanks to @a-lurker for being a catalyst for these changes.

[akbooer]

Development Branch: 2020 Release 6.29

Following yesterday's changes in v20.6.28, "wall-clock" time is now added to the system profiling instrumentation:

• new plugin device attribute wall(s) contains the running total of wall-clock time (with sub-millisecond resolution)
• new system variable luup.openLuup.wall_table contains the wall-clock times (with sub-millisecond resolution) for all plugins
• openLuup console Scheduler > Plugins page shows cpu and wall-clock times and their ratio.

The final item above should be a good indicator of when ( >> 1) a plugin is taking much more elapsed time than the CPU that it is actually using – diagnostic of blocking socket I/O or wanton use of luup.sleep().

[akbooer]

Development Branch: 2020 Release 7.4

INDEPENDENCE DAY Edition

...nothing to do with the USA, but the day in the UK when Covid-19 restrictions are reduced, and more shops and businesses can open!

• added Required Files sub-page to the Plugins page group, showing which files/modules are required by plugins (and how many times.)

This is in preparation for being able to specify replacement modules without altering the plugin code itself. Applications include replacing dkjson with RapidJSON, and the like.

Not fixed:

• the log file customisation in Lua Startup (sorry @CatmanV2, next time.)

---

[akbooer]

Development Branch: 2020 Release 10.1

Thanks to today's influx of Shelly devices here, I've implemented a shorter data request for running scenes.

So instead of:

http://openLuupIP:3480/data_request?id=action&serviceId=urn:micasaverde-com:serviceId:HomeAutomationGateway1&action=RunScene&SceneNum=1

or even:

http://openLuupIP:3480/data_request?id=action&DeviceNum=2&serviceId=openLuup&action=RunScene&SceneNum=1

You can simply write:

http://openLuupIP:3480/data_request?id=run_scene&SceneNum=1

...much easier to program into all those new switched and relays!

[akbooer]

Development Branch: 2020 Release 10.2

Shelly-itis has hit hard... here is the beginnings of a Shelly-like API for all openLuup devices:

Switches:

Turn on / off / toggle with:

http://openLuupIP:3480/relay/NNN?turn=on
http://openLuupIP:3480/relay/NNN?turn=off
http://openLuupIP:3480/relay/NNN?turn=toggle

where NNN is the device number.

Scenes:

Run scenes with:

http://openLuupIP:3480/scene/NNN

where NNN is the scene id.

This makes programming Shelly i3 switches for actioning any openLuup device particularly easy, and consistent with controlling Shelly devices from openLuup (or any HTTP command.)

[akbooer]

Development Branch: 2020 Release 12.10

Fix long-standing error with retrieving icon files from Vera using VeraBridge GetVeraFiles action.

[akbooer]

Development Branch: 2021 Release 2.11

Two significant new developments available in BETA form:

1. integral MQTT server (broker)
2. integral Shelly device bridge

MQTT server

This server provides Quality of Service (QoS) 0: At most once delivery, only. That is (according to the specification): "The message is delivered according to the capabilities of the underlying network. No response is sent by the receiver and no retry is performed by the sender. The message arrives at the receiver either once or not at all."

IMHO, QoS 0 is perfectly adequate for almost any home automation application operating over an internal LAN. Messages are hugely unlikely to get lost somewhere.

From openLuup Lua Startup, Scene Lua, or plugins, you can directly publish and subscribe.

To publish:

local mqtt = luup.openLuup.mqtt
mqtt.publish ("My/Topic/Name", "My Application message")

To subscribe, you use a standard Luup request handler:

function MyMQTThandler (topic, message)
  -- your handler code here
end

luup.register_handler ("MyMQTThandler", "mqtt:My/Topic/Name")

You can register any number of handlers to different topics, or a single handler to many topics. Currently only one wildcard topic is allowed, '#', which subscribes you to all user messages.

The MQTT server has to be enabled in Lua startup with the line:

luup.attr_set ("openLuup.MQTT.Port", 1883)    -- choose any free port, you might not want to use this MQTT default

Shelly Bridge

The Shelly bridge is implemented in an entirely new way for openLuup bridges, requiring no configuration at all, and installing itself automatically when the first Shelly device announces itself via MQTT. Shelly devices get their configured name (or ID, if no name was set.)

The bridge functionality is in three parts:

• MQTT client – as described above, it creates announced devices and updates their variables. All published states are reflected in device variables. Some devices (eg. Shelly Swich v2.5) have specific child devices created for them too.
• ShellyBridge plugin – provides the standard Luup device interface, handling all the child device action calls.
• Shelly CGI interface for all openLuup devices – as described in release v20.10.2 above, providing a Shelly-like HTTP command structure for any compatible openLuup device to turn on/off/toggle and retrieve status, etc.

Currently only i3 and sw2.5 devices are supported.

Here's a screen shot of one of each of those on a test system. The i3 is represented as a scene controller. The sw2.5 has two BinarySwitch child devices:

Here are the variables in a sw2.5 parent device:

Here's the variables for one of the child switches:

This is all BETA, so very interested in reports (and logs) good, or bad.

[akbooer]

Development Branch: 2021 Release 2.18

MQTT server now supports login credentials.

To set Username and Password for MQTT connections use the following commands in Lua Startup:

luup.attr_set ("openLuup.MQTT.Username", "---username---")
luup.attr_set ("openLuup.MQTT.Password", "---password---")

[akbooer]

Development Branch: 2021 Release 3.2

• MQTT server now supports all forms of the '#' wildcard (but not yet '+')
• ACE editor updated to 1.4.12

[akbooer]

Development Branch: 2021 Release 3.5b

• openLuup_IP:3480/relay/NNN?turn=... allows device Id or name for NNN
• openLuup_IP:3480/scene/NNN? allows scene Id or name for NNN
• openLuup console control panel for Shellies links to their native configuration pages
• MQTT disconnect error fixed

Examples of Shelly device control panels:

with scheduled timer pages:

[akbooer]

Development Branch: 2021 Release 3.11b

%(#cc0000)[This release includes a significant update for the monitoring and control of devices, and represents an intention to move away from the need for any UPnP Vera-style requests or polling.]

Using the built-in MQTT QoS 0 server, two new sets of PUBLISH topics may be sent by openLuup:

• topics named openLuup/update/device_no/short_service_id/var_name (where short_service_id is the useful part of the serviceId – ie. the last alphanumeric bit.) These are sent as the named variable changes value (ie. not if it is set to the same value.) The message text is simply the new string content of the variable.
• a single topic named openLuup/status is sent every few seconds (user definable) cycling in a round-robin way through all of the devices. The JSON-formatted text message contains the current status of all the variables in that particular device. This, then, is the 'push' replacement for long-polling with a /data_request?id=status HTTP request.

In addition, existing functionality allows simple Shelly-style HTTP requests to control switches and lights. These will be extended to dimmers and other controls in due course. Since Release 2021 3.5b you can use the requests:

• openLuup_IP:3480/relay/NNN?turn=[on/off/toggle] allows device Id or name for NNN
• openLuup_IP:3480/scene/NNN? allows scene Id or name for NNN

The new MQTT published messages may be controlled by setting parameters in Startup Lua:

  luup.attr_set ("openLuup.MQTT.Port", 1883)
  luup.attr_set ("openLuup.MQTT.PublishVariableUpdates", true) -- publish every variable update
  luup.attr_set ("openLuup.MQTT.PublishDeviceStatus", 2)       -- publish a single device status every N seconds (0 = never)

The JSON format of a single device status report, which is very compact, is that of nested tables indexed by strings: device number / short ServiceId / variable name; such that flattening the table appropriately would give the MQTT topic used in openLuup/update messages (aside from that prefix.) For example, for openLuup (device #2):

{"2":{
    "HaDevice1":{
      "CommFailure":"0",
      "CommFailureTime":"0"
    },
    "altui1":{
      "DisplayLine1":"8Mb, 0.1%cpu, 0.1days",
      "DisplayLine2":"[Home]"
    },
    "openLuup":{
      "CpuLoad":"0.1",
      "HouseMode":"1",
      "Memory_Mb":"7.6",
      "StartTime":"2021-03-11T14:53:02",
      "Uptime_Days":"0.06",
      "Version":"v21.3.11",
      "Vnumber":"210311"
    }
  }}

With a round-robin interval of 2 seconds, a moderately-sized setup of 150 devices would be cycled through in five minutes, providing a sanity check that no transient device variable updates have been missed.

Additionally, the MQTT server now publishes some of the $SYS/broker/ statistics, as defined by Mosquitto

Also included in this release is the openLuup console dimmer control change suggested by @rafale77.

[akbooer]

Development Branch: 2021 Release 3.17

The openLuup MQTT QoS 0 server now incorporates a UDP -> MQTT bridge (currently, uni-directional.)

This means that any machine which can send a UDP datagram (and, frankly, that should be everything) can publish to subscribers of the openLuup MQTT server without the need for any MQTT client library. The use of UDP means that there is very little overhead to sending the data, and that there is absolutely no possibility of blocking the host process. The UDP 'fire and forget' concept also meshes very well with the MQTT QoS 0 service level.

The UDP -> MQTT bridge is enabled at openLuup startup with the additional configuration parameter, which defines the listening port number:

luup.attr_set ("openLuup.MQTT.Bridge_UDP", 2883) -- UDP bridge PORT number

The UDP datagram format for publishing is:

TopicName/=/ApplicationMessage

By way of example, here's a tiny bit of Lua Startup code, which I've tested on a Vera:

local socket = require "socket"
local sock = socket.udp()
sock: setpeername("172.16.42.121", "2883"). -- send to openLuup IP and port
local pk = (luup.attr_get "PK_AccessPoint")
pk = "Vera-" .. pk: match "%d+"

function test_watch (d,s,v,o,n)
  local msg = table.concat ({pk,'update',d,s,v,'=',n}, '/')
  sock: send (msg)
end

local services = {   -- list any services you want to watch here
  "urn:micasaverde-com:serviceId:EnergyMetering1",
  "urn:upnp-org:serviceId:SwitchPower1",
 }

for _,sid in ipairs(services) do
  luup.variable_watch ("test_watch", sid)
end

This sets up a device variable watch for all variables in specific services in all devices. When one of those variables changes, a UDP datagram of the form:

Vera-PK_AccessPoint/update/dev_number/serviceId/var_name/=/value

is sent to the UDP -> MQTT bridge and converted into a PUBLISH. For example:

• topic: Vera-35104005/update/203/urn:micasaverde-com:serviceId:EnergyMetering1/KWH
• message: 18900.9060

Here's an MQTT Explorer view of my test system:

The updates are effectively instantaneous, and if Vera was in my long-term future I might write a new VeraBridge plugin to take advantage of this. I doubt that will happen, but I have plenty of other basic devices which could benefit from this transport mechanism. I may take that approach for an updated ZWay bridge. Future developments might include a bi-directional bridge, so that clients can also receive commands over UDP.

[akbooer]

Development Branch: 2021 Release 3.25b

• yet more checks around socket.send()
• error log lines now highlighted in %(#ff2600)[red] (for @rafale77)

[akbooer]

Development Branch: 2021 Release 3.26

In the presence of errors, the console log pages now show a red %(#ff2600)[Messages] button which toggles the message window to show just the error lines of the log...