SmartHome Community

Buxton

@archers Very Cool gadget. I soldered to the flash pins, which given the size of the contacts was challenging, but managed to make it work. Now waiting for my BLE sensors to arrive.

Energy Monitoring built into the dual relay!

Buxton

@archers

I was seeing similar network issues and also came to the conclusion that the socket library was most likely at fault. My solution is to use Mosquitto as my main broker, which accepts all MQTT traffic (topic # in 0) with all my MQTT devices pointing to it, and then Mosquitto filters push traffic to openLuup. Below is my config file that displays the filters:

allow_anonymous true
password_file /mosquitto/data/PW.txt
listener 1883
connection openLuup
address 127.0.0.1:1882
topic tele/# out 
topic stat/# out 
topic BlueIris/# out 
topic # in 0
cleansession false
notifications true
username *****
password *******
bridge_protocol_version mqttv311
try_private false
log_timestamp true
log_timestamp_format %Y-%m-%d--T_%H:%M:%S

As you can see, Mosquitto runs on the same server as openLuup. It is started by a docker compose file. The config filters eliminated the network errors on openLuup and my openLuup install now runs for days on end without any errors at all. I also have HA running on the same server via docker compose, though I only use it for its Hacs Alexa integration. I pipe my Alexa calls to HA using an HA token and a crude plugin that I wrote. I have not found a use for HA outside of openLuup yet, though there are some interesting integrations I will eventually try out.

As regards MQTT I don't think you need to worry about network traffic so much as mqtt is an extremely light protocol, at least in so far as compared to cameras and hi-def wireless etc ( I have a bunch of these high bandwidth devices on my network in their own subnets). I have found that the thing that tends to bog down is the lua socket function and as long as you limit its connections, you will probably alleviate most of the network problems.

Nginx is one of the best web servers available, specializing in load balancing millions of connections, and from what I've read, it is written in Lua. Which suggests that the lua socket module itself is causing the network issues as Nginx most likely rolled their own network library.

Buxton

@akbooer I think I found the culprit causing the random connection disconnects to Mosquitto. The LWT payload for a given device is a simple string, which doesn't seem to decode in the json decode call. So I added the below code to send a json string to the decoder. The errors have disappeared and I now see the LWT variable in the service variables. The variable reads "LWT : Online"

--line 165
local valid = {SENSOR = true, STATE = true, RESULT = true, LWT = true}

--line 172
  -- begin code
  local info, err = json.decode (message)
   
  if message then
	if not info then  -- json did not decode because of single string parameter
		message = '{' .. '"' .. mtype .. '"' .. ':' .. '"' .. message .. '"' .. '}'
	end
  end
  -- end code

  
  local info, err = json.decode (message)

This is probably not the way you would handle the error, but it does work.

Buxton

@akbooer Tasmota energy sensor

{
  "Time": "2021-03-28T21:51:01",
  "ENERGY": {
    "TotalStartTime": "2020-06-07T00:10:43",
    "Total": 2356.063,
    "Yesterday": 7.056,
    "Today": 6.459,
    "Period": 24,
    "Power": 285,
    "ApparentPower": 302,
    "ReactivePower": 99,
    "Factor": 0.95,
    "Voltage": 123,
    "Current": 2.453
  }
}

Buxton

No, you can still configure UDP. You just need login credentials now.

If I have a few moments, I will try to set up one of my Pi machines this weekend with an instance of Mosquitto. My HA server that runs my docker Mosquitto is headless, and I run Ubuntu server, so command line captures of packets are just a drag. My Pi has an hdmi port so I should be able to load the gui version of wireshark, and then test/capture the traffic between the two instances. I will let you know.

Buxton

@rafale77 hey Rafale,

I went down the very same road a while back and threw in towel because the polling by the MQTT plugin created CPU drags that stopped openLuup from functioning "reliably". The instability was also in part because I use two other must-have plugins that rely on polling, and I imagine that the combination of the three was creating a scenario that caused intermittent failures. And I too ended up implementing MQTT in Home Assistant and then using RealDB's virtual HTTP plugin to send commands to my WiFi devices--albeit not knowing the status of the devices in openLuup after the send.

I'm looking at RigPapa's socket proxy and WebSocket plugins to see if I can transform my polling plugins to Async. The MQTT plugin is too complex for me to convert though, so if you take a crack at it, and are successful, I would very much appreciate you publishing your results, as MQTT is becoming a must for me.

GitHub - toggledbits/sockproxyd: A socket proxy for Vera/openLuup systems

A socket proxy for Vera/openLuup systems. Contribute to toggledbits/sockproxyd development by creating an account on GitHub.

GitHub - toggledbits/LuWS: A WebSocket client implementation for Luup (Vera and openLuup) systems, with optional async receive (responsive)

A WebSocket client implementation for Luup (Vera and openLuup) systems, with optional async receive (responsive) - GitHub - toggledbits/LuWS: A WebSocket client implementation for Luup (Vera and op...

Buxton

@akbooer I hate to add more to the pile... but I'm still seeing a receive error for the connection to mosquitto. openLuup 2021.04.29b

Here's the log error:

2021-05-01 14:22:38.815   openLuup.io.server:: MQTT:1882 connection closed  tcp{client}: 0x5579919e9c58
2021-05-01 14:22:38.816   openLuup.mqtt:: RECEIVE ERROR: closed tcp{client}: 0x5579919e9c58
2021-05-01 14:22:43.935   luup.io.incoming:: bytes received: 51, status: OK tcp{client}: 0x557990e7f528
2021-05-01 14:22:48.435   luup.io.incoming:: bytes received: 51, status: OK tcp{client}: 0x557990e7f528
2021-05-01 14:22:49.763   luup.variable_set:: 10181.urn:micasaverde-com:serviceId:EnergyMetering1.KWHReading was: 1619904116 now: 1619904168 #hooks:0
2021-05-01 14:22:50.158   openLuup.io.server:: HTTP:3480 connection closed openLuup.server.receive closed tcp{client}: 0x557991971ff8
2021-05-01 14:22:53.834   openLuup.io.server:: MQTT:1882 connection from 127.0.0.1 tcp{client}: 0x5579920dc0b8
2021-05-01 14:22:53.834   openLuup.mqtt:: client is in ERROR empty
2021-05-01 14:22:53.834   openLuup.mqtt:: credentials is in ERROR empty
2021-05-01 14:22:53.834   openLuup.mqtt:: subscriptions is in ERROR empty

using the below error trapping in function "MQTTservlet"

  local function MQTTservlet (client)
  
	if client == nil then
	  _log ("client is in ERROR nil")
	else
	  	if table.concat(client) == "" then
		_log ("client is in ERROR empty")
		else
	   _log (table.concat {"MQTT ERROR: ", table.concat(client)})
	    end
	end 
	
	if credentials == nil then
	  _log ("credentials is in ERROR nil")
	else
	  	if table.concat(credentials) == "" then
		_log ("credentials is in ERROR empty")
		else
	   _log (table.concat {"MQTT ERROR: ", table.concat(credentials)})
	    end
	end 
	
	if subscriptions == nil then
	  _log ("subscriptions is in ERROR nil")
	else
	  	if table.concat(subscriptions) == "" then
		_log ("subscriptions is in ERROR empty")
		else
	    _log (table.concat {"MQTT ERROR: ", table.concat(subscriptions)})
	    end
	end
	
    return function () incoming (client, credentials, subscriptions) end
  end

I can't find a deeper layer in the stack where I can trap for the incoming message to see what's in the message that is throwing the error. As near as I can tell, if openLuup tries to connect to a running mosquitto instance, then it fails to see the topics and messages, and passes empty--but not nil--strings when the servlet interface sees incoming bytes.

If I restart mosquitto, openLuup then sees the topics and messages and the error messages stop--and the connection to mosquitto remains stable.

This behavior does not occur when I aim an IOT device directly at openLuup--in that the connection to the device always resumes when openLuup reloads--in other words, I don't need to restart the IOT device to enable the connection.

Buxton

@akbooer Below is the relevant output of a typical packet between mosquitto and a mosquitto bridged instance. In this case, mosquitto is sending update data to the bridge regarding a tasmota device/switch I use to remotely reboot my Vera. The format is definitely MQTT 3.1 and not 5.0, as 5.0 would not parse correctly in the wireshark viewer. The data payload is at the top of the window as Wireshark will truncate long messages. I can PM you the entire capture as it's not much, but may contain technical info that's best kept private. Let me know.

I'll try to capture some traffic between openLuup and mosquitto later.

{"Version":"9.1.0(tasmota)","BuildDateTime":"2020-11-07T11:57:45","Module or Template":"Gosund-WP5","RestartReason":"Software/System restart","Uptime":"6T05:50:22","Hostname":"power_MainVera-0278","IPAddress":"10.17.2.33","RSSI":"100","Signal (dBm)":"-17","WiFi LinkCount":5,"WiFi Downtime":"0T00:00:10","MqttCount":14,"LoadAvg":19}



Frame 9: 433 bytes on wire (3464 bits), 433 bytes captured (3464 bits) on interface eth0, id 0
Ethernet II, Src: Advansus_0a:8c:3a (00:19:0f:0a:8c:3a), Dst: 96:62:08:fb:22:8a (96:62:08:fb:22:8a)
Internet Protocol Version 4, Src: 10.17.2.41, Dst: 10.17.2.110
Transmission Control Protocol, Src Port: 40664, Dst Port: 1882, Seq: 3, Ack: 3, Len: 367
MQ Telemetry Transport Protocol, Publish Message
    Header Flags: 0x30, Message Type: Publish Message, QoS Level: At most once delivery (Fire and Forget)
        0011 .... = Message Type: Publish Message (3)
        .... 0... = DUP Flag: Not set
        .... .00. = QoS Level: At most once delivery (Fire and Forget) (0)
        .... ...0 = Retain: Not set
    Msg Len: 364
    Topic Length: 30
    Topic: tele/power_MainVera/HASS_STATE
    Message [truncated--see above]: {"Version":"9.1.0(tasmota)","BuildDateTime":"2020-11-07T11:57:45","Module or Template":"Gosund-WP5","RestartReason":"Software/System restart","Uptime":"6T05:50:22","Hostname":"power_MainVera-0278","IPAddress":"10.17.2

Buxton

@buxton In the above, I'm seeing an extra closing right hand bracket in the JSON string.

Buxton

I've been waiting for something like this. Very cool. Thx for the post

Buxton

@toggledbits @akbooer Sorry, I will try to be clearer.
" I don't know what "assigning the luup log function to a table" means"

In many of Patrick's plugins, he uses a global lua table to hold plugin data and functions. He assigns shorthand names that call common luup functions, then stores those truncated names in the table. Then the first class functions are called from the table. I believe it is this abstraction that is contributing to log entries that show nil values, as direct calls of "luup.log" do not show nil values. The technique should work in openluup, and most often it does, but sometimes it doesn't.

Here's an example from the log where it does not work:

2022-01-15 14:13:12.551   luup.variable_set:: 258.urn:micasaverde-com:serviceId:HaDevice1.CommFailure was: 0 now: 0 #hooks:0
2022-01-15 14:13:12.551   luup.variable_set:: 258.urn:micasaverde-com:serviceId:HaDevice1.CommFailureTime was: 0 now: 0 #hooks:0
2022-01-15 14:13:12.551   luup_log:149: Reactor: Starting nil (#nil)
2022-01-15 14:13:12.551   luup.variable_set:: 149.urn:toggledbits-com:serviceId:Reactor.Message was: Starting Unused1 now: Starting Basic Functions #hooks:0
2022-01-15 14:13:12.552   luup.variable_set:: 265.urn:toggledbits-com:serviceId:ReactorSensor.Invert was: nil now:  #hooks:0
2022-01-15 14:13:12.552   luup.variable_set:: 265.urn:toggledbits-com:serviceId:ReactorSensor.Invert was:  now: nil #hooks:0

In the log entry "luup_log:149: Reactor: Starting nil (#nil)" the text of the log entry is there "Reactor: Starting", but the variable following the text is nil. This is what I meant by "picking up on variables". The log entries are being displayed, but anything that is a variable is showing as nil. The variables themselves are not actually nil because I can see from program execution that they are doing the job that they're supposed to be doing. It is only in the log entries that they show as nil, at least insofar as what is visible to me.

As for using nil to delete a variable, that seems to be understood and is separate from the log problem.

I believe that this is a scope problem, thus the title of the post. But I don't believe this is a reactor scope problem in that the same issue does not occur on the Vera itself. So for whatever reason, in openLuup, the scope of the variable being displayed in the log is different from the scope of the log function--that the value of the variable is somehow hidden from the function.

This is a problem as it relates to the readability of the log, an annoyance more than anything, so I have no expectations to have it fixed. In my original screenshots, you can see multiple log entries that display as nil, so that's why I brought this to your attention as I felt that if there are scope issues at work with the logging function, then it could be happening elsewhere as well.

The only other thing that I would add is that in my startup lua, I call a module in the _G space so that I can access functions in the lua section of Reactor activities. My call is _G.myStartUpLua = require("myStartUpLua") I don't believe that this should have an effect on the log, but perhaps it does.

Buxton

@therealdb Yes, that's where I'm headed too. Agreed that Hass is very stable. It's just that there are real structural changes being made all the time, and before I sink my time into learning the details of the software, I want to be sure that pretty much everything is final in the engine.

Buxton

@toggledbits Yes, I think that's a large part of the problem as I'm seeing the same behavior with deleted groups in various sensors. The sensor groups flop between nil and empty in the attempt of trying to delete the variable.

There seems to be a separate issue of the openLuup log not picking up on variables for your various plugins log entries.

I'm guessing this has to do with assigning the luup log function to a table and then calling a truncated name ie L(...) when writing to the log. So there is some scope issue going on that is different on openLuup from that of a Vera, brought about by a level of abstraction.

Buxton

@therealdb Well that's a shame. One more nail in the coffin. I'll probably move to HA as soon as the various versions of the software are consolidated. It's a great platform, but for my needs, it is still too subject to breaking structural changes.

Buxton

@toggledbits OK. Does that mean that the log entries as shown above are expected ie. that the flopping (from nil to empty and back again) for reactor expression variables is routine?

Buxton

@akbooer Yes, I tried browser refresh without any change. If you look at the log, you see the variables flopping from nil to empty and then back again to nil.

@therealdb are you seeing anything like this in your openLuup reactor installation? I'm only seeing this nil behavior with Reactor expression variables.

Buxton

Hi Patrick/AK Booer

I am able to see Reactor expressions in the AltUI UI per the below and the expressions work as they should in my reactor sensors. When an expression changes, the reactor sensor responds accordingly.

However, I cannot see the expressions in luup state variables or the luup logs:

I tried reinstalling openLuup (latest development) and when that failed to change the noted behavior, I reinstalled lua5.1. There was also no change. Rebooting the machine also produced no change.
I suspect this is an openLuup issue as I also see nil values for some plugins:

And other plugins are fine:

2022-01-12 20:47:12.027   luup.variable_set:: 63.urn:upnp-micasaverde-com:serviceId:Weather1.CurrentDewPoint was: 32 now: 29.7 #hooks:0
2022-01-12 20:47:12.028   luup.variable_set:: 63.urn:upnp-micasaverde-com:serviceId:Weather1.WindSpeed was: 1.76 now: 2.8 #hooks:0
2022-01-12 20:47:12.028   luup.variable_set:: 63.urn:micasaverde-com:serviceId:HumiditySensor1.CurrentLevel was: 30 now: 27 #hooks:0
2022-01-12 20:47:12.028   luup.variable_set:: 65.urn:micasaverde-com:serviceId:HumiditySensor1.CurrentLevel was: 30 now: 27 #hooks:0
2022-01-12 20:47:12.028   luup.variable_set:: 63.urn:upnp-micasaverde-com:serviceId:Weather1.LastUpdate was: 1642047430 now: 1642049231 #hooks:0

Any ideas on how to troubleshoot this.....

Buxton

@akbooer Ok, that's clear enough. Thanks

Buxton

@buxton Yeah, running lua within the Reactor plugin/MSR is an incredibly powerful feature that, combined with expressions, pretty much allows you to do whatever you want to do. A wonderful piece of software.

Now I need to figure out how to construct the closure that will surround the watch callback function. Tricky, because there are so many variables that go into deciding when the routine should branch, and when branching should be avoided.

Buxton

@akbooer OK thanks, that's what I needed to know.

One final question if you could. How does scope affect a watch called from start-up lua? I'm asking because I imagine the scope for a watch is plugin specific, and therefore is confined to the plugin.

The watch will be created in the initialization code of a global module, that is in turn referenced in start-up lua ie. "_G.myStartUpLua = require("myStartUpLua") "

This module contains functions that are called from Reactor activities similar to this:

Will the global watch callback function be visible to the watch if both are created within "myStartUpLua".

Setting Aeotec Siren 6 volume and duration not working

MSR controller status panel

Super newbie here. Need help with lots of stuff

Short survey

Home Assistant add-on

Way to copy or export Alerts?

Alexa TTS volume does not update as expected

Include Entity by Name in Dynamic Group Controller

Need help in finding the right smart switch for my kitchen lights

[Solved] MSR actions aren't running anymore-- can't find what's broken

Reactor (Multi-System/Multi-Hub) Announcements

My MQTTController templates contribution

Sonoff zbmini-l and philips hue

Ezlo Controller - Error Incompatible serialization data - api cloud Timeout

MQTT log levels & log rotation

Help. Installing Reactor on Home Assistant.

Neural TTS with Sonos & Hubitat

Cheapest platform on which to run MSR

Buxton

@Buxton

Best posts made by Buxton

GitHub - toggledbits/sockproxyd: A socket proxy for Vera/openLuup systems

GitHub - toggledbits/LuWS: A WebSocket client implementation for Luup (Vera and openLuup) systems, with optional async receive (responsive)

Latest posts made by Buxton