Best posts made by Buxton | Smart Home

Buxton

@archers Very Cool gadget. I soldered to the flash pins, which given the size of the contacts was challenging, but managed to make it work. Now waiting for my BLE sensors to arrive.

Energy Monitoring built into the dual relay!

Buxton

@archers

I was seeing similar network issues and also came to the conclusion that the socket library was most likely at fault. My solution is to use Mosquitto as my main broker, which accepts all MQTT traffic (topic # in 0) with all my MQTT devices pointing to it, and then Mosquitto filters push traffic to openLuup. Below is my config file that displays the filters:

allow_anonymous true
password_file /mosquitto/data/PW.txt
listener 1883
connection openLuup
address 127.0.0.1:1882
topic tele/# out 
topic stat/# out 
topic BlueIris/# out 
topic # in 0
cleansession false
notifications true
username *****
password *******
bridge_protocol_version mqttv311
try_private false
log_timestamp true
log_timestamp_format %Y-%m-%d--T_%H:%M:%S

As you can see, Mosquitto runs on the same server as openLuup. It is started by a docker compose file. The config filters eliminated the network errors on openLuup and my openLuup install now runs for days on end without any errors at all. I also have HA running on the same server via docker compose, though I only use it for its Hacs Alexa integration. I pipe my Alexa calls to HA using an HA token and a crude plugin that I wrote. I have not found a use for HA outside of openLuup yet, though there are some interesting integrations I will eventually try out.

As regards MQTT I don't think you need to worry about network traffic so much as mqtt is an extremely light protocol, at least in so far as compared to cameras and hi-def wireless etc ( I have a bunch of these high bandwidth devices on my network in their own subnets). I have found that the thing that tends to bog down is the lua socket function and as long as you limit its connections, you will probably alleviate most of the network problems.

Nginx is one of the best web servers available, specializing in load balancing millions of connections, and from what I've read, it is written in Lua. Which suggests that the lua socket module itself is causing the network issues as Nginx most likely rolled their own network library.

Buxton

@akbooer I think I found the culprit causing the random connection disconnects to Mosquitto. The LWT payload for a given device is a simple string, which doesn't seem to decode in the json decode call. So I added the below code to send a json string to the decoder. The errors have disappeared and I now see the LWT variable in the service variables. The variable reads "LWT : Online"

--line 165
local valid = {SENSOR = true, STATE = true, RESULT = true, LWT = true}

--line 172
  -- begin code
  local info, err = json.decode (message)
   
  if message then
	if not info then  -- json did not decode because of single string parameter
		message = '{' .. '"' .. mtype .. '"' .. ':' .. '"' .. message .. '"' .. '}'
	end
  end
  -- end code

  
  local info, err = json.decode (message)

This is probably not the way you would handle the error, but it does work.

Buxton

@akbooer Tasmota energy sensor

{
  "Time": "2021-03-28T21:51:01",
  "ENERGY": {
    "TotalStartTime": "2020-06-07T00:10:43",
    "Total": 2356.063,
    "Yesterday": 7.056,
    "Today": 6.459,
    "Period": 24,
    "Power": 285,
    "ApparentPower": 302,
    "ReactivePower": 99,
    "Factor": 0.95,
    "Voltage": 123,
    "Current": 2.453
  }
}

Buxton

No, you can still configure UDP. You just need login credentials now.

If I have a few moments, I will try to set up one of my Pi machines this weekend with an instance of Mosquitto. My HA server that runs my docker Mosquitto is headless, and I run Ubuntu server, so command line captures of packets are just a drag. My Pi has an hdmi port so I should be able to load the gui version of wireshark, and then test/capture the traffic between the two instances. I will let you know.

Buxton

@rafale77 hey Rafale,

I went down the very same road a while back and threw in towel because the polling by the MQTT plugin created CPU drags that stopped openLuup from functioning "reliably". The instability was also in part because I use two other must-have plugins that rely on polling, and I imagine that the combination of the three was creating a scenario that caused intermittent failures. And I too ended up implementing MQTT in Home Assistant and then using RealDB's virtual HTTP plugin to send commands to my WiFi devices--albeit not knowing the status of the devices in openLuup after the send.

I'm looking at RigPapa's socket proxy and WebSocket plugins to see if I can transform my polling plugins to Async. The MQTT plugin is too complex for me to convert though, so if you take a crack at it, and are successful, I would very much appreciate you publishing your results, as MQTT is becoming a must for me.

GitHub - toggledbits/sockproxyd: A socket proxy for Vera/openLuup systems

A socket proxy for Vera/openLuup systems. Contribute to toggledbits/sockproxyd development by creating an account on GitHub.

GitHub - toggledbits/LuWS: A WebSocket client implementation for Luup (Vera and openLuup) systems, with optional async receive (responsive)

A WebSocket client implementation for Luup (Vera and openLuup) systems, with optional async receive (responsive) - toggledbits/LuWS

Buxton

@akbooer I hate to add more to the pile... but I'm still seeing a receive error for the connection to mosquitto. openLuup 2021.04.29b

Here's the log error:

2021-05-01 14:22:38.815   openLuup.io.server:: MQTT:1882 connection closed  tcp{client}: 0x5579919e9c58
2021-05-01 14:22:38.816   openLuup.mqtt:: RECEIVE ERROR: closed tcp{client}: 0x5579919e9c58
2021-05-01 14:22:43.935   luup.io.incoming:: bytes received: 51, status: OK tcp{client}: 0x557990e7f528
2021-05-01 14:22:48.435   luup.io.incoming:: bytes received: 51, status: OK tcp{client}: 0x557990e7f528
2021-05-01 14:22:49.763   luup.variable_set:: 10181.urn:micasaverde-com:serviceId:EnergyMetering1.KWHReading was: 1619904116 now: 1619904168 #hooks:0
2021-05-01 14:22:50.158   openLuup.io.server:: HTTP:3480 connection closed openLuup.server.receive closed tcp{client}: 0x557991971ff8
2021-05-01 14:22:53.834   openLuup.io.server:: MQTT:1882 connection from 127.0.0.1 tcp{client}: 0x5579920dc0b8
2021-05-01 14:22:53.834   openLuup.mqtt:: client is in ERROR empty
2021-05-01 14:22:53.834   openLuup.mqtt:: credentials is in ERROR empty
2021-05-01 14:22:53.834   openLuup.mqtt:: subscriptions is in ERROR empty

using the below error trapping in function "MQTTservlet"

  local function MQTTservlet (client)
  
	if client == nil then
	  _log ("client is in ERROR nil")
	else
	  	if table.concat(client) == "" then
		_log ("client is in ERROR empty")
		else
	   _log (table.concat {"MQTT ERROR: ", table.concat(client)})
	    end
	end 
	
	if credentials == nil then
	  _log ("credentials is in ERROR nil")
	else
	  	if table.concat(credentials) == "" then
		_log ("credentials is in ERROR empty")
		else
	   _log (table.concat {"MQTT ERROR: ", table.concat(credentials)})
	    end
	end 
	
	if subscriptions == nil then
	  _log ("subscriptions is in ERROR nil")
	else
	  	if table.concat(subscriptions) == "" then
		_log ("subscriptions is in ERROR empty")
		else
	    _log (table.concat {"MQTT ERROR: ", table.concat(subscriptions)})
	    end
	end
	
    return function () incoming (client, credentials, subscriptions) end
  end

I can't find a deeper layer in the stack where I can trap for the incoming message to see what's in the message that is throwing the error. As near as I can tell, if openLuup tries to connect to a running mosquitto instance, then it fails to see the topics and messages, and passes empty--but not nil--strings when the servlet interface sees incoming bytes.

If I restart mosquitto, openLuup then sees the topics and messages and the error messages stop--and the connection to mosquitto remains stable.

This behavior does not occur when I aim an IOT device directly at openLuup--in that the connection to the device always resumes when openLuup reloads--in other words, I don't need to restart the IOT device to enable the connection.

Buxton

@akbooer Below is the relevant output of a typical packet between mosquitto and a mosquitto bridged instance. In this case, mosquitto is sending update data to the bridge regarding a tasmota device/switch I use to remotely reboot my Vera. The format is definitely MQTT 3.1 and not 5.0, as 5.0 would not parse correctly in the wireshark viewer. The data payload is at the top of the window as Wireshark will truncate long messages. I can PM you the entire capture as it's not much, but may contain technical info that's best kept private. Let me know.

I'll try to capture some traffic between openLuup and mosquitto later.

{"Version":"9.1.0(tasmota)","BuildDateTime":"2020-11-07T11:57:45","Module or Template":"Gosund-WP5","RestartReason":"Software/System restart","Uptime":"6T05:50:22","Hostname":"power_MainVera-0278","IPAddress":"10.17.2.33","RSSI":"100","Signal (dBm)":"-17","WiFi LinkCount":5,"WiFi Downtime":"0T00:00:10","MqttCount":14,"LoadAvg":19}



Frame 9: 433 bytes on wire (3464 bits), 433 bytes captured (3464 bits) on interface eth0, id 0
Ethernet II, Src: Advansus_0a:8c:3a (00:19:0f:0a:8c:3a), Dst: 96:62:08:fb:22:8a (96:62:08:fb:22:8a)
Internet Protocol Version 4, Src: 10.17.2.41, Dst: 10.17.2.110
Transmission Control Protocol, Src Port: 40664, Dst Port: 1882, Seq: 3, Ack: 3, Len: 367
MQ Telemetry Transport Protocol, Publish Message
    Header Flags: 0x30, Message Type: Publish Message, QoS Level: At most once delivery (Fire and Forget)
        0011 .... = Message Type: Publish Message (3)
        .... 0... = DUP Flag: Not set
        .... .00. = QoS Level: At most once delivery (Fire and Forget) (0)
        .... ...0 = Retain: Not set
    Msg Len: 364
    Topic Length: 30
    Topic: tele/power_MainVera/HASS_STATE
    Message [truncated--see above]: {"Version":"9.1.0(tasmota)","BuildDateTime":"2020-11-07T11:57:45","Module or Template":"Gosund-WP5","RestartReason":"Software/System restart","Uptime":"6T05:50:22","Hostname":"power_MainVera-0278","IPAddress":"10.17.2

Buxton

@buxton In the above, I'm seeing an extra closing right hand bracket in the JSON string.

Buxton

I've been waiting for something like this. Very cool. Thx for the post

Buxton

@akbooer
And a ping response from bridge to main instance:

Frame 33: 68 bytes on wire (544 bits), 68 bytes captured (544 bits) on interface eth0, id 0
Ethernet II, Src: 96:62:08:fb:22:8a (96:62:08:fb:22:8a), Dst: Advansus_0a:8c:3a (00:19:0f:0a:8c:3a)
Internet Protocol Version 4, Src: 10.17.2.110, Dst: 10.17.2.41
Transmission Control Protocol, Src Port: 1882, Dst Port: 40664, Seq: 5, Ack: 2851, Len: 2
MQ Telemetry Transport Protocol, Ping Response
    Header Flags: 0xd0, Message Type: Ping Response
        1101 .... = Message Type: Ping Response (13)
        .... 0000 = Reserved: 0
    Msg Len: 0

Buxton

@toggledbits I was able to install the container through compose, however, I had to make some changes to get it working. See below for my compose file:

 MSR:
  container_name: reactor
  image: toggledbits/reactor:latest-generic-amd64
  restart: "on-failure"
  environment:
   REACTOR_DATA_PREFIX: /var/reactor
   TZ: America/Los_Angeles
  expose:
   - 8111
  ports:
   - 8111:8111
  volumes:
   - /home/username/reactor:/var/reactor
   - /etc/localtime:/etc/localtime:ro
  tmpfs: /tmp
#  logging:
#   driver: "json-file"
#   options:
#    max-file: 5
#    max-size: 2m

The changes I made are:
1.) substitute "MSR" for "web" for the service name. This was just a precaution against a generic service name interfering with container management programs I use, and was not a needed/critical change to get things working.
2.) simplify the volume syntax for binding a data volume. The syntax you have on your website caused a yaml compile error with version: '3.7' compose.
3.) comment out the logging options. These options compiled, but threw a runtime JSON error that stopped the container from coming up:

ERROR: for MSR  Cannot create container for service MSR: json: cannot unmarshal number into Go struct field LogConfig.HostConfig.LogConfig.Config of type string
ERROR: Encountered errors while bringing up the project.

I hope to cut out some time next week to start forming some logic. The web UI looks great.

Buxton

@akbooer No errors in 2021.04.18. Thanks for this as the changes also stabilized my Mosquitto bridge connection, which tended to flop with every error message.

2021-04-18 15:29:04.173   luup.tasmota:262: Topic ignored : tele/power_ServerWork/LWT : Online
2021-04-18 15:29:04.175   luup.tasmota:262: Topic ignored : tele/power_MainVera/LWT : Online
2021-04-18 15:29:04.176   luup.tasmota:262: Topic ignored : tele/power_HAServer/LWT : Online
2021-04-18 15:29:04.177   luup.tasmota:262: Topic ignored : tele/power_SideLandscape/LWT : Online
2021-04-18 15:29:04.178   luup.tasmota:262: Topic ignored : tele/power_GarageVera/LWT : Online

Buxton

@akbooer

The Connect packet:

MQ Telemetry Transport Protocol, Connect Command
    Header Flags: 0x10, Message Type: Connect Command
    Msg Len: 94
    Protocol Name Length: 4
    Protocol Name: MQTT
    Version: Unknown (132)
    Connect Flags: 0xec, User Name Flag, Password Flag, Will Retain, QoS Level: At least once delivery (Acknowledged deliver), Will Flag
        1... .... = User Name Flag: Set
        .1.. .... = Password Flag: Set
        ..1. .... = Will Retain: Set
        ...0 1... = QoS Level: At least once delivery (Acknowledged deliver) (1)
        .... .1.. = Will Flag: Set
        .... ..0. = Clean Session Flag: Not set
        .... ...0 = (Reserved): Not set
    Keep Alive: 60
    Client ID Length: 14
    Client ID: Thing.MosquittoBridge
    Will Topic Length: 43
    Will Topic: $SYS/broker/connection/Thing.MosquittoBridge/state
    Will Message Length: 1
    Will Message: 0
    User Name Length: 6
    User Name: YYYYYY
    Password Length: 10
    Password: XXXXXXXXXX

Buxton

@akbooer No historian errors and all "checked" variables are publishing to my InfluxDB server.

Buxton

@akbooer
The subscribe packet:

Frame 156: 81 bytes on wire (648 bits), 81 bytes captured (648 bits) on interface eth0, id 0
Ethernet II, Src: Advansus_0a:8c:3a (00:19:0f:0a:8c:3a), Dst: 96:62:08:fb:22:8a (96:62:08:fb:22:8a)
Internet Protocol Version 4, Src: 10.17.2.41, Dst: 10.17.2.110
Transmission Control Protocol, Src Port: 36446, Dst Port: 1882, Seq: 147, Ack: 9, Len: 15
MQ Telemetry Transport Protocol, Unsubscribe Request
    Header Flags: 0xa2, Message Type: Unsubscribe Request
        1010 .... = Message Type: Unsubscribe Request (10)
        .... 0010 = Reserved: 2
    Msg Len: 5
    Message Identifier: 2
    Topic Length: 1
    Topic: #
MQ Telemetry Transport Protocol, Subscribe Request
    Header Flags: 0x82, Message Type: Subscribe Request
        1000 .... = Message Type: Subscribe Request (8)
        .... 0010 = Reserved: 2
    Msg Len: 6
    Message Identifier: 3
    Topic Length: 1
    Topic: #
    Requested QoS: At most once delivery (Fire and Forget) (0)

Buxton

@akbooer

The connect ACK:

MQ Telemetry Transport Protocol, Connect Ack
    Header Flags: 0x20, Message Type: Connect Ack
        0010 .... = Message Type: Connect Ack (2)
        .... 0000 = Reserved: 0
    Msg Len: 2
    Acknowledge Flags: 0x00
        0000 000. = Reserved: Not set
        .... ...0 = Session Present: Not set
    Reason Code: Success (0)

Buxton

@akbooer Yes but as you can see on the connect, the version is "Version: Unknown (132)"

This is what is causing the problem. After much searching and trying different configs, I stumbled on the following which solved the problem. From mosquitto.org

try_private [ true | false ]
If try_private is set to true, the bridge will attempt to indicate to the remote broker that it is a bridge not an ordinary client. If successful, this means that loop detection will be more effective and that retained messages will be propagated correctly. Not all brokers support this feature so it may be necessary to set try_private to false if your bridge does not connect properly.

Defaults to true.

So I set the attribute to false in my bridge config and immediately connected openLuup to the mosquitto broker. The connect packet shows the right version, and with a luup reload, all of my mosquitto broker topics populated in mqtt explorer that was pointed at openLuup. However, I don't see the topics in the mqtt console on openLuup?? Which is odd because I not only see the openLuup topics in explorer, but I see the topics actively changing.

I'm not a good one to suggest code changes, but since this mosquitto setting defaults to true, can you try to incorporate the try_private flag in openLuup's MQTT server.... It took a long time to track this down and I imagine anyone else that tries to connect the two servers will be in for a similar bug fix adventure.

Buxton

image url)

openLuup Explorer

Buxton

@akbooer Yes, I was thinking along those lines as the bridge config allows filters. The latest openLuup version now works fine with try_private flag set to default (true). Thanks for nailing this down and your work is definitely appreciated. Below is the connection to openLuup.

Here's my Mosquitto config for anyone who wants to bridge the two brokers:

allow_anonymous true
password_file /mosquitto/data/PW.txt
listener 1883
connection openLuup
address 127.0.0.1:1882
topic # out 0
topic # in 0
cleansession false
notifications true
username XXXXX
password YYYYYYYYYY
bridge_protocol_version mqttv311

Most of these settings can/should be modified to suit one's particular needs, but the settings should be employed. The password file for mosquitto needs to be encrypted with mosquitto's built-in encryption tool. The directions are straightforward and are described in on-line documents.

Buxton

Farewell, oh good and faithful servant!

http request action & digest auth

Reactor (Multi-System/Multi-Hub) Announcements

Cloning actions in reactions does not work

Has ping command been removed?

ReferenceError with Home Assistant data & build 26140

Integrations and 'Loaded'

Possible mismatch between binary_sensor in HA and MSR

[Solved] Rulesets with multiple groups in Set Reaction not working post-26116

Reactor build 26116: empty group reaction

Temperature control with TRVs

Upcoming Storage Change -- Got Back-ups?

Next Release?

Controller Z-Wave JS UI: "Location" attribute not visible in Reactor entities

Do you Matter?

Variables not updating properly

[Solved] Loading Screen Safari

VEC Virtual Switch Auto Off

Changes operator does not always detect change

Dynamic MQTT topics and parameters

Posts

GitHub - toggledbits/sockproxyd: A socket proxy for Vera/openLuup systems

GitHub - toggledbits/LuWS: A WebSocket client implementation for Luup (Vera and openLuup) systems, with optional async receive (responsive)