openLuup: MQTT server
-
@akbooer yes that makes sense of course!
If I have understood it right you do it in two steps in HA, step one is to point it to the broker and step two is then to add Mqtt devices.
In other words I hope that the first step does not then subscribe to "everything" as MqttExplorer perhaps does.I was seeing similar network issues and also came to the conclusion that the socket library was most likely at fault. My solution is to use Mosquitto as my main broker, which accepts all MQTT traffic (topic # in 0) with all my MQTT devices pointing to it, and then Mosquitto filters push traffic to openLuup. Below is my config file that displays the filters:
allow_anonymous true password_file /mosquitto/data/PW.txt listener 1883 connection openLuup address 127.0.0.1:1882 topic tele/# out topic stat/# out topic BlueIris/# out topic # in 0 cleansession false notifications true username ***** password ******* bridge_protocol_version mqttv311 try_private false log_timestamp true log_timestamp_format %Y-%m-%d--T_%H:%M:%SAs you can see, Mosquitto runs on the same server as openLuup. It is started by a docker compose file. The config filters eliminated the network errors on openLuup and my openLuup install now runs for days on end without any errors at all. I also have HA running on the same server via docker compose, though I only use it for its Hacs Alexa integration. I pipe my Alexa calls to HA using an HA token and a crude plugin that I wrote. I have not found a use for HA outside of openLuup yet, though there are some interesting integrations I will eventually try out.
As regards MQTT I don't think you need to worry about network traffic so much as mqtt is an extremely light protocol, at least in so far as compared to cameras and hi-def wireless etc ( I have a bunch of these high bandwidth devices on my network in their own subnets). I have found that the thing that tends to bog down is the lua socket function and as long as you limit its connections, you will probably alleviate most of the network problems.
Nginx is one of the best web servers available, specializing in load balancing millions of connections, and from what I've read, it is written in Lua. Which suggests that the lua socket module itself is causing the network issues as Nginx most likely rolled their own network library.
-
I was seeing similar network issues and also came to the conclusion that the socket library was most likely at fault. My solution is to use Mosquitto as my main broker, which accepts all MQTT traffic (topic # in 0) with all my MQTT devices pointing to it, and then Mosquitto filters push traffic to openLuup. Below is my config file that displays the filters:
allow_anonymous true password_file /mosquitto/data/PW.txt listener 1883 connection openLuup address 127.0.0.1:1882 topic tele/# out topic stat/# out topic BlueIris/# out topic # in 0 cleansession false notifications true username ***** password ******* bridge_protocol_version mqttv311 try_private false log_timestamp true log_timestamp_format %Y-%m-%d--T_%H:%M:%SAs you can see, Mosquitto runs on the same server as openLuup. It is started by a docker compose file. The config filters eliminated the network errors on openLuup and my openLuup install now runs for days on end without any errors at all. I also have HA running on the same server via docker compose, though I only use it for its Hacs Alexa integration. I pipe my Alexa calls to HA using an HA token and a crude plugin that I wrote. I have not found a use for HA outside of openLuup yet, though there are some interesting integrations I will eventually try out.
As regards MQTT I don't think you need to worry about network traffic so much as mqtt is an extremely light protocol, at least in so far as compared to cameras and hi-def wireless etc ( I have a bunch of these high bandwidth devices on my network in their own subnets). I have found that the thing that tends to bog down is the lua socket function and as long as you limit its connections, you will probably alleviate most of the network problems.
Nginx is one of the best web servers available, specializing in load balancing millions of connections, and from what I've read, it is written in Lua. Which suggests that the lua socket module itself is causing the network issues as Nginx most likely rolled their own network library.
@buxton said in openLuup: MQTT server:
from what I've read, it is written in Lua.
It's written in C. And the BSD socket library that underpins the Unix/Linux versions is time-tested and battle-hardened. It could never be Lua, IMO, as the true multithreading/multitasking required is not a feature Lua possesses (and thus places a limit on Lua's socket handling).
-
luasocket definitely has limitations... I have investigated replacements in the past and never found one with sufficient maturity. There are many projects on the side attempting to improve on it. At the end I just updated the library to the latest on github.
Here is one example:
-
@buxton said in openLuup: MQTT server:
from what I've read, it is written in Lua.
It's written in C. And the BSD socket library that underpins the Unix/Linux versions is time-tested and battle-hardened. It could never be Lua, IMO, as the true multithreading/multitasking required is not a feature Lua possesses (and thus places a limit on Lua's socket handling).
@toggledbits ah OK. Then I'm confusing the dev with lua scripts used within nginx.
-
I was seeing similar network issues and also came to the conclusion that the socket library was most likely at fault. My solution is to use Mosquitto as my main broker, which accepts all MQTT traffic (topic # in 0) with all my MQTT devices pointing to it, and then Mosquitto filters push traffic to openLuup. Below is my config file that displays the filters:
allow_anonymous true password_file /mosquitto/data/PW.txt listener 1883 connection openLuup address 127.0.0.1:1882 topic tele/# out topic stat/# out topic BlueIris/# out topic # in 0 cleansession false notifications true username ***** password ******* bridge_protocol_version mqttv311 try_private false log_timestamp true log_timestamp_format %Y-%m-%d--T_%H:%M:%SAs you can see, Mosquitto runs on the same server as openLuup. It is started by a docker compose file. The config filters eliminated the network errors on openLuup and my openLuup install now runs for days on end without any errors at all. I also have HA running on the same server via docker compose, though I only use it for its Hacs Alexa integration. I pipe my Alexa calls to HA using an HA token and a crude plugin that I wrote. I have not found a use for HA outside of openLuup yet, though there are some interesting integrations I will eventually try out.
As regards MQTT I don't think you need to worry about network traffic so much as mqtt is an extremely light protocol, at least in so far as compared to cameras and hi-def wireless etc ( I have a bunch of these high bandwidth devices on my network in their own subnets). I have found that the thing that tends to bog down is the lua socket function and as long as you limit its connections, you will probably alleviate most of the network problems.
Nginx is one of the best web servers available, specializing in load balancing millions of connections, and from what I've read, it is written in Lua. Which suggests that the lua socket module itself is causing the network issues as Nginx most likely rolled their own network library.
@buxton said in openLuup: MQTT server:
I was seeing similar network issues and also came to the conclusion that the socket library was most likely at fault. My solution is to use Mosquitto as my main broker, which accepts all MQTT traffic (topic # in 0) with all my MQTT devices pointing to it, and then Mosquitto filters push traffic to openLuup. Below is my config file that displays the filters:
@Buxton thank you for the good advice!
I tried to set up the Mqtt integration in HA by directing it to the Mqtt address (mqtt://ipadress, port, user and passwd) on my OpenLuup, but it does not work. It just says "failed to connect" every time. Perhaps it only works with Mosquitto? Makes it a bit more complicated all by a sudden. I will see if I can try something in the direction of what you have set up later.I have not found a use for HA outside of openLuup yet, though there are some interesting integrations I will eventually try out.
I am in the same situation, no use at the moment, just a bit interested to see how it works. As you say there seem to be some interesting integrations there. Also there are more possibilities to present various information on the HA dashboard that could be nice for some use cases maybe. ESPHome also looks a bit interesting as a complement to Tasmota.
One use case that could be of interest to get into HA (in combination with ESPHome) is the energy metering of the house.As a sidenote my Shelly devices integrated really nicely into HA without affecting the Mqtt integration to OpenLuup.
-
@buxton said in openLuup: MQTT server:
I was seeing similar network issues and also came to the conclusion that the socket library was most likely at fault. My solution is to use Mosquitto as my main broker, which accepts all MQTT traffic (topic # in 0) with all my MQTT devices pointing to it, and then Mosquitto filters push traffic to openLuup. Below is my config file that displays the filters:
@Buxton thank you for the good advice!
I tried to set up the Mqtt integration in HA by directing it to the Mqtt address (mqtt://ipadress, port, user and passwd) on my OpenLuup, but it does not work. It just says "failed to connect" every time. Perhaps it only works with Mosquitto? Makes it a bit more complicated all by a sudden. I will see if I can try something in the direction of what you have set up later.I have not found a use for HA outside of openLuup yet, though there are some interesting integrations I will eventually try out.
I am in the same situation, no use at the moment, just a bit interested to see how it works. As you say there seem to be some interesting integrations there. Also there are more possibilities to present various information on the HA dashboard that could be nice for some use cases maybe. ESPHome also looks a bit interesting as a complement to Tasmota.
One use case that could be of interest to get into HA (in combination with ESPHome) is the energy metering of the house.As a sidenote my Shelly devices integrated really nicely into HA without affecting the Mqtt integration to OpenLuup.
I got the setup in HA to work so that it uses the OpenLuup Mqtt broker.
Instead of using the "Configuration->Integrations->Add Integrations->Mqtt" way that just gave an error message every time, I added it manually in the configuration.yaml file by simply adding:
mqtt: broker: ipadress port: 1883 username: user password: passwdAfter a restart it works. I have tested both to send a test message from HA to OpenLuup and also creating a few sensors in HA that subscribe to a topic from a Tasmota device on the broker in OpenLuup.
I still have no use case for it at the moment, but at least I learnt something on the way.
-
Trying to look at @Crille 's report, updated my openLuup to the latest (development branch), started up, MQTT server doesn't appear to be running... can't connect and netstat reports nothing open on 1883. What do I need to do to get it running? Is there a configuration guide?
-
MQTT server doesn't appear to be running...
Have you assigned it a port?
local attr = luup.attr_set do -- enable MQTT attr ("openLuup.MQTT.Port", 1883) attr ("openLuup.MQTT.Bridge_UDP", 2883) -- UDP bridge attr ("openLuup.MQTT.PublishVariableUpdates", true) -- publish every variable update attr ("openLuup.MQTT.PublishDeviceStatus", 2) -- publish a single device status every N seconds (0 = never) end -
…documentation was in this release note…
2020 Release 10.2
-
Excellent, thanks! On first blush, it looks like the service isn't closing connections completely on restart/shutdown; they stay open for a minute or so and then time out. Not sure if that's related, but may need a look.
Also I can't connect my Home Assistant 2021.11.5 to the broker. This in the logs:
2021-11-24 09:59:41.248 openLuup.io.server:: MQTT:1883 connection from 192.168.0.10 tcp{client}: 0x561b1d43bdb8 2021-11-24 09:59:41.249 openLuup.io.server:: MQTT:1883 connection closed tcp{client}: 0x561b1d43bdb8 2021-11-24 09:59:41.249 openLuup.mqtt:: CONNECT: Unknown protocol name: 'MQIsdp' tcp{client}: 0x561b1d43bdb8 2021-11-24 09:59:42.255 openLuup.io.server:: MQTT:1883 connection from 192.168.0.10 tcp{client}: 0x561b1d4499a8 2021-11-24 09:59:42.256 openLuup.io.server:: MQTT:1883 connection closed tcp{client}: 0x561b1d4499a8 2021-11-24 09:59:42.256 openLuup.mqtt:: CONNECT: Unknown protocol name: 'MQIsdp' tcp{client}: 0x561b1d4499a8 2021-11-24 09:59:44.262 openLuup.io.server:: MQTT:1883 connection from 192.168.0.10 tcp{client}: 0x561b1d4630c8 2021-11-24 09:59:44.263 openLuup.io.server:: MQTT:1883 connection closed tcp{client}: 0x561b1d4630c8 2021-11-24 09:59:44.263 openLuup.mqtt:: CONNECT: Unknown protocol name: 'MQIsdp' tcp{client}: 0x561b1d4630c8 2021-11-24 09:59:57.907 openLuup.io.server:: MQTT:1883 connection from 192.168.0.10 tcp{client}: 0x561b1d501108 2021-11-24 09:59:57.907 openLuup.io.server:: MQTT:1883 connection closed tcp{client}: 0x561b1d501108 2021-11-24 09:59:57.908 openLuup.mqtt:: CONNECT: Unknown protocol name: 'MQIsdp' tcp{client}: 0x561b1d501108 2021-11-24 09:59:58.913 openLuup.io.server:: MQTT:1883 connection from 192.168.0.10 tcp{client}: 0x561b1d50e638 2021-11-24 09:59:58.914 openLuup.io.server:: MQTT:1883 connection closed tcp{client}: 0x561b1d50e638 2021-11-24 09:59:58.914 openLuup.mqtt:: CONNECT: Unknown protocol name: 'MQIsdp' tcp{client}: 0x561b1d50e638Author of Multi-system Reactor and Reactor, DelayLight, Switchboard, and about a dozen other plugins that run on Vera and openLuup.
-
Excellent, thanks! On first blush, it looks like the service isn't closing connections completely on restart/shutdown; they stay open for a minute or so and then time out. Not sure if that's related, but may need a look.
Also I can't connect my Home Assistant 2021.11.5 to the broker. This in the logs:
2021-11-24 09:59:41.248 openLuup.io.server:: MQTT:1883 connection from 192.168.0.10 tcp{client}: 0x561b1d43bdb8 2021-11-24 09:59:41.249 openLuup.io.server:: MQTT:1883 connection closed tcp{client}: 0x561b1d43bdb8 2021-11-24 09:59:41.249 openLuup.mqtt:: CONNECT: Unknown protocol name: 'MQIsdp' tcp{client}: 0x561b1d43bdb8 2021-11-24 09:59:42.255 openLuup.io.server:: MQTT:1883 connection from 192.168.0.10 tcp{client}: 0x561b1d4499a8 2021-11-24 09:59:42.256 openLuup.io.server:: MQTT:1883 connection closed tcp{client}: 0x561b1d4499a8 2021-11-24 09:59:42.256 openLuup.mqtt:: CONNECT: Unknown protocol name: 'MQIsdp' tcp{client}: 0x561b1d4499a8 2021-11-24 09:59:44.262 openLuup.io.server:: MQTT:1883 connection from 192.168.0.10 tcp{client}: 0x561b1d4630c8 2021-11-24 09:59:44.263 openLuup.io.server:: MQTT:1883 connection closed tcp{client}: 0x561b1d4630c8 2021-11-24 09:59:44.263 openLuup.mqtt:: CONNECT: Unknown protocol name: 'MQIsdp' tcp{client}: 0x561b1d4630c8 2021-11-24 09:59:57.907 openLuup.io.server:: MQTT:1883 connection from 192.168.0.10 tcp{client}: 0x561b1d501108 2021-11-24 09:59:57.907 openLuup.io.server:: MQTT:1883 connection closed tcp{client}: 0x561b1d501108 2021-11-24 09:59:57.908 openLuup.mqtt:: CONNECT: Unknown protocol name: 'MQIsdp' tcp{client}: 0x561b1d501108 2021-11-24 09:59:58.913 openLuup.io.server:: MQTT:1883 connection from 192.168.0.10 tcp{client}: 0x561b1d50e638 2021-11-24 09:59:58.914 openLuup.io.server:: MQTT:1883 connection closed tcp{client}: 0x561b1d50e638 2021-11-24 09:59:58.914 openLuup.mqtt:: CONNECT: Unknown protocol name: 'MQIsdp' tcp{client}: 0x561b1d50e638@toggledbits said in openLuup: MQTT server:
Also I can't connect my Home Assistant 2021.11.5 to the broker. This in the logs:
Looks like it is using a modified protocol name MQIsdp rather than the default MQTT.
At the moment, it's not configurable on the openLuup side (but, of course, it could be.) Can you change this in HA?
-
@akbooer doesn't appear to have an option to change it:
On the other issue, my MQTT library is attempting to reconnect, but this is the only thing logged:
2021-11-24 10:05:18.394 openLuup.io.server:: MQTT:1883 connection from 192.168.0.66 tcp{client}: 0x55e743b50ef8There are no other messages with that ID. Can I get some more debug out of the server somehow? The library on my end is the latest version of the MQTT library commonly used in NodeJS; it's been around for a long time, and works perfectly with Mosquitto and other brokers I've tried. Initial connection always works, it's just reconnects that fail, so clearly there is something between the two that's keeping some history and maybe altering connect behavior, but I have no idea what and no visibility into it (at the moment).
Author of Multi-system Reactor and Reactor, DelayLight, Switchboard, and about a dozen other plugins that run on Vera and openLuup.
-
@akbooer doesn't appear to have an option to change it:
On the other issue, my MQTT library is attempting to reconnect, but this is the only thing logged:
2021-11-24 10:05:18.394 openLuup.io.server:: MQTT:1883 connection from 192.168.0.66 tcp{client}: 0x55e743b50ef8There are no other messages with that ID. Can I get some more debug out of the server somehow? The library on my end is the latest version of the MQTT library commonly used in NodeJS; it's been around for a long time, and works perfectly with Mosquitto and other brokers I've tried. Initial connection always works, it's just reconnects that fail, so clearly there is something between the two that's keeping some history and maybe altering connect behavior, but I have no idea what and no visibility into it (at the moment).
@toggledbits said in openLuup: MQTT server:
Can I get some more debug out of the server somehow?
attr ("openLuup.MQTT.DEBUG", true)will write more detail to the log (and, IIRC, standard output.)
You've apparently not got and username/password credentials set. These can be configured under openLuup...
attr ("openLuup.MQTT.Username", "foobar") -- Password, etc... -
Do credentials have to be set?
-
OK. While waiting, was testing some other things... it sure looks like the orphan/unclosed sockets are part of the issue.
-
Protocol name is the Home Assistant issue. I think we're past that -- it's a non-starter.
What I've been talking about my last couple of messages is that the NodeJS MQTT client library cannot reconnect to openLuup. The initial connection works, but if openLuup is restarted, the client can't reconnect to the openLuup MQTT broker. Here is the progress of debug for initial connection:
2021-11-24 11:46:32.961 openLuup.io.server:: MQTT:1883 connection from 192.168.0.66 tcp{client}: 0x560c3a0fdca8 2021-11-24 11:46:33.056 openLuup.mqtt:: CONNECT tcp{client}: 0x560c3a0fdca8 2021-11-24 11:46:33.056 openLuup.mqtt:: ClientId: reactor_mqtt 2021-11-24 11:46:33.056 openLuup.mqtt:: WillTopic: reactor/mqtt/LWT 2021-11-24 11:46:33.056 openLuup.mqtt:: WillMessage: offline 2021-11-24 11:46:33.057 openLuup.mqtt:: UserName: 2021-11-24 11:46:33.057 openLuup.mqtt:: Password: 2021-11-24 11:46:33.082 openLuup.mqtt:: SUBSCRIBE tcp{client}: 0x560c3a0fdca8 2021-11-24 11:46:33.082 openLuup.mqtt:: Packet Id: 0x17d0 2021-11-24 11:46:33.082 openLuup.mqtt:: Topic: # 2021-11-24 11:46:33.082 openLuup.mqtt:: reactor_mqtt SUBSCRIBE to # tcp{client}: 0x560c3a0fdca8 2021-11-24 11:46:33.098 openLuup.mqtt:: PUBLISH tcp{client}: 0x560c3a0fdca8 2021-11-24 11:46:33.099 openLuup.mqtt:: PUBLISH tcp{client}: 0x560c3a0fdca8 2021-11-24 11:46:33.099 openLuup.mqtt:: PUBLISH tcp{client}: 0x560c3a0fdca8All good and works fine until openLuup is restarted. Then things come a gutser.
Here is the progress of messages after openLuup is restarted, when the NodeJS MQTT client attempts to reconnect:
2021-11-24 11:47:56.357 openLuup.io.server:: starting MQTT:1883 server on port: 1883 tcp{server}: 0x5581839b06d8 2021-11-24 11:47:56.359 luup_log:2: starting MQTT round-robin device status messages 2021-11-24 11:47:56.359 luup_log:2: starting MQTT $SYS/broker statistics 2021-11-24 11:48:02.518 openLuup.io.server:: MQTT:1883 connection from 192.168.0.66 tcp{client}: 0x558183c2ccc8 2021-11-24 11:48:02.520 openLuup.mqtt:: CONNECT tcp{client}: 0x558183c2ccc8 2021-11-24 11:48:02.520 openLuup.mqtt:: ClientId: reactor_mqtt 2021-11-24 11:48:02.520 openLuup.mqtt:: WillTopic: reactor/mqtt/LWT 2021-11-24 11:48:02.520 openLuup.mqtt:: WillMessage: offline 2021-11-24 11:48:02.520 openLuup.mqtt:: UserName: 2021-11-24 11:48:02.520 openLuup.mqtt:: Password: 2021-11-24 11:48:02.523 openLuup.mqtt:: PUBLISH tcp{client}: 0x558183c2ccc8 2021-11-24 11:50:03.091 openLuup.io.server:: MQTT:1883 connection closed EXPIRED tcp{client}: 0x558183c2ccc8 2021-11-24 11:50:08.099 openLuup.io.server:: MQTT:1883 connection from 192.168.0.66 tcp{client}: 0x558183ee52b8 2021-11-24 11:50:08.100 openLuup.mqtt:: CONNECT tcp{client}: 0x558183ee52b8 2021-11-24 11:50:08.100 openLuup.mqtt:: ClientId: reactor_mqtt 2021-11-24 11:50:08.100 openLuup.mqtt:: WillTopic: reactor/mqtt/LWT 2021-11-24 11:50:08.100 openLuup.mqtt:: WillMessage: offline 2021-11-24 11:50:08.100 openLuup.mqtt:: UserName: 2021-11-24 11:50:08.101 openLuup.mqtt:: Password: 2021-11-24 11:50:08.103 openLuup.mqtt:: PUBLISH tcp{client}: 0x558183ee52b8The entire dialog stalls on the first PUBLISH after reconnect, and eventually the connection expires and is recycled, and that loop continues forever more. On the MSR end, the MQTT client library never sends the
connectedevent, so apparently something in the handshake hasn't completed. I have no idea what is causing this, but it is unique to openLuup's MQTT server. This does not happen with other brokers I've tried (e.g. Mosquitto). -
Excellent, thanks! On first blush, it looks like the service isn't closing connections completely on restart/shutdown; they stay open for a minute or so and then time out. Not sure if that's related, but may need a look.
Also I can't connect my Home Assistant 2021.11.5 to the broker. This in the logs:
2021-11-24 09:59:41.248 openLuup.io.server:: MQTT:1883 connection from 192.168.0.10 tcp{client}: 0x561b1d43bdb8 2021-11-24 09:59:41.249 openLuup.io.server:: MQTT:1883 connection closed tcp{client}: 0x561b1d43bdb8 2021-11-24 09:59:41.249 openLuup.mqtt:: CONNECT: Unknown protocol name: 'MQIsdp' tcp{client}: 0x561b1d43bdb8 2021-11-24 09:59:42.255 openLuup.io.server:: MQTT:1883 connection from 192.168.0.10 tcp{client}: 0x561b1d4499a8 2021-11-24 09:59:42.256 openLuup.io.server:: MQTT:1883 connection closed tcp{client}: 0x561b1d4499a8 2021-11-24 09:59:42.256 openLuup.mqtt:: CONNECT: Unknown protocol name: 'MQIsdp' tcp{client}: 0x561b1d4499a8 2021-11-24 09:59:44.262 openLuup.io.server:: MQTT:1883 connection from 192.168.0.10 tcp{client}: 0x561b1d4630c8 2021-11-24 09:59:44.263 openLuup.io.server:: MQTT:1883 connection closed tcp{client}: 0x561b1d4630c8 2021-11-24 09:59:44.263 openLuup.mqtt:: CONNECT: Unknown protocol name: 'MQIsdp' tcp{client}: 0x561b1d4630c8 2021-11-24 09:59:57.907 openLuup.io.server:: MQTT:1883 connection from 192.168.0.10 tcp{client}: 0x561b1d501108 2021-11-24 09:59:57.907 openLuup.io.server:: MQTT:1883 connection closed tcp{client}: 0x561b1d501108 2021-11-24 09:59:57.908 openLuup.mqtt:: CONNECT: Unknown protocol name: 'MQIsdp' tcp{client}: 0x561b1d501108 2021-11-24 09:59:58.913 openLuup.io.server:: MQTT:1883 connection from 192.168.0.10 tcp{client}: 0x561b1d50e638 2021-11-24 09:59:58.914 openLuup.io.server:: MQTT:1883 connection closed tcp{client}: 0x561b1d50e638 2021-11-24 09:59:58.914 openLuup.mqtt:: CONNECT: Unknown protocol name: 'MQIsdp' tcp{client}: 0x561b1d50e638@toggledbits said in openLuup: MQTT server:
Also I can't connect my Home Assistant 2021.11.5 to the broker. This in the logs:
I may have missed this, but was this solved?
The reason is that I am currently on Home Assistant 2021.10.06 and was planning to upgrade to the latest 2021.11.5.
On 2021.10.6 I am using the Mqtt server in OpenLuup and it is working fine.
