openLuup email server

[therealdb]

that's huge @akbooer! I'll trying to use it in production next week, hopefully. I've been very busy lately, unfortunately.

[Buxton]

Awesome AK. This will integrate all my MQTT devices. Have you tackled the Mosquitto bridge issue.

[akbooer]

@buxton said in openLuup: MQTT server:

Have you tackled the Mosquitto bridge issue.

There have been some refinements in the MQTT server implementation, but I couldn’t say that I’ve changed anything significant. I did voice some questions about your Mosquitto settings, but I’m not at all knowledgeable about Mosquitto, so I can’t really say. What I do believe is that the implementation is sufficiently compliant with the standard to be working really well with straightforward clients like the Shelly switches
(and the MQTT Explorer application.)

Have you ever successfully bridged two Mosquitto instances?

[Buxton]

OK --successfully bridged two Mosquitto instances -- yes. Comes right up. FYI influxdb 2.0 has removed auto graphite compatibility--that is according to their website. I don't know what changed. I do know that they are jumping on the must use credentials bandwagon, so I'm having to do a lot of research to get everything working again with the various containers that use influxdb as their repository.

[akbooer]

Bridging: great. Any chance of finding out the chain of messages exchanged whilst bridging? Have you tried again with openLuup? My only thought is that is demands a QoS of greater than zero. A debug log would clarify.

InfluxDB: I’ve always used UDP to send data. Is that going away?

[Buxton]

No, you can still configure UDP. You just need login credentials now.

If I have a few moments, I will try to set up one of my Pi machines this weekend with an instance of Mosquitto. My HA server that runs my docker Mosquitto is headless, and I run Ubuntu server, so command line captures of packets are just a drag. My Pi has an hdmi port so I should be able to load the gui version of wireshark, and then test/capture the traffic between the two instances. I will let you know.

[Buxton]

@akbooer Below is the relevant output of a typical packet between mosquitto and a mosquitto bridged instance. In this case, mosquitto is sending update data to the bridge regarding a tasmota device/switch I use to remotely reboot my Vera. The format is definitely MQTT 3.1 and not 5.0, as 5.0 would not parse correctly in the wireshark viewer. The data payload is at the top of the window as Wireshark will truncate long messages. I can PM you the entire capture as it's not much, but may contain technical info that's best kept private. Let me know.

I'll try to capture some traffic between openLuup and mosquitto later.

{"Version":"9.1.0(tasmota)","BuildDateTime":"2020-11-07T11:57:45","Module or Template":"Gosund-WP5","RestartReason":"Software/System restart","Uptime":"6T05:50:22","Hostname":"power_MainVera-0278","IPAddress":"10.17.2.33","RSSI":"100","Signal (dBm)":"-17","WiFi LinkCount":5,"WiFi Downtime":"0T00:00:10","MqttCount":14,"LoadAvg":19}



Frame 9: 433 bytes on wire (3464 bits), 433 bytes captured (3464 bits) on interface eth0, id 0
Ethernet II, Src: Advansus_0a:8c:3a (00:19:0f:0a:8c:3a), Dst: 96:62:08:fb:22:8a (96:62:08:fb:22:8a)
Internet Protocol Version 4, Src: 10.17.2.41, Dst: 10.17.2.110
Transmission Control Protocol, Src Port: 40664, Dst Port: 1882, Seq: 3, Ack: 3, Len: 367
MQ Telemetry Transport Protocol, Publish Message
    Header Flags: 0x30, Message Type: Publish Message, QoS Level: At most once delivery (Fire and Forget)
        0011 .... = Message Type: Publish Message (3)
        .... 0... = DUP Flag: Not set
        .... .00. = QoS Level: At most once delivery (Fire and Forget) (0)
        .... ...0 = Retain: Not set
    Msg Len: 364
    Topic Length: 30
    Topic: tele/power_MainVera/HASS_STATE
    Message [truncated--see above]: {"Version":"9.1.0(tasmota)","BuildDateTime":"2020-11-07T11:57:45","Module or Template":"Gosund-WP5","RestartReason":"Software/System restart","Uptime":"6T05:50:22","Hostname":"power_MainVera-0278","IPAddress":"10.17.2

[Buxton]

@akbooer
And a ping response from bridge to main instance:

Frame 33: 68 bytes on wire (544 bits), 68 bytes captured (544 bits) on interface eth0, id 0
Ethernet II, Src: 96:62:08:fb:22:8a (96:62:08:fb:22:8a), Dst: Advansus_0a:8c:3a (00:19:0f:0a:8c:3a)
Internet Protocol Version 4, Src: 10.17.2.110, Dst: 10.17.2.41
Transmission Control Protocol, Src Port: 1882, Dst Port: 40664, Seq: 5, Ack: 2851, Len: 2
MQ Telemetry Transport Protocol, Ping Response
    Header Flags: 0xd0, Message Type: Ping Response
        1101 .... = Message Type: Ping Response (13)
        .... 0000 = Reserved: 0
    Msg Len: 0

[Buxton]

@akbooer

The Connect packet:

MQ Telemetry Transport Protocol, Connect Command
    Header Flags: 0x10, Message Type: Connect Command
    Msg Len: 94
    Protocol Name Length: 4
    Protocol Name: MQTT
    Version: Unknown (132)
    Connect Flags: 0xec, User Name Flag, Password Flag, Will Retain, QoS Level: At least once delivery (Acknowledged deliver), Will Flag
        1... .... = User Name Flag: Set
        .1.. .... = Password Flag: Set
        ..1. .... = Will Retain: Set
        ...0 1... = QoS Level: At least once delivery (Acknowledged deliver) (1)
        .... .1.. = Will Flag: Set
        .... ..0. = Clean Session Flag: Not set
        .... ...0 = (Reserved): Not set
    Keep Alive: 60
    Client ID Length: 14
    Client ID: Thing.MosquittoBridge
    Will Topic Length: 43
    Will Topic: $SYS/broker/connection/Thing.MosquittoBridge/state
    Will Message Length: 1
    Will Message: 0
    User Name Length: 6
    User Name: YYYYYY
    Password Length: 10
    Password: XXXXXXXXXX

[Buxton]

@akbooer

The connect ACK:

MQ Telemetry Transport Protocol, Connect Ack
    Header Flags: 0x20, Message Type: Connect Ack
        0010 .... = Message Type: Connect Ack (2)
        .... 0000 = Reserved: 0
    Msg Len: 2
    Acknowledge Flags: 0x00
        0000 000. = Reserved: Not set
        .... ...0 = Session Present: Not set
    Reason Code: Success (0)

[Buxton]

@akbooer
The subscribe packet:

Frame 156: 81 bytes on wire (648 bits), 81 bytes captured (648 bits) on interface eth0, id 0
Ethernet II, Src: Advansus_0a:8c:3a (00:19:0f:0a:8c:3a), Dst: 96:62:08:fb:22:8a (96:62:08:fb:22:8a)
Internet Protocol Version 4, Src: 10.17.2.41, Dst: 10.17.2.110
Transmission Control Protocol, Src Port: 36446, Dst Port: 1882, Seq: 147, Ack: 9, Len: 15
MQ Telemetry Transport Protocol, Unsubscribe Request
    Header Flags: 0xa2, Message Type: Unsubscribe Request
        1010 .... = Message Type: Unsubscribe Request (10)
        .... 0010 = Reserved: 2
    Msg Len: 5
    Message Identifier: 2
    Topic Length: 1
    Topic: #
MQ Telemetry Transport Protocol, Subscribe Request
    Header Flags: 0x82, Message Type: Subscribe Request
        1000 .... = Message Type: Subscribe Request (8)
        .... 0010 = Reserved: 2
    Msg Len: 6
    Message Identifier: 3
    Topic Length: 1
    Topic: #
    Requested QoS: At most once delivery (Fire and Forget) (0)

[Buxton]

@akbooer

Finally, the subscribe ACK:

MQ Telemetry Transport Protocol, Subscribe Ack
    Header Flags: 0x90, Message Type: Subscribe Ack
        1001 .... = Message Type: Subscribe Ack (9)
        .... 0000 = Reserved: 0
    Msg Len: 3
    Message Identifier: 3
    Reason Code: Granted QoS 0 (0)

[akbooer]

@Buxton
Thanks so much for taking the time to capture those snapshots.

What I see is:

• PUBLISH – requests QoS 0
• PINGRESP – standard fixed format
• CONNECT – v3.1.1 protocol, asks for WillQoS 1 and CleanSession 0
• CONNACK – responds with SessionPresent 0
• SUBSCRIBE – ask for QoS 0
• SUBACK - grants QoS 0

This all seems fine (as, indeed, it is) but, more importantly, appears to be quite consistent with what the openLuup MQTT server can offer.

Given, that it would be revealing to see how the Mosquitto / openLuup bridge fails...

Much appreciate the help (hence the upvotes to each of the posted images!)

AK

[ArcherS]

The update to include Mqtt and Shelly support is a great addition to OpenLuup!

I updated to the latest version (2021 Release 3.11b) to start testing the MQTT functionality. The first aim for me is to get data into OpenLuup from my Tasmota based sensors, and also in the next step to get my Shelly devices into OpenLuup with MQTT.
Today I use Tasmota rules, SiteSensor and http commands from the Shellies, migrating to MQTT will make this much more simple and robust.

I have got it to work with setting up the server with credentials. I can see in my test Tasmota device that it connects to the server, and I can also see in MqttExplorer that it connects. So far so good!

However what I have not quite understood is how to get the subscription of device data into e.g. a virtual temp or humidity sensor in OpenLuup.

@akbooer In the instruction it says that one should use a standard Luup request handler to subscribe, where do I put that? Also how would the handler code typically look like?

As a comparison, below is what I used in the settings.json that I used fo the Mqtt Bridge from @therealdb.

"Devices": [
		{
			"ClientID": "tasmota_test",

			"TopicName": "tele/tasmota_test/SENSOR",
			"TopicPath": "AM2301.Temperature",

			"DeviceID": 216,
			"Service": "urn:upnp-org:serviceId:TemperatureSensor1",
			"Variable": "CurrentTemperature"
		}
	],

Sorry for the stupid questions!

[akbooer]

There are no stupid questions... just poor documentation.

You're referring to the instructions here?

https://smarthome.community/topic/20/openluup-version-log/12?_=1615649462664

Not entirely sure where you are putting your code? Lua Startup??

[ArcherS]

@akbooer Yes it the instructions in you post February 11th.

At the moment I have not put any code anywhere since I am not sure where it should go, and what code to write!

(For the example from the bridge from @therealdb it is in a separate .json file. I think it may have been confusing of me to add it to the post. I added it just to show the topic, OL deviceId, etc for my test device.)

[akbooer]

Ah, OK. If I understand it, you want to subscribe to that topic (not quite sure about the TopicPath variable) and update the device variable referenced.

So all you need, in Startup Lua is:

function MyMQTThandler (topic, message)
  luup.variable_set ("urn:upnp-org:serviceId:TemperatureSensor1", "CurrentTemperature",  message, 216)
end

luup.register_handler ("MyMQTThandler", "mqtt:tele/tasmota_test/SENSOR/AM2301.Temperature")

I may have the topic wrong, I'm just guessing that I should concatenate the two parts in your JSON, but maybe not.

You can easily generalise this to work for more sensors and use wildcard topic subscriptions.

[ArcherS]

@akbooer thanks!
I will try it and report back, I think it explains the missing parts I needed!

[ArcherS]

@akbooer Without adding the function and handler into the Startup Lua I noticed I get this error message when doing "Reload Luup engine" that I have not seen with the old OpenLuup version:

3/13/2021, 5:12:40 PM	#1: openLuup:./openLuup/timers.lua:349: bad argument #1 to 'max' (number expected, got nil)

Do you know what this can be?

This is what I have added into Startup Lua:

luup.attr_set ("openLuup.MQTT.Port", 1883)
luup.attr_set ("openLuup.MQTT.Username", "user")
luup.attr_set ("openLuup.MQTT.Password", "passwd")

[akbooer]

New to me, I haven't seen that on my three systems currently running the latest. But the error message is a start for me to look.

Speaking of errors, what I have seen is an AltUI startup error (in the delayed startup code, so not quite the first bits mentioning AltUI in the log.) I know what this is, and how to fix it, and have suggested that to amg0. But it doesn't happen on all my systems, so have yet to track down the real root cause. Would be interested if you see anything similar.