MQTT Plugin

ArcherS · Jan 28, 2021, 7:04 PM

@therealdb I finally decided to try and setup MQTT for my Tasmota and Shelly devices.
After having built the bridge i.a.w. the instructions I have copied it to my Ubuntu PC that also is running OpenLuup. I put all the files in a folder in "Home" and given permissions sudo chmod -R a+rwx /path/to/folder.
(I built the files on my Win 10 PC in an Ubuntu WSL including -r linux-x64.)

The Tasmota test device I use says "MQT: Connected" in the console so it seems that it connects to the bridge when it starts. I also set up MQTT Explorer and it also connects to the bridge and shows the topics from the Tasmota device.

Despite this no temp value is set for the virtual sensor in OpenLuup I use for the test.
I have double checked the topic name, the topic path and the OpenLuup information and it seems correct.

appsettings.json:

{
	"MQTT": {
		"Username": "luup",
		"Password": "openluup",
		"Port": 1883
	},

	"Luup": {
		"ipAddress": "127.0.0.1",
		"port": 3480
	},

	"Devices": [
		{
			"ClientID": "tasmota_test",

			"TopicName": "tele/tasmota_test/SENSOR",
			"TopicPath": "AM2301.Temperature",

			"DeviceID": 216,
			"Service": "urn:upnp-org:serviceId:TemperatureSensor1",
			"Variable": "CurrentTemperature"
		}
	],

	/* remove comment when you're OK with less logs
	"Serilog": {
		"MinimumLevel": {
			"Default": "Information",
			"Override": {
				"Microsoft": "Fatal",
				"System": "Fatal"
			}
		}
	}
	*/
}

Sensor topic on Tasmota console is posted when Tasmota posts the data:

13:58:05 MQT: tele/tasmota_test/SENSOR = {"Time":"2021-01-28T13:58:05","AM2301":{"Temperature":23.5,"Humidity":37.3,"DewPoint":8.1},"TempUnit":"C"}

Device in OpenLuup:

The log file says:

2021-01-28 12:51:57.992 +01:00 [INF] [MQTTServer] Broker is starting on port 1883...
2021-01-28 12:59:21.709 +01:00 [INF] [MQTTServer] Broker is starting on port 1883...
2021-01-28 13:55:27.236 +01:00 [INF] [MQTTServer] Broker is starting on port 1883...
2021-01-28 13:57:48.402 +01:00 [INF] [MQTTServer] Broker is starting on port 1883...

Any ideas on what could be wrong? I have most likely missed something obvious.

//ArcherS

akbooer · Jan 28, 2021, 3:00 PM

Sorry, nothing to suggest. I took a look at the MQTT code and while ago and it's rather poorly written.

FYI, though, I am in the process of writing a Shelly bridge for openLuup, which will have its own MQTT server and be able to automatically create configured devices for any Shellys that use it.

Turns out that it's rather easier to write a minimal MQTT server than it is an MQTT client.

ArcherS · replied to akbooer on Jan 28, 2021, 3:00 PM

Just to clarify, what I installed was the stand alone MQTT Bridge from @therealdb and not the old MQTT plugin. Perhaps it was a bit confusing of me to post it here...

I agree that MQTT in some form for sure is a great addition to OpenLuup and Shelly/Tasmota devices.
Automatically adding devices sounds promising!

//ArcherS

akbooer · wrote on Jan 28, 2021, 4:10 PM

It uses the paho/mqtt library and utilities (albeit updated to v3.) I thought of using that as a base, but after inspection, started from scratch.

Yup, the only config would be to point the Shelly device at the server.

therealdb · replied to ArcherS on Jan 28, 2021, 7:04 PM

@archers Today is a bad day, so I'll take a look at it tomorrow.

Maybe it's time to add some features I added to my branch lately.

therealdb · Jan 29, 2021, 3:11 AM

@ArcherS I did some tests. here's what I got:

[09:08:32 VRB] [MQTTServer] Message Received...
[sonoff-pool] - tele/tasmota/SENSOR - {"Time":"2020-07-26T08:21:19","DS18B20":{"Id":"02131D86D0AA","Temperature":25.3},"TSL2561":{"Illuminance":0.049},"SR04":{"Distance":45.421},"TempUnit":"C"}
[09:08:36 VRB] [MQTTServer] Processed message: got 25.3
[09:08:36 VRB] Matched value
[09:08:37 VRB] [LuupWrapper] Calling Uri: http://127.0.0.1:3480/data_request?id=variableset&output_format=json&DeviceNum=501&serviceId=urn:upnp-org:serviceId:TemperatureSensor1&Variable=CurrentTemperature&Value=25.3&RunAsync=1 2
[09:08:37 VRB] [MQTTServer] Message Received...
[dbdomoticz] - owntracks/update/daniele - {"t":"c","tst":1611145530,"acc":65,"_type":"transition","event":"enter","lon":xx.615871,"topic":"owntracks\/DB\/daniele\/event","lat":xxx.995452999999998,"wtst":1610963724,"tid":"DB","desc":"Home"}

So, you should see at least the commands being reported. Check for permissions for the deamon and look at the console logs of your device. Maybe there's something (username? password? that's not correct.

EDIT: I just pushed a new release updated to .NET 5. Just get the new SDK, the build procedure is the same.

ArcherS · Jan 29, 2021, 11:42 AM

@therealdb the permissions etc look right, also username password are correct, I get "connected" in the Tasmota console, so it seems to connect.

Is your log above from the MqttBridge logfile? If so I cannot see it, it only says:
2021-01-28 12:51:57.992 +01:00 [INF] [MQTTServer] Broker is starting on port 1883...

I think i will try to build it again. When looking in github I cannot see any updates to the files for some reason.

//ArcherS

therealdb · Jan 29, 2021, 2:12 PM

@archers crazy days at work. push is in progress.

yes, this was from my own logs (where I attached owntracks as well - via a mqtt realy I have to test things around that's pushing mqtt messages from my prod system to my test ones).

ArcherS · Jan 29, 2021, 9:15 AM

@therealdb I tried to buid again from the new files, but I cannot get the Luup.MqttBridge to run the way it did before the update.
Unclear what is wrong, I think I did everything in the same was as before. I noticed that the icon on the Luup.MqttBridge file is different when looking at it in Ubuntu "Files".

//ArcherS

therealdb · replied to ArcherS on Jan 29, 2021, 4:52 PM

@archers what's the output of dotnet publish -c Release -r linux-x64? You can do it under Windows and it will produce executables for Linux as well.

If you want just a single file, go with

dotnet publish -c Release -r linux-x64 -p:PublishSingleFile=true --self-contained true

this will produce a single file with everything inside. I'm updating the doc as well.

therealdb · Jan 29, 2021, 6:45 PM

@rogero @ArcherS @rafale77 I just published a compiled version for Linux-64. It's under releases. No SDK need to be installed, just chmod and you're ready to go.

Release Version 0.30 · dbochicchio/luup-mqtt

First compiled release. Just for Linux-x64. Build for Windows or Raspberry using .NET 5 SDK.

If you're running Raspberry and need ARM-64, let me know and I could add it compiled as well.

ArcherS · Jan 29, 2021, 7:02 PM

@therealdb thanks for the build!

When using your latest build the Luup.MqttBridge file for some reason is a "shared library (application/x-sharedlib)" and not as when building from the older version "executable (application/x-executable)" despite having set permissions.
Really strange, I managed to get around this by creating a .sh file containing "./Luup.MqttBridge" and running this.
Over first hurdle...

However when I launch the new built file I still only get this in the log file:

[19:17:23 INF] [MQTTServer] Broker is starting v 0.30.210109 on port 1883...

It is as if it hangs while trying to start/starting.

In the Tasmota console it says connected so the user/passwd seems ok:

19:17:21 MQT: Connected

So obviously not quite there yet.
Any ideas on what the pronlem could be?

//ArcherS

therealdb · replied to ArcherS on Jan 29, 2021, 7:02 PM

@archers post your config and I'll try to replicate. Maybe there's some error not correctly surfacing.
Plus, post a couple of messages you are seeing on the console.

ArcherS · Jan 29, 2021, 2:20 PM

So I did some more testing and found out that I had still "DVES_%06X" as client in the Tasmota device...
I am sure I had tripple checked everything, stupid...

So I changed this to what it should be and now I get it to work , i.e. the device in OpenLuup gets the temp value. Success!

However the log does still not give more that the starting message for some reason.
Any ideas on this? Btw I still have this in the appsettings, not sure if that could be related:?

/* remove comment when you're OK with less logs
	"Serilog": {
		"MinimumLevel": {
			"Default": "Information",
			"Override": {
				"Microsoft": "Fatal",
				"System": "Fatal"
			}
		}
	}
	*/

I will do some more testing later on and see.
Thanks for the assistance!

//ArcherS

therealdb · Jan 30, 2021, 8:36 AM

@archers I'll triple check logs tomorrow. I have a new build ready, specifically supporting raw values to be pushed to devices (ie: owntracks).

ArcherS · replied to therealdb on Jan 30, 2021, 8:36 AM

@therealdb I noticed that the bridge had gone down over night. I restarted it and got the below error message after approx 10-15 minutes when it went down again. It ran for longer yesterday, maybe 1-2 hours before crashing.

I now started the old version (that works, it was of course my error on the Client in the Tasmota that was the problem), just to see if it also will go down.

//ArcherS

[08:55:53 INF] [MQTTServer] Broker is starting v 0.30.210109 on port 1883...
Unhandled exception. System.Threading.Tasks.TaskCanceledException: The request was canceled due to the configured HttpClient.Timeout of 10 seconds elapsing.
 ---> System.TimeoutException: The operation was canceled.
 ---> System.Threading.Tasks.TaskCanceledException: The operation was canceled.
 ---> System.IO.IOException: Unable to read data from the transport connection: Operation canceled.
 ---> System.Net.Sockets.SocketException (125): Operation canceled
   --- End of inner exception stack trace ---
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.GetResult(Int16 token)
   at System.Net.Http.HttpConnection.FillAsync(Boolean async)
   at System.Net.Http.HttpConnection.ReadNextResponseHeaderLineAsync(Boolean async, Boolean foldedHeadersAllowed)
   at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at Microsoft.Extensions.Http.Logging.LoggingHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at Polly.Retry.AsyncRetryEngine.ImplementationAsync[TResult](Func`3 action, Context context, CancellationToken cancellationToken, ExceptionPredicates shouldRetryExceptionPredicates, ResultPredicates`1 shouldRetryResultPredicates, Func`5 onRetryAsync, Int32 permittedRetryCount, IEnumerable`1 sleepDurationsEnumerable, Func`4 sleepDurationProvider, Boolean continueOnCapturedContext)
   at Polly.AsyncPolicy`1.ExecuteAsync(Func`3 action, Context context, CancellationToken cancellationToken, Boolean continueOnCapturedContext)
   at Microsoft.Extensions.Http.PolicyHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at Polly.Bulkhead.AsyncBulkheadEngine.ImplementationAsync[TResult](Func`3 action, Context context, Func`2 onBulkheadRejectedAsync, SemaphoreSlim maxParallelizationSemaphore, SemaphoreSlim maxQueuedActionsSemaphore, CancellationToken cancellationToken, Boolean continueOnCapturedContext)
   at Polly.AsyncPolicy`1.ExecuteAsync(Func`3 action, Context context, CancellationToken cancellationToken, Boolean continueOnCapturedContext)
   at Microsoft.Extensions.Http.PolicyHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at Microsoft.Extensions.Http.Logging.LoggingScopeHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.SendAsyncCore(HttpRequestMessage request, HttpCompletionOption completionOption, Boolean async, Boolean emitTelemetryStartStop, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   --- End of inner exception stack trace ---
   at System.Net.Http.HttpClient.SendAsyncCore(HttpRequestMessage request, HttpCompletionOption completionOption, Boolean async, Boolean emitTelemetryStartStop, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.GetStringAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
   at Luup.MqttBridge.Services.LuupWrapper.RunCommandAsync(Int32 deviceID, String action, String actionCommand, String commandParameter, String command, String serviceID, String value)
   at Luup.MqttBridge.Services.LuupWrapper.UpdateVariablesAsync(Int32 deviceID, String serviceID, String variableName, String value)
   at Luup.MqttBridge.Services.Mqtt.MQTTServer.ProcessMessageAsync(MqttDevice device, MqttApplicationMessage applicationMessage)
   at Luup.MqttBridge.Services.Mqtt.MQTTServer.<>c__DisplayClass9_0.<<StartAsync>b__3>d.MoveNext()
--- End of stack trace from previous location ---
   at System.Threading.Tasks.Task.<>c.<ThrowAsync>b__140_1(Object state)
   at System.Threading.QueueUserWorkItemCallbackDefaultContext.Execute()
   at System.Threading.ThreadPoolWorkQueue.Dispatch()
/home/user/mqttbridge/LuupMqttBridge.sh: line 1: 20380 Aborted                 (core dumped) ./Luup.MqttBridge

therealdb · Jan 30, 2021, 9:47 AM

Yep, I noticed on my own version as well. It's because the HTTP endpoint crashed (are you using openLuup, right?) or timedout after 10 secs. I'll probably make it configurable in next version and be more aggressive towards crashes.
It's still very beta, even if I'm running a modified version in production for my own.

New release coming up on GitHub in minutes

ArcherS · Jan 30, 2021, 9:56 AM

@therealdb yep, I am on OpenLuup.
I tested and the old version also crashes in the same way after 30-60 minutes.

//ArcherS

therealdb · Jan 30, 2021, 10:02 AM

@archers are you able to compile it yourself, for an easy fix? I have a new version ready, but I'll need more debug in a real world scenario. But I'm ready to push it, if you can try. I also added support for alternate topic (useful if you want to intercept two mqtt messages with different topics and payload, but execute the same action) and wildcard support for clienid (useful if you have messages coming from other brokers and the original clientid is lost).

ArcherS · Jan 30, 2021, 10:08 AM

@therealdb I was able to build the previous versions, so I can try the new version if you push it.

openweather plugin ?

Openluup: Datayours

openLuup: Shelly Bridge plugin

zigbee2mqtt and openLuup

UPnP event proxy plugin using systemd service file

Sonos system alerts truncated

Generic support for vacuums

Switchboard plugin

Cannot publish new version in ALTAPP Store

openLuup: UI Device / Tile Text

Reactor scope issues

Virtual Devices Plug-in update (with async HTTP support)

AltUI will not update under openLuup (Vera: not sure what it does)

Reactor: double click action

Virtual Pronto Remote plugin

openLuup: SmartSwitch plugin

Reactor icons for local lan

openLuup: Tasmota MQTT Bridge

AltUi

AltUI sort?

MQTT Plugin

Release Version 0.30 · dbochicchio/luup-mqtt

Recent Topics