MSR SMTP Notifications fail - <wsapi:CRIT> Error: Engine has not been started
-
MSR Version: latest-23218
Running as Docker Container : Docker version 24.0.5, build ced0996
OS: Linux 15BPMSR1 6.1.26-05272-g26c406245a2c #1 SMP PREEMPT_DYNAMIC Thu Apr 27 10:15:40 UTC 2023 aarch64 aarch64 aarch64 GNU/LinuxSMTP notifications had been working fine and I am unsure exactly what action(s) stopped it from working.
I have verified that the SMTP server configured in notifications.yml is working properly.
To duplicate the problem I am forcing an SMTP email notification using the set reaction play button, the reactor.log shows the following errors:
[latest-23218]2023-08-21T14:23:22.462Z <wsapi:ERR> [WSAPI]wsapi#1 can't queue reaction rule-lgxmrba4:S: [Error] Engine has not been started [-] [latest-23218]2023-08-21T14:23:22.463Z <wsapi:CRIT> Error: Engine has not been started [-] Error: Engine has not been started at Engine.queueReaction (/opt/reactor/server/lib/Engine.js:1494:46) at WSAPI.clientMessage (/opt/reactor/server/wsapi/wsapi.js:782:64) at WebSocket.<anonymous> (/opt/reactor/server/wsapi/wsapi.js:301:157) at WebSocket.emit (node:events:527:28) at WebSocket.emit (node:domain:475:12) at Receiver.receiverOnMessage (/opt/reactor/node_modules/ws/lib/websocket.js:1068:20) at Receiver.emit (node:events:527:28) at Receiver.emit (node:domain:475:12) at Receiver.dataMessage (/opt/reactor/node_modules/ws/lib/receiver.js:517:14) at Receiver.getData (/opt/reactor/node_modules/ws/lib/receiver.js:435:17)
I have tried:
OS level: apt udate and apt upgrade and reboot
docker pull toggledbits/reactor:latest-aarch64 latest-aarch64: Pulling from toggledbits/reactor Digest: sha256:d8a019d457c9fef61f7c3e6908db1f1d0a2d14b6f7c5e9d4fc536c791503d152 Status: Image is up to date for toggledbits/reactor:latest-aarch64 docker.io/toggledbits/reactor:latest-aarch64
docker stop reactor
docker start reactordocker compose down
docker compose up -dTo no avail.
I feel like I'm missing something obvious but can't put my finger on it.
Thank you in advance for any guidance.
-bh -
Your Engine didn't start. Maybe other recent config changes? Look earlier in the log file, at the startup, for additional messages.
-
Thanks for the input.
I'm not sure if deleting this zero length file was the right thing to do or not but it fixed the problem, the engine starts now and SMTP works.
From the log: [latest-23218]2023-08-21T20:55:09.695Z <IndividualFileStrategy:ERR> [IndividualFileStrategy][IndividualFileStrategy#/var/reactor/storage/states] failed to read cs-rule-lbhys108 in /var/reactor/storage/states/cs-rule-lbhys108.json:
So basically the file /var/reactor/storage/states/cs-rule-lbhys108.json had a file length of "0"
-rwx------ 1 501 staff 0 Jul 20 02:00 cs-rule-lbhys108.jsonI moved the file in question to a tmp directory, restarted reactor, the engine started and SMTP works now.
Question:
Was moving / effectively deleting this file from reactors view a bad thing to do that will cause future problems and is there anything else that need to do other than getting rid of that zero length file?Thanks,
-bh -
Removing the file is fine. It's just a state file used to preserve the rule state across restarts. These are most often corrupted by disk-full conditions, so you might check your disk space.
-
That was likely it now that you mention it. I did have a problem awhile back where I fat fingered a 2:00AM cron job that was supposed to trim backup files to only keep the last x days of *.tar.gz files which filled the partition. I bet the file date reflects when that happened
Thanks again for the help and clarifying. Take care,
-bh -
T toggledbits locked this topic on