Urgent Help MSR stopped running

[cw-kid]

I just restarted MSR and it never came back up again.

I cannot connect to it now in the browser it says connection refused.

I can still SSH in to the Debian Linux box OK, so I rebooted that but still no MSR.

if I run this command:

systemctl --type=service

It says "activating" for Reactor all the others say "active".

I then tried updating MSR to the latest build that seemed to work as normal but MSR is still down and not accessible.

Not sure what happened, I only pressed the Restart button in the MSR gui.

[cw-kid]

I also have Node Red and HA Bridge running on the same Linux box, I can still access those two OK but not MSR..

[kfxo]

Maybe the logs will have a clue? The other day I moved HA to a new computer and MSR was trying to connect to the old machine and MSR would not load. Sure enough the logs indicated that MSR could not connect to HA. Fixed that and all was well.

[cw-kid]

I think you might be onto the right thing. I looked in to the log and it was complaining it can't connect to the InfluxDB running on another Raspberry Pi. That Pi has gone down at the moment and I can't connect to it for some reason.

I don't know when the Pi went down? Might have been last week for all I know or this evening, I am not sure. So may or may not be related.

So I edited the Reactor.yaml file and commented out all lines to do with the InfluxDB plugin and rebooted the Debian Linux box where MSR is running, but MSR is also still down and not accessible.

[cw-kid]

This is the reactor.service file I haven't changed anything in there.

[Unit]
Description=Multi System Reactor
After=network.target

[Service]
Type=simple
User=stuart
Environment=NODE_PATH=/home/stuart/reactor
WorkingDirectory=/home/stuart/reactor
ExecStart=/usr/bin/node app -p
Restart=on-failure
RestartSec=5s

[Install]
WantedBy=multi-user.target

If I run these commands:

sudo systemctl stop reactor

and then

sudo systemctl start reactor

and then

systemctl --type=service

It still says that the reactor service is "activating"

reactor.service loaded activating auto-restart Multi System Reactor

[kfxo]

The logs dont say anything else? If not I dont know and you might have to wait for @toggledbits

[cw-kid]

The timestamps in the logs don't seem to match the actual time here now, so I am having issues working out what is new or not in the logs.

[cw-kid]

I guess MSR is not currently logging as its down. The last line in the log is this at 16:16 pm which was 4 hours ago.

[latest-22053]2022-03-18T16:16:32.655Z InfluxFeed:ERR [InfluxFeed]InfluxFeed#influx write failed (data lost): [Error]Er

And there was lost of these lines:

[latest-22053]2022-03-18T16:16:29.947Z InfluxFeed:ERR [InfluxFeed]InfluxFeed#influx write failed (data lost): [Error]Error: connect ECONNREFUSED 192.168.0.36:8086

But I don't know when the Pi went down? and the InfluxDB so may or may not be related.

[kfxo]

Not sure what else to try as you already commented out the Influx. Maybe back up your MSR files and try a fresh install.

[toggledbits]

Troubleshooting section of the manual.

My guess is (a) some hub isn't connecting and you didn't wait long enough for retries to time out (most common, but not only reason), and then you damaged the config file trying to edit out InfluxDB, which won't get in the way, it's just chatty. Follow the steps in the docs. Also, If you installed node-red after MSR, there is a bug in their install script that can corrupt your nodejs install system-wide. Hopefully it's not that. Back on Monday. Good luck!

[cw-kid]

Initially I commented out all the plugin lines in the reactor.yaml file. After that didn't work I just deleted the entire plugins section from the file and that didn't work either.

[cw-kid]

Can't remember if I installed Node-Red or MSR first. Node-Red has been on this Linux Debian box along with MSR since day 1 when I first set it up.

[cw-kid]

I took a copy of the reactor.yaml file before I edited it, I just restored that and reboot and its still down in the browser.

[toggledbits]

Again, follow the steps in the Troubleshooting section of the docs.

[cw-kid]

@toggledbits

I am trying to download the log files from the /var/log folder now.

Is this bad ?

EDIT:

Was in the wrong directory, this looks better.

[toggledbits]

You should not be running npm as root.

[cw-kid]

I normally login as root. When I run it under my name I get this:

[cw-kid]

Am I running out of disk space?

[cw-kid]

The syslog is massive and I am not sure what I am looking for, however I just saw this:

Mar 18 18:03:33 HP-Thin01 node[407]: NODE_PATH /home/stuart/reactor
Mar 18 18:03:33 HP-Thin01 node[407]: node:internal/fs/utils:344
Mar 18 18:03:33 HP-Thin01 node[407]:     throw err;
Mar 18 18:03:33 HP-Thin01 node[407]:     ^
Mar 18 18:03:33 HP-Thin01 node[407]: Error: ENOSPC: no space left on device, write
Mar 18 18:03:33 HP-Thin01 node[407]:     at Object.writeSync (node:fs:884:3)
Mar 18 18:03:33 HP-Thin01 node[407]:     at Object.writeFileSync (node:fs:2159:26)
Mar 18 18:03:33 HP-Thin01 node[407]:     at Object.<anonymous> (/home/stuart/reactor/app.js:236:133)
Mar 18 18:03:33 HP-Thin01 node[407]:     at Module._compile (node:internal/modules/cjs/loader:1101:14)
Mar 18 18:03:33 HP-Thin01 node[407]:     at Object.Module._extensions..js (node:internal/modules/cjs/loader:1153:10)
Mar 18 18:03:33 HP-Thin01 node[407]:     at Module.load (node:internal/modules/cjs/loader:981:32)
Mar 18 18:03:33 HP-Thin01 node[407]:     at Function.Module._load (node:internal/modules/cjs/loader:822:12)
Mar 18 18:03:33 HP-Thin01 node[407]:     at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:81:12)
Mar 18 18:03:33 HP-Thin01 node[407]:     at node:internal/main/run_main_module:17:47 {
Mar 18 18:03:33 HP-Thin01 node[407]:   errno: -28,
Mar 18 18:03:33 HP-Thin01 node[407]:   syscall: 'write',
Mar 18 18:03:33 HP-Thin01 node[407]:   code: 'ENOSPC'
Mar 18 18:03:33 HP-Thin01 node[407]: }
Mar 18 18:03:33 HP-Thin01 systemd[1]: reactor.service: Main process exited, code=exited, status=1/FAILURE
Mar 18 18:03:33 HP-Thin01 systemd[1]: reactor.service: Failed with result 'exit-code'.

If I am running out of disk space, I am not sure really how and where to find out what is talking up all the space and how to clear it.

I am only running MSR, Node-Red and HA Bridge nothing else on this box.

Thanks

[cw-kid]

OK I have deleted some log files and got some free space back

/dev/sda2 is the main partition by the looks of it.

However after a reboot MSR still isn't loading up.

So my system is knackered currently as all my automation rules run on MSR.