Urgent Help MSR stopped running
-
You should not be running npm as root.
-
The syslog is massive and I am not sure what I am looking for, however I just saw this:
Mar 18 18:03:33 HP-Thin01 node[407]: NODE_PATH /home/stuart/reactor Mar 18 18:03:33 HP-Thin01 node[407]: node:internal/fs/utils:344 Mar 18 18:03:33 HP-Thin01 node[407]: throw err; Mar 18 18:03:33 HP-Thin01 node[407]: ^ Mar 18 18:03:33 HP-Thin01 node[407]: Error: ENOSPC: no space left on device, write Mar 18 18:03:33 HP-Thin01 node[407]: at Object.writeSync (node:fs:884:3) Mar 18 18:03:33 HP-Thin01 node[407]: at Object.writeFileSync (node:fs:2159:26) Mar 18 18:03:33 HP-Thin01 node[407]: at Object.<anonymous> (/home/stuart/reactor/app.js:236:133) Mar 18 18:03:33 HP-Thin01 node[407]: at Module._compile (node:internal/modules/cjs/loader:1101:14) Mar 18 18:03:33 HP-Thin01 node[407]: at Object.Module._extensions..js (node:internal/modules/cjs/loader:1153:10) Mar 18 18:03:33 HP-Thin01 node[407]: at Module.load (node:internal/modules/cjs/loader:981:32) Mar 18 18:03:33 HP-Thin01 node[407]: at Function.Module._load (node:internal/modules/cjs/loader:822:12) Mar 18 18:03:33 HP-Thin01 node[407]: at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:81:12) Mar 18 18:03:33 HP-Thin01 node[407]: at node:internal/main/run_main_module:17:47 { Mar 18 18:03:33 HP-Thin01 node[407]: errno: -28, Mar 18 18:03:33 HP-Thin01 node[407]: syscall: 'write', Mar 18 18:03:33 HP-Thin01 node[407]: code: 'ENOSPC' Mar 18 18:03:33 HP-Thin01 node[407]: } Mar 18 18:03:33 HP-Thin01 systemd[1]: reactor.service: Main process exited, code=exited, status=1/FAILURE Mar 18 18:03:33 HP-Thin01 systemd[1]: reactor.service: Failed with result 'exit-code'.
If I am running out of disk space, I am not sure really how and where to find out what is talking up all the space and how to clear it.
I am only running MSR, Node-Red and HA Bridge nothing else on this box.
Thanks
-
Ok, so now look at the logs again. Disk space problem hiding others, now you should be able to see them.
-
OK these are the latest entries in the syslog to do with reactor
Mar 18 22:07:48 HP-Thin01 node[522]: NODE_PATH /home/stuart/reactor Mar 18 22:07:48 HP-Thin01 node[522]: FileLogStream { type: 'file', name: 'reactor.log', maxsize: 2, keep: 5 } Mar 18 22:07:48 HP-Thin01 node[522]: [latest-22069]2022-03-18T22:07:48.123Z <app:null> Reactor latest-22069-ebeefad starting on v16.13.1 Mar 18 22:07:48 HP-Thin01 node[522]: [latest-22069]2022-03-18T22:07:48.126Z <app:INFO> Process ID 522; platform linux/x64 #1 SMP Debian 4.19.208-1 (2021-09-29); locale en_GB.UTF-8 Mar 18 22:07:48 HP-Thin01 node[522]: [latest-22069]2022-03-18T22:07:48.127Z <app:INFO> Basedir /home/stuart/reactor; data in /home/stuart/reactor/storage Mar 18 22:07:48 HP-Thin01 node[522]: [latest-22069]2022-03-18T22:07:48.128Z <app:INFO> NODE_PATH=/home/stuart/reactor Mar 18 22:07:48 HP-Thin01 node[522]: [latest-22069]2022-03-18T22:07:48.135Z <app:INFO> Configured locale (undefined); selected locale(s) en_GB.UTF-8 Mar 18 22:07:48 HP-Thin01 node[522]: [latest-22069]2022-03-18T22:07:48.195Z <app:INFO> Loaded locale en-US Mar 18 22:07:48 HP-Thin01 node[522]: [latest-22069]2022-03-18T22:07:48.223Z <Capabilities:null> Module Capabilities v21333 Mar 18 22:07:48 HP-Thin01 node[522]: [latest-22069]2022-03-18T22:07:48.354Z <Plugin:null> Module Plugin v21186 Mar 18 22:07:48 HP-Thin01 node[522]: [latest-22069]2022-03-18T22:07:48.369Z <TimerBroker:null> Module TimerBroker v21333 Mar 18 22:07:48 HP-Thin01 node[522]: [latest-22069]2022-03-18T22:07:48.381Z <default:INFO> Module Entity v22042 Mar 18 22:07:48 HP-Thin01 node[522]: [latest-22069]2022-03-18T22:07:48.391Z <Controller:null> Module Controller v22003 Mar 18 22:07:48 HP-Thin01 node[522]: [latest-22069]2022-03-18T22:07:48.392Z <default:null> Module Structure v22047 Mar 18 22:07:48 HP-Thin01 node[522]: [latest-22069]2022-03-18T22:07:48.413Z <default:null> Module Ruleset v21096 Mar 18 22:07:48 HP-Thin01 node[522]: [latest-22069]2022-03-18T22:07:48.414Z <default:null> Module Rulesets v21096 Mar 18 22:07:48 HP-Thin01 node[522]: [latest-22069]2022-03-18T22:07:48.433Z <GlobalExpression:null> Module GlobalExpression v21333 Mar 18 22:07:48 HP-Thin01 node[522]: [latest-22069]2022-03-18T22:07:48.474Z <Predicate:null> Module Predicate v22067 Mar 18 22:07:48 HP-Thin01 node[522]: [latest-22069]2022-03-18T22:07:48.484Z <AlertManager:null> Module AlertManager v21355 Mar 18 22:07:48 HP-Thin01 node[522]: [latest-22069]2022-03-18T22:07:48.490Z <default:null> Module Rule v22043 Mar 18 22:07:48 HP-Thin01 node[522]: [latest-22069]2022-03-18T22:07:48.500Z <GlobalReaction:null> Module GlobalReaction v21333 Mar 18 22:07:48 HP-Thin01 node[522]: [latest-22069]2022-03-18T22:07:48.504Z <default:null> Module Engine v22053 Mar 18 22:07:48 HP-Thin01 node[522]: [latest-22069]2022-03-18T22:07:48.505Z <default:null> Module httpapi v22021 Mar 18 22:07:48 HP-Thin01 node[522]: [latest-22069]2022-03-18T22:07:48.510Z <default:null> Module httpproxy v21333 Mar 18 22:07:48 HP-Thin01 node[522]: [latest-22069]2022-03-18T22:07:48.554Z <default:null> Module wsapi v21334 Mar 18 22:07:48 HP-Thin01 node[522]: [latest-22069]2022-03-18T22:07:48.558Z <app:NOTICE> Starting Structure... ar 18 22:07:48 HP-Thin01 node[522]: [latest-22069]2022-03-18T22:07:48.558Z <app:NOTICE> Starting Structure... Mar 18 22:07:48 HP-Thin01 node[522]: [latest-22069]2022-03-18T22:07:48.567Z <IndividualFileStrategy:ERR> [IndividualFileStrategy][IndividualFileStrategy#/home/stuart/reactor/storage] failed to read sys_alerts in /home/stuart/reactor/storage/states/sys_alerts.json: [SyntaxError]SyntaxError: Unexpected end of JSON input Mar 18 22:07:48 HP-Thin01 node[522]: Trapped unhandled Promise rejection SyntaxError: Unexpected end of JSON input Mar 18 22:07:48 HP-Thin01 node[522]: at JSON.parse (<anonymous>) Mar 18 22:07:48 HP-Thin01 node[522]: at IndividualFileStrategy.getDataObject (/home/stuart/reactor/server/lib/IndividualFileStrategy.js:123:47) Mar 18 22:07:48 HP-Thin01 node[522]: at Container.getDataObject (/home/stuart/reactor/server/lib/Container.js:98:51) Mar 18 22:07:48 HP-Thin01 node[522]: at Structure.start (/home/stuart/reactor/server/lib/Structure.js:280:631) Mar 18 22:07:48 HP-Thin01 node[522]: at Object.<anonymous> (/home/stuart/reactor/app.js:288:80) Mar 18 22:07:48 HP-Thin01 node[522]: at Module._compile (node:internal/modules/cjs/loader:1101:14) Mar 18 22:07:48 HP-Thin01 node[522]: at Object.Module._extensions..js (node:internal/modules/cjs/loader:1153:10) Mar 18 22:07:48 HP-Thin01 node[522]: at Module.load (node:internal/modules/cjs/loader:981:32) Mar 18 22:07:48 HP-Thin01 node[522]: at Function.Module._load (node:internal/modules/cjs/loader:822:12) Mar 18 22:07:48 HP-Thin01 node[522]: at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:81:12) Mar 18 22:07:48 HP-Thin01 node[522]: SyntaxError: Unexpected end of JSON input Mar 18 22:07:48 HP-Thin01 node[522]: at JSON.parse (<anonymous>) Mar 18 22:07:48 HP-Thin01 node[522]: at IndividualFileStrategy.getDataObject (/home/stuart/reactor/server/lib/IndividualFileStrategy.js:123:47) Mar 18 22:07:48 HP-Thin01 node[522]: at Container.getDataObject (/home/stuart/reactor/server/lib/Container.js:98:51) Mar 18 22:07:48 HP-Thin01 node[522]: at Structure.start (/home/stuart/reactor/server/lib/Structure.js:280:631) Mar 18 22:07:48 HP-Thin01 node[522]: at Object.<anonymous> (/home/stuart/reactor/app.js:288:80) Mar 18 22:07:48 HP-Thin01 node[522]: at Module._compile (node:internal/modules/cjs/loader:1101:14) Mar 18 22:07:48 HP-Thin01 node[522]: at Object.Module._extensions..js (node:internal/modules/cjs/loader:1153:10) Mar 18 22:07:48 HP-Thin01 node[522]: at Module.load (node:internal/modules/cjs/loader:981:32) Mar 18 22:07:48 HP-Thin01 node[522]: at Function.Module._load (node:internal/modules/cjs/loader:822:12) Mar 18 22:07:48 HP-Thin01 node[522]: at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:81:12) Mar 18 22:07:48 HP-Thin01 node[522]: Promise { Mar 18 22:07:48 HP-Thin01 node[522]: <rejected> SyntaxError: Unexpected end of JSON input Mar 18 22:07:48 HP-Thin01 node[522]: at JSON.parse (<anonymous>) Mar 18 22:07:48 HP-Thin01 node[522]: at IndividualFileStrategy.getDataObject (/home/stuart/reactor/server/lib/IndividualFileStrategy.js:123:47) Mar 18 22:07:48 HP-Thin01 node[522]: at Container.getDataObject (/home/stuart/reactor/server/lib/Container.js:98:51) Mar 18 22:07:48 HP-Thin01 node[522]: at Structure.start (/home/stuart/reactor/server/lib/Structure.js:280:631) Mar 18 22:07:48 HP-Thin01 node[522]: at Object.<anonymous> (/home/stuart/reactor/app.js:288:80) Mar 18 22:07:48 HP-Thin01 node[522]: at Module._compile (node:internal/modules/cjs/loader:1101:14) Mar 18 22:07:48 HP-Thin01 node[522]: at Object.Module._extensions..js (node:internal/modules/cjs/loader:1153:10) Mar 18 22:07:48 HP-Thin01 node[522]: at Module.load (node:internal/modules/cjs/loader:981:32) Mar 18 22:07:48 HP-Thin01 node[522]: at Function.Module._load (node:internal/modules/cjs/loader:822:12) Mar 18 22:07:48 HP-Thin01 node[522]: at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:81:12) Mar 18 22:07:48 HP-Thin01 node[522]: } Mar 18 22:07:48 HP-Thin01 node[522]: Trace Mar 18 22:07:48 HP-Thin01 node[522]: at process.<anonymous> (/home/stuart/reactor/app.js:271:162) Mar 18 22:07:48 HP-Thin01 node[522]: at process.emit (node:events:390:28) Mar 18 22:07:48 HP-Thin01 node[522]: at process.emit (node:domain:475:12) Mar 18 22:07:48 HP-Thin01 node[522]: at emit (node:internal/process/promises:136:22) Mar 18 22:07:48 HP-Thin01 node[522]: at processPromiseRejections (node:internal/process/promises:242:25) Mar 18 22:07:48 HP-Thin01 node[522]: at processTicksAndRejections (node:internal/process/task_queues:97:32) Mar 18 22:07:48 HP-Thin01 node[522]: [latest-22069]2022-03-18T22:07:48.582Z <app:ERR> Trapped unhandled Promise rejection: [SyntaxError]SyntaxError: Unexpected end of JSON input Mar 18 22:07:48 HP-Thin01 node[522]: [latest-22069]2022-03-18T22:07:48.582Z <app:ERR> Please refer to the console log for trace Mar 18 22:07:48 HP-Thin01 systemd[1]: reactor.service: Succeeded.
Mar 18 22:05:28 HP-Thin01 node[402]: [latest-22069]2022-03-18T22:05:28.048Z <IndividualFileStrategy:ERR> [IndividualFileStrategy][IndividualFileStrategy#/home/stuart/reactor/storage] failed to read sys_alerts in /home/stuart/reactor/storage/states/sys_alerts.json: [SyntaxError]SyntaxError: Unexpected end of JSON input Mar 18 22:05:28 HP-Thin01 node[402]: Trapped unhandled Promise rejection SyntaxError: Unexpected end of JSON input Mar 18 22:05:28 HP-Thin01 node[402]: at JSON.parse (<anonymous>) Mar 18 22:05:28 HP-Thin01 node[402]: at IndividualFileStrategy.getDataObject (/home/stuart/reactor/server/lib/IndividualFileStrategy.js:123:47) Mar 18 22:05:28 HP-Thin01 node[402]: at Container.getDataObject (/home/stuart/reactor/server/lib/Container.js:98:51) Mar 18 22:05:28 HP-Thin01 node[402]: at Structure.start (/home/stuart/reactor/server/lib/Structure.js:280:631) Mar 18 22:05:28 HP-Thin01 node[402]: at Object.<anonymous> (/home/stuart/reactor/app.js:288:80) Mar 18 22:05:28 HP-Thin01 node[402]: at Module._compile (node:internal/modules/cjs/loader:1101:14) Mar 18 22:05:28 HP-Thin01 node[402]: at Object.Module._extensions..js (node:internal/modules/cjs/loader:1153:10) Mar 18 22:05:28 HP-Thin01 node[402]: at Module.load (node:internal/modules/cjs/loader:981:32) Mar 18 22:05:28 HP-Thin01 node[402]: at Function.Module._load (node:internal/modules/cjs/loader:822:12) Mar 18 22:05:28 HP-Thin01 node[402]: at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:81:12) Mar 18 22:05:28 HP-Thin01 node[402]: SyntaxError: Unexpected end of JSON input Mar 18 22:05:28 HP-Thin01 node[402]: at JSON.parse (<anonymous>) Mar 18 22:05:28 HP-Thin01 node[402]: at IndividualFileStrategy.getDataObject (/home/stuart/reactor/server/lib/IndividualFileStrategy.js:123:47) Mar 18 22:05:28 HP-Thin01 node[402]: at Container.getDataObject (/home/stuart/reactor/server/lib/Container.js:98:51) Mar 18 22:05:28 HP-Thin01 node[402]: at Structure.start (/home/stuart/reactor/server/lib/Structure.js:280:631) Mar 18 22:05:28 HP-Thin01 node[402]: at Object.<anonymous> (/home/stuart/reactor/app.js:288:80) Mar 18 22:05:28 HP-Thin01 node[402]: at Module._compile (node:internal/modules/cjs/loader:1101:14) Mar 18 22:05:28 HP-Thin01 node[402]: at Object.Module._extensions..js (node:internal/modules/cjs/loader:1153:10) Mar 18 22:05:28 HP-Thin01 node[402]: at Module.load (node:internal/modules/cjs/loader:981:32) Mar 18 22:05:28 HP-Thin01 node[402]: at Function.Module._load (node:internal/modules/cjs/loader:822:12) Mar 18 22:05:28 HP-Thin01 node[402]: at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:81:12) Mar 18 22:05:28 HP-Thin01 node[402]: Promise { Mar 18 22:05:28 HP-Thin01 node[402]: <rejected> SyntaxError: Unexpected end of JSON input Mar 18 22:05:28 HP-Thin01 node[402]: at JSON.parse (<anonymous>) Mar 18 22:05:28 HP-Thin01 node[402]: at IndividualFileStrategy.getDataObject (/home/stuart/reactor/server/lib/IndividualFileStrategy.js:123:47) Mar 18 22:05:28 HP-Thin01 node[402]: at Container.getDataObject (/home/stuart/reactor/server/lib/Container.js:98:51) Mar 18 22:05:28 HP-Thin01 node[402]: at Structure.start (/home/stuart/reactor/server/lib/Structure.js:280:631) Mar 18 22:05:28 HP-Thin01 node[402]: at Object.<anonymous> (/home/stuart/reactor/app.js:288:80) Mar 18 22:05:28 HP-Thin01 node[402]: at Module._compile (node:internal/modules/cjs/loader:1101:14) Mar 18 22:05:28 HP-Thin01 node[402]: at Object.Module._extensions..js (node:internal/modules/cjs/loader:1153:10) Mar 18 22:05:28 HP-Thin01 node[402]: at Module.load (node:internal/modules/cjs/loader:981:32) Mar 18 22:05:28 HP-Thin01 node[402]: at Function.Module._load (node:internal/modules/cjs/loader:822:12) Mar 18 22:05:28 HP-Thin01 node[402]: at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:81:12) Mar 18 22:05:28 HP-Thin01 node[402]: } Mar 18 22:05:28 HP-Thin01 node[402]: Trace Mar 18 22:05:28 HP-Thin01 node[402]: at process.<anonymous> (/home/stuart/reactor/app.js:271:162) Mar 18 22:05:28 HP-Thin01 node[402]: at process.emit (node:events:390:28) Mar 18 22:05:28 HP-Thin01 node[402]: at process.emit (node:domain:475:12) Mar 18 22:05:28 HP-Thin01 node[402]: at emit (node:internal/process/promises:136:22) Mar 18 22:05:28 HP-Thin01 node[402]: at processPromiseRejections (node:internal/process/promises:242:25) Mar 18 22:05:28 HP-Thin01 node[402]: at processTicksAndRejections (node:internal/process/task_queues:97:32) Mar 18 22:05:28 HP-Thin01 node[402]: [latest-22069]2022-03-18T22:05:28.097Z <app:ERR> Trapped unhandled Promise rejection: [SyntaxError]SyntaxError: Unexpected end of JSON input
-
The disk going full has resulted in the corruption of your storage folder. You will need to restore a backup.
-
The disk going full has resulted in the corruption of your storage folder. You will need to restore a backup.
-
@cw-kid it's the folder called
storage
in your Reactor folder. -
@cw-kid it's the folder called
storage
in your Reactor folder.Is that the folder with the rulesets in it ?
I have a backup of all the folders and files, but it's not very recent.Hope I am not going to lose the more recent changes to the rules.
-
-
That's good sleuthing and good work. Yes, the
states
folder gets most of the I/O. If it looks like everything else is intact, that's fine. Rules and rule sets would only be damaged if you happened to edit them during the time the disk was full.On the Reactor master device, there are flags for warning you when disk space is becoming limited; rules with notifications may be a good idea, to avoid future unpleasant surprises.
-
I am assuming I have had a lucky escape here then, all my rules look to be present still, I am assuming "States" is just that as the name suggests the current state of the rules and that they will rebuild and repopulate themselves.
I didn't have any auto backup in place as not really sure how to do that in Linux command line, so I was just now and again manually copying the entire contents of the /reactor folder down on to my PC.
-
All correct.
-
In Entities under Reactor System I have all this information
reactor_system.alert_count=0 reactor_system.alert_last=1647657325518 reactor_system.alert_severity=null reactor_system.alerts=[] reactor_system.arch="x64" reactor_system.hostname="HP-Thin01" reactor_system.internet_ok=null reactor_system.platform="linux" reactor_system.reactor_memory_used=138702848 reactor_system.reactor_uptime=1082 reactor_system.system_load=[0.13,0.15,0.11] reactor_system.system_memory_free=2020356096 reactor_system.system_memory_size=3537993728 reactor_system.system_uptime=17002.98 reactor_system.volume_critical_reactor_base=false reactor_system.volume_critical_reactor_data=false reactor_system.volume_critical_reactor_logs=false reactor_system.volume_reactor_base=[5390756,13825496,0.389,"/home/stuart/reactor"] reactor_system.volume_reactor_data=[5390756,13825496,0.389,"/home/stuart/reactor/storage"] reactor_system.volume_reactor_logs=[5390756,13825496,0.389,"/home/stuart/reactor/logs"] sys_system.state=true Capabilities: reactor_system, sys_system Actions: reactor_system.clear_alert, reactor_system.clear_alerts, sys_system.restart
Memory free is that disk or RAM ?
-