[Solved] build 26150 - engine not starting
-
@toggledbits I pulled the image (well, Watchtower did) and within minutes the whole system went offline.
The log looks like it ends with 26143.
[latest-26143]2026-05-31T15:23:44.547Z <default:INFO> Closing container Container#Predicate#rule-mkahsmgf/26qq82mw-cons [latest-26143]2026-05-31T15:23:44.547Z <default:INFO> Closing container Container#Predicate#rule-lqyfljfi/22f8on0t-cons [latest-26143]2026-05-31T15:23:44.547Z <default:INFO> Closing container Container#Predicate#rule-lsb61rw8/24oenqi2-cons [latest-26143]2026-05-31T15:23:44.547Z <default:INFO> Closing container Container#Predicate#rule-lrh58he0/rule-lrh58he0:S-1c00gfib-cons [latest-26143]2026-05-31T15:23:44.547Z <default:INFO> Closing container Container#Predicate#rule-lrh58he0/1c00dylr-cons [latest-26143]2026-05-31T15:23:44.547Z <default:INFO> Closing container Container#Predicate#rule-lrh58he0/1nam9w5u-cons [latest-26143]2026-05-31T15:23:44.547Z <default:INFO> Closing container Container#Predicate#rule-miuh2qqi/22ls4lql-cons [latest-26143]2026-05-31T15:23:44.547Z <default:INFO> Closing container Container#Predicate#rule-kwc6rmci/rule-kwc6rmci:S-1vj8sdfc-cons [latest-26143]2026-05-31T15:23:44.547Z <default:INFO> Closing container Container#Predicate#rule-kwc6rmci/rule-kwc6rmci:S-1qanz01x-cons [latest-26143]2026-05-31T15:23:44.547Z <default:INFO> Closing container Container#Predicate#rule-ladyja6a/24lq19p6-cons [latest-26143]2026-05-31T15:23:44.547Z <default:INFO> Closing container Container#Predicate#rule-mk0o8iox/23oy468y-cons [latest-26143]2026-05-31T15:23:44.547Z <default:INFO> Closing container Container#Predicate#rule-miscg2h3/rule-miscg2h3:S-22gmbq1c-cons [latest-26143]2026-05-31T15:23:44.547Z <default:INFO> Closing container Container#Predicate#re-kxgrfjke/238p0old-cons [latest-26143]2026-05-31T15:23:44.547Z <default:INFO> Closing container Container#Predicate#re-m7ccsso5/re-m7ccsso5-1r0myjxa-cons [latest-26143]2026-05-31T15:23:44.547Z <default:INFO> Closing container Container#Predicate#rule-ladyja6a/19nl9wq2-cons [latest-26143]2026-05-31T15:23:44.547Z <default:INFO> Closing container Container#Predicate#rule-ladyja6a/rule-ladyja6a:S-yl3xk9t-cons [latest-26143]2026-05-31T15:23:44.547Z <default:INFO> Closing container Container#Predicate#rule-ladyja6a/rule-ladyja6a:S-yl3vv5m-cons [latest-26143]2026-05-31T15:23:44.547Z <default:INFO> Closing container Container#Predicate#re-ln7j2nqp/re-ln7j2nqp-22mx9lzd-cons [latest-26143]2026-05-31T15:23:44.547Z <default:INFO> Closing container Container#Predicate#re-ln7j2nqp/22mx87c8-cons [latest-26143]2026-05-31T15:23:44.547Z <default:INFO> Closing container Container#Predicate#rule-licneppy/1mzwe7ht-cons [latest-26143]2026-05-31T15:23:44.547Z <default:INFO> Closing container Container#Predicate#rule-ml3194ih/25jqt1j4-cons [latest-26143]2026-05-31T15:23:44.547Z <default:INFO> Closing container Container#Predicate#re-kxgrg7kf/227hshak-cons [latest-26143]2026-05-31T15:23:44.547Z <default:INFO> Closing container Container#Predicate#rule-reactorexmachina/13ua1p95-cons [latest-26143]2026-05-31T15:23:44.547Z <default:INFO> Closing container Container#Predicate#rule-reactorexmachina/13uagam7-cons [latest-26143]2026-05-31T15:23:44.547Z <default:INFO> Closing container Container#Predicate#rule-grpvl9oypg/rule-grpvl9oypg:R-134x4cbv-cons [latest-26143]2026-05-31T15:23:44.547Z <default:INFO> Closing container Container#Predicate#rule-grpvl9oypg/rule-grpvl9oypg:R-134x2fyl-cons [latest-26143]2026-05-31T15:23:44.547Z <default:INFO> Closing container Container#Predicate#re-lscjrws1/238p5c22-cons [latest-26143]2026-05-31T15:23:44.547Z <default:INFO> Closing container Container#Predicate#rule-lbwr0jvq/1xkczf03-cons [latest-26143]2026-05-31T15:23:44.547Z <Structure:NOTICE> Structure Structure#1 stopped [latest-26143]2026-05-31T15:23:44.547Z <app:NOTICE> Closing APIs... [latest-26143]2026-05-31T15:23:44.547Z <wsapi:NOTICE> wsapi: closing... [latest-26143]2026-05-31T15:23:44.547Z <wsapi:NOTICE> wsapi: disconnecting from "192.168.1.23#82" (1001 service closing) [latest-26143]2026-05-31T15:23:44.548Z <httpapi:NOTICE> HTTP API closing... [latest-26143]2026-05-31T15:23:44.549Z <wsapi:NOTICE> wsapi: server closed [latest-26143]2026-05-31T15:23:44.549Z <httpapi:INFO> HTTP server closed. [latest-26143]2026-05-31T15:23:44.549Z <app:NOTICE> Stopping timers... [latest-26143]2026-05-31T15:23:44.551Z <app:null> Shutdown complete, process ID 1 [latest-26143]2026-05-31T15:23:44.551Z <app:null> Closing logs... [latest-26143]2026-05-31T15:23:44.551Z <default:null> Closing logI can SSH to the VM. Alas, I do not have the previous image for 26143 as I'm a little too quick sometimes on housekeeping.
-
Huh. I'm running the 26150 amd64 container with no issue... sure would be nice to have logs. If you go for them, startup is where it's at, and you may need to look at the container syslog if an error is being thrown before the logger is up and running. Some early message get written to the console, and that usually ends up directed to the system logs.
-
Huh. I'm running the 26150 amd64 container with no issue... sure would be nice to have logs. If you go for them, startup is where it's at, and you may need to look at the container syslog if an error is being thrown before the logger is up and running. Some early message get written to the console, and that usually ends up directed to the system logs.
@toggledbits if I have time here I'll try to pull the image again and capture logging.
-
Huh. I'm running the 26150 amd64 container with no issue... sure would be nice to have logs. If you go for them, startup is where it's at, and you may need to look at the container syslog if an error is being thrown before the logger is up and running. Some early message get written to the console, and that usually ends up directed to the system logs.
@toggledbits Repulled, same thing happened: unreachable.
2026-05-31T19:33:04.533976919Z at wrapModuleLoad (node:internal/modules/cjs/loader:255:19) 2026-05-31T19:33:04.533978974Z at Module.require (node:internal/modules/cjs/loader:1600:12) 2026-05-31T19:33:04.533980967Z at require (node:internal/modules/helpers:153:16) 2026-05-31T19:33:04.533983008Z 2026-05-31T19:33:04.533984917Z Node.js v24.16.0 2026-05-31T19:33:56.035775922Z Reactor latest-26150-7b86db04 app 26150 configuration from /var/reactor/config 2026-05-31T19:33:56.053129729Z NODE_PATH /opt/reactor:/opt/reactor/node_modules 2026-05-31T19:33:56.054371749Z Failed to parse /var/reactor/config/logging.yaml YAMLException: bad indentation of a mapping entry in "/var/reactor/config/logging.yaml" (56:34) 2026-05-31T19:33:56.054391711Z 2026-05-31T19:33:56.054396228Z 53 | - type: file # filename is derived from rule ID 2026-05-31T19:33:56.054399741Z 54 | keep: 2 2026-05-31T19:33:56.054402822Z 55 | level: 999 2026-05-31T19:33:56.054405736Z 56 | recycle: true# recycle: true 2026-05-31T19:33:56.054408789Z ---------------------------------------^ 2026-05-31T19:33:56.054456413Z At about line 55; near 2026-05-31T19:33:56.054461049Z > level: 5 2026-05-31T19:33:56.054464372Z > streams: 2026-05-31T19:33:56.054509289Z > - type: file # filename is derived from rule ID 2026-05-31T19:33:56.054513314Z > keep: 2 2026-05-31T19:33:56.054515586Z > level: 999 2026-05-31T19:33:56.054517599Z > recycle: true# recycle: true 2026-05-31T19:33:56.054478049Z Use https://yamlvalidator.com/ and correct all indicated errors. 2026-05-31T19:33:56.054522741Z failed to read configuration file /var/reactor/config/logging.yaml; attemping last-known-good 2026-05-31T19:33:56.054619109Z unable to load configuration data for logging 2026-05-31T19:33:56.056778406Z /opt/reactor/server/lib/Logger.js:429 2026-05-31T19:33:56.056793267Z const _0x98a8e2 = a22_0xf250f4, _0x36af7e = log_config['getConfig'](_0x98a8e2(0x14d), _0x98a8e2(0xfd), {}); 2026-05-31T19:33:56.056795940Z ^ 2026-05-31T19:33:56.056798247Z 2026-05-31T19:33:56.056800114Z TypeError: Cannot read properties of undefined (reading 'getConfig') 2026-05-31T19:33:56.056801991Z at Logger.getLogger (/opt/reactor/server/lib/Logger.js:429:63) 2026-05-31T19:33:56.056803906Z at Logger.getDefaultLogger (/opt/reactor/server/lib/Logger.js:460:40) 2026-05-31T19:33:56.056805752Z at Object.<anonymous> (/opt/reactor/server/lib/Logger.js:761:619) 2026-05-31T19:33:56.056807789Z at Module._compile (node:internal/modules/cjs/loader:1854:14) 2026-05-31T19:33:56.056809659Z at Object..js (node:internal/modules/cjs/loader:1985:10) 2026-05-31T19:33:56.056811482Z at Module.load (node:internal/modules/cjs/loader:1577:32) 2026-05-31T19:33:56.056813567Z at Module._load (node:internal/modules/cjs/loader:1379:12) 2026-05-31T19:33:56.056815388Z at wrapModuleLoad (node:internal/modules/cjs/loader:255:19) 2026-05-31T19:33:56.056817221Z at Module.require (node:internal/modules/cjs/loader:1600:12) 2026-05-31T19:33:56.056819018Z at require (node:internal/modules/helpers:153:16) 2026-05-31T19:33:56.056820871Z 2026-05-31T19:33:56.056822616Z Node.js v24.16.0This is what the container logs show.
-
Did you read the log entries? Pretty strong starting point there...
The logs are showing that your
logging.yamlhas an unrecoverable error. First, check to make sure you have only spaces, no tabs, for indenting.If that's not it, indent the
- type: fileline two spaces, as well as thekeep,level, andrecyclelines that follow it. -
Did you read the log entries? Pretty strong starting point there...
The logs are showing that your
logging.yamlhas an unrecoverable error. First, check to make sure you have only spaces, no tabs, for indenting.If that's not it, indent the
- type: fileline two spaces, as well as thekeep,level, andrecyclelines that follow it.@toggledbits I checked, rechecked, RErechecked and couldn't find any issue with that YAML.
That being said, I completely removed the extra rule logging I had added and restarted. Brought down the new image and everything is fine.
So... my question NOW is: why did it run fine until this build? That logging has been there for several weeks.
-
@toggledbits I checked, rechecked, RErechecked and couldn't find any issue with that YAML.
That being said, I completely removed the extra rule logging I had added and restarted. Brought down the new image and everything is fine.
So... my question NOW is: why did it run fine until this build? That logging has been there for several weeks.
@gwp1 said in [Solved] build 26150 - engine not starting:
@toggledbits I checked, rechecked, RErechecked and couldn't find any issue with that YAML.
Upload it to me, I'd like to take a look.
@gwp1 said in [Solved] build 26150 - engine not starting:
So... my question NOW is: why did it run fine until this build? That logging has been there for several weeks.
No idea. But, having once gotten a clean startup, it now has a last known good version of that config file, so whatever it's not happy with won't prevent startup in future.






