Organizing/ structuring rule sets and rules

[cw-kid]

I just read on the last MSR update that we need to update node.js. I just checked and on my Linux Debian machine its version 16.17.0.

I then at the terminal ran this command but it says I already have the latest version of node.js which doesn't seem right.

sudo apt-get install nodejs
Reading package lists... Done
Building dependency tree
Reading state information... Done
nodejs is already the newest version (16.17.0-deb-1nodesource1).
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

Any tips?

Thanks

EDIT:

I ran these commands instead to update all packages.

sudo apt update

then

sudo apt upgrade

I then did a node -v and now its version 16.20.2

However the announcement says "all nodejs versions earlier than 18 are now end-of-life".

EDIT2:

I then tried these commands:

npm install -g n

n latest

Now a node -v says its version 21.4.0

However after restarting MSR I still get this alert error:

[cw-kid]

Looks like it changed the path of the node installation? Maybe MSR is still looking at the old path?
This was from the terminal output:

 installing : node-v21.4.0
       mkdir : /usr/local/n/versions/node/21.4.0
       fetch : https://nodejs.org/dist/v21.4.0/node-v21.4.0-linux-x64.tar.xz
     copying : node/21.4.0
   installed : v21.4.0 (with npm 10.2.4)

Note: the node command changed location and the old location may be remembered in your current shell.
         old : /usr/bin/node
         new : /usr/local/bin/node
If "node --version" shows the old version then start a new shell, or reset the location hash with:
hash -r  (for bash, zsh, ash, dash, and ksh)
rehash   (for csh and tcsh)

There is a file called "node" under both usr/bin and usr/local/bin directories, I can see these using WinSCP to look at the file structure.

This was also in the terminal before the above:

 npm install -g n

added 1 package, and audited 2 packages in 2s

found 0 vulnerabilities
npm notice
npm notice New major version of npm available! 8.19.4 -> 10.2.5
npm notice Changelog: https://github.com/npm/cli/releases/tag/v10.2.5
npm notice Run npm install -g npm@10.2.5 to update!
npm notice
root@HP-Thin01:~# ^C
root@HP-Thin01:~# npm install -g npm@10.2.5
npm ERR! code EBADENGINE
npm ERR! engine Unsupported engine
npm ERR! engine Not compatible with your version of node/npm: npm@10.2.5
npm ERR! notsup Not compatible with your version of node/npm: npm@10.2.5
npm ERR! notsup Required: {"node":"^18.17.0 || >=20.5.0"}
npm ERR! notsup Actual:   {"npm":"8.19.4","node":"v16.20.2"}

npm ERR! A complete log of this run can be found in:
npm ERR!     /root/.npm/_logs/2023-12-08T14_42_53_914Z-debug-0.log

I am stuck now and don't know how to fix this?

Thanks

[cw-kid]

This is the contents of that log file it mentioned:

0 verbose cli /usr/bin/node /usr/bin/npm
1 info using npm@8.19.4
2 info using node@v16.20.2
3 timing npm:load:whichnode Completed in 1ms
4 timing config:load:defaults Completed in 7ms
5 timing config:load:file:/usr/lib/node_modules/npm/npmrc Completed in 3ms
6 timing config:load:builtin Completed in 3ms
7 timing config:load:cli Completed in 10ms
8 timing config:load:env Completed in 1ms
9 timing config:load:project Completed in 26ms
10 timing config:load:file:/root/.npmrc Completed in 10ms
11 timing config:load:user Completed in 11ms
12 timing config:load:file:/usr/etc/npmrc Completed in 1ms
13 timing config:load:global Completed in 1ms
14 timing config:load:validate Completed in 1ms
15 timing config:load:credentials Completed in 6ms
16 timing config:load:setEnvs Completed in 4ms
17 timing config:load Completed in 73ms
18 timing npm:load:configload Completed in 74ms
19 timing npm:load:mkdirpcache Completed in 4ms
20 timing npm:load:mkdirplogs Completed in 1ms
21 verbose title npm install npm@10.2.5
22 verbose argv "install" "--global" "npm@10.2.5"
23 timing npm:load:setTitle Completed in 5ms
24 timing config:load:flatten Completed in 14ms
25 timing npm:load:display Completed in 24ms
26 verbose logfile logs-max:10 dir:/root/.npm/_logs
27 verbose logfile /root/.npm/_logs/2023-12-08T14_42_53_914Z-debug-0.log
28 timing npm:load:logFile Completed in 25ms
29 timing npm:load:timers Completed in 0ms
30 timing npm:load:configScope Completed in 0ms
31 timing npm:load Completed in 138ms
32 silly logfile start cleaning logs, removing 1 files
33 silly logfile done cleaning log files
34 http fetch GET 200 https://registry.npmjs.org/npm 126ms (cache hit)
35 timing command:install Completed in 275ms
36 verbose stack Error: Unsupported engine
36 verbose stack     at Object.checkEngine (/usr/lib/node_modules/npm/node_modules/npm-install-checks/lib/index.js:14:25)
36 verbose stack     at Install.exec (/usr/lib/node_modules/npm/lib/commands/install.js:109:16)
36 verbose stack     at async module.exports (/usr/lib/node_modules/npm/lib/cli.js:78:5)
37 verbose pkgid npm@10.2.5
38 verbose cwd /root
39 verbose Linux 4.19.0-21-amd64
40 verbose node v16.20.2
41 verbose npm  v8.19.4
42 error code EBADENGINE
43 error engine Unsupported engine
44 error engine Not compatible with your version of node/npm: npm@10.2.5
45 error notsup Not compatible with your version of node/npm: npm@10.2.5
45 error notsup Required: {"node":"^18.17.0 || >=20.5.0"}
45 error notsup Actual:   {"npm":"8.19.4","node":"v16.20.2"}
46 verbose exit 1
47 timing npm Completed in 774ms
48 verbose code 1
49 error A complete log of this run can be found in:
49 error     /root/.npm/_logs/2023-12-08T14_42_53_914Z-debug-0.log

[Pabla]

So i ran into almost the exact same issue and this is how I managed to update NodeJS. Instead of running sudo apt update and sudo apt upgrade I followed the install instructions on NodeJS Github.

To make Reactor happy I copied the path the old nodejs file was in and in my case it was /home/arman/.local/lib/nodejs I believe yours looks like it should be the same. Before doing anything copy and delete the NodeJS folder and put it somewhere where you can access it again if you run into any issues. Following the steps replace anywhere where it says /usr/local/lib/nodejs with /home/{user}/.local/lib/nodejs. This is where I have a band-aid repair to get Reactor back up and working. I renamed the new folder in the /home/{user}/.local/lib/nodejs directory from the new version of NodeJS to the previous version. In my case it I changed it from node-v20.10.0-linux-armv7l to node-v16.15.1-linux-armv7l. Then Reactor was happy and no error saying I need to update Node. This is a temporary fix however.

I am no Linux expert so please take my advice with caution lol!

[gwp1]

I had same issue and went back to the MSR manual where it noted if you didn't have a system install of nodejs MSR would do a local one for just it's user.

I reran the opening steps from the manual and it updated to 18.x just fine.

[cw-kid]

Thanks for the replies and tips.

In the end after more Googling for my Debian Buster install this seems to have fixed it for me.

I removed the new /usr/local/bin/node directory that got installed the other day.

rm /usr/local/bin/node

Then running the command "which node" the path was back to the old path of /usr/bin/node and "node -v" was showing the old version again of 16.20.2

I then found these commands on Google and tried them:

curl -fsSL https://deb.nodesource.com/setup_current.x | sudo -E bash -

sudo apt-get install -y nodejs

This seems to have upgraded the 16.20.2 version with a newer version, I saw this line in the terminal output

Unpacking nodejs (20.5.1-deb-1nodesource1) over (16.20.2-deb-1nodesource1)

Now when I do a node -v it says its version 20.5.1 and the alert message in MSR has now gone.

"which node" still says the path is the original path of /usr/bin/node also.

I have no idea if this was the correct way to do it but I think it has worked.

[wmarcolin]

I think we all have a similar problem. My step-by-step was as follows.

sudo apt remove nodejs
sudo apt remove npm
sudo apt autoremove

sudo apt purge nodejs
sudo apt purge npm
sudo apt autoremove

sudo apt-get update
sudo apt-get install -y ca-certificates curl gnupg
sudo mkdir -p /etc/apt/keyrings
curl -fsSL https://deb.nodesource.com/gpgkey/nodesource-repo.gpg.key | sudo gpg --dearmor -o /etc/apt/keyrings/nodesource.gpg

NODE_MAJOR=20
echo "deb [signed-by=/etc/apt/keyrings/nodesource.gpg] https://deb.nodesource.com/node_$NODE_MAJOR.x nodistro main" | sudo tee /etc/apt/sources.list.d/nodesource.list

sudo apt-get update
sudo apt-get install nodejs -y

Now comes the very rare story, if I'm not as toot I get this information.

node -v
v20.10.0

npm -v
10.2.3

PERFECT!!

But after the su --, I become root and the version that comes is 12? and then I really don't understand. I did the tasks like this, as root and without root, in the same way.

If any expert can help me understand this madness.

Thanks.

[toggledbits]

The PATH environment variable for root is different from that of regular users. You can echo $PATH in each to see the differences. The path is processed in order, first come first served.

[wmarcolin]

Well, I've just solved my problem, whether it's the right way or not I can't say, but it was brute force, I used the "which node" command, I located all the directories regardless of version and deleted them (rm node -r), this while being root. Then I did a fresh install solved.

[toggledbits]

@wmarcolin That may have worked, but it's not strictly right and may lead to complications down the road, like versions of node and npm being out of sync.

NOTE: Some of this can be used as instructions. Read all carefully and understand what you are doing before you actually do anything. If you don't understand, ask first, understand first, then proceed (always with caution and careful attention to detail).

Generally speaking, it's not a good idea to tamper with the system-installed version of any utility if there is one. If you use a command like apt-get install node or similar and it puts a version into /usr/bin, that's where you should leave it, and never change it (unless apt or yum does). That's because the system packages are versioned together as a group, so other system utilities may have dependencies on that system-blessed version of the package and its associated utilities and libraries.

When you want a newer version of node, the generally-recommended way is to do a custom install, which can be done either to a user-specific directory (so that one user can have a private version), or system-wide in a location under /usr/local (usually installed in /usr/local/lib with symbolic links from /usr/local/bin for the executables).

Either way, the version of node that runs is the first one encountered in the PATH environment then in effect.

Reactor has been tested with all three common install methods: the native system packages, when up to date; a /usr/local install; and a user-directory-specific install. All work fine.

Taking the structure of one of my RPis running Raspbian Buster (Debian 10), but should apply to almost any Linux-based platform...

The default system install puts a node and npm in /usr/bin:

pi@rpi4-2:~ $ ls -l /usr/bin/node
-rwxr-xr-x 1 root root 5536 Feb 24  2023 /usr/bin/node
pi@rpi4-2:~ $ /usr/bin/node -v
v10.24.0
pi@rpi4-2:~ $

Notice that when I ran the node command, I ran it with a full path (i.e. as /usr/bin/node and not just typing node). That's because I have a newer version of node installed in /usr/local that is configured to take precedence. And also notice how old that system-installed package is!

My later-version installation in /usr/local/lib (specifically) looks like this:

pi@rpi4-2:~ $ ls -l /usr/local/lib/
total 12
drwxr-xr-x 3 root root  4096 Nov 26 08:59 nodejs
drwxrwsr-x 4 root staff 4096 Nov 26 09:13 python2.7
drwxrwsr-x 3 root staff 4096 May  7  2021 python3.7
pi@rpi4-2:~ $ ls -l /usr/local/lib/nodejs
total 42112
drwxr-xr-x 6 1001 1001     4096 Oct 13 10:03 node-v18.18.2-linux-armv7l
-rw-r--r-- 1 root root 20932900 Oct 13 10:05 node-v18.18.2-linux-armv7l.tar.xz
-rw-r--r-- 1 root root 22182716 Nov 22 07:30 node-v20.10.0-linux-armv7l.tar.xz
pi@rpi4-2:~ $

You can see here, I have a nodejs subdirectory in /usr/local/lib. In here, I've got the downloaded archives for two versions (an 18 and a 20), of which the 18 is currently unpacked in the directory node-v18.18.2-linux-armv7l. That's a completely self-contained package for nodejs, ready to run. All that was needed was to configure it to run.

Configuring this version of node to run took two additional steps. First, I went to /usr/local/bin, and symbolically linked the executables from the v18 package, like this (notice I'm doing this step as root) :

root@rpi4-2:~# cd /usr/local/bin
root@rpi4-2:/usr/local/bin# ln -sf /usr/local/lib/nodejs/node-v18.18.2-linux-armv7l/bin/* .
root@rpi4-2:/usr/local/bin# ls -l
total 4
lrwxrwxrwx 1 root root 61 Dec 11 12:30 corepack -> /usr/local/lib/nodejs/node-v18.18.2-linux-armv7l/bin/corepack
lrwxrwxrwx 1 root root 57 Dec 11 12:30 node -> /usr/local/lib/nodejs/node-v18.18.2-linux-armv7l/bin/node
lrwxrwxrwx 1 root root 56 Dec 11 12:30 npm -> /usr/local/lib/nodejs/node-v18.18.2-linux-armv7l/bin/npm
lrwxrwxrwx 1 root root 56 Dec 11 12:30 npx -> /usr/local/lib/nodejs/node-v18.18.2-linux-armv7l/bin/npx
root@rpi4-2:/usr/local/bin#

The first command changes directory to /usr/local/bin, and the second symbolically links all files in the package's bin subdirectory to the current directory (which is our /usr/local/bin). The ls command shows the result of that operation. So you can see, the executable in /usr/local/bin/node isn't an executable at all, it's a symbolic link to the executable in the package directory. Same with the other nodejs commands in the package.

The last step is to get that executable in /usr/local/bin/node to take precedence over the older version in the system install (i.e. the one in /usr/bin/node). All that is required to do this is that /usr/local/bin/node be seen by the shell before /usr/bin/node. This is done by making sure /usr/local/bin comes before /usr/bin in the PATH environment variable. On most systems, this is already the default case -- it's set up that way and you don't need to do anything. On some systems, you may need to add it. This can be done by adding a snippet like the following to the end of /etc/profile (to make it a system-wide change):

PATH=/usr/local/bin:$PATH
export PATH

This puts /usr/local/bin at the head of the PATH. Again, this usually isn't necessary, because /usr/local/bin is already early in the path on most systems by default. You can check your path by doing echo $PATH as I said before. If you do end up having to make this change, you'll need to log out and log back in to make it take effect. And, any time you edit anything in /etc, it's a good idea to make a backup copy of the unmodified file first, in case you goof something up and need to revert (cp /etc/profile /etc/profile.old is sufficient).

If you're installing a per-user version of nodejs, the structure is fairly similar, but rather than unpacking the nodejs archive into /usr/local/lib/nodejs, you unpack it into ~/.local/lib/nodejs/ (remember ~ is a synonym for $HOME and means the current user's home directory path) with the same structure that I showed above for /usr/local/.... Notice we are not root here!

# For user-local install, all work is done NOT as root. Here, I'm just "pi"
pi@rpi4-2:~ $ mkdir -p ~/.local/lib
pi@rpi4-2:~ $ mkdir -p ~/.local/bin
pi@rpi4-2:~ $ cd ~/.local/lib/
pi@rpi4-2:~/.local/lib $ mkdir -p nodejs
pi@rpi4-2:~/.local/lib $ cd nodejs
pi@rpi4-2:~/.local/lib/nodejs $ wget 'https://nodejs.org/dist/latest-v20.x/node-v20.10.0-linux-armv7l.tar.xz'
--2023-12-11 12:46:55--  https://nodejs.org/dist/latest-v20.x/node-v20.10.0-linux-armv7l.tar.xz
Resolving nodejs.org (nodejs.org)... 104.20.23.46, 104.20.22.46, 2606:4700:10::6814:162e, ...
Connecting to nodejs.org (nodejs.org)|104.20.23.46|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 22182716 (21M) [application/x-xz]
Saving to: ânode-v20.10.0-linux-armv7l.tar.xz.1â

node-v20.10.0-linux-armv7l.tar.x 100%[==========================================================>]  21.15M  18.5MB/s    in 1.1s

2023-12-11 12:46:57 (18.5 MB/s) - ânode-v20.10.0-linux-armv7l.tar.xz.1â saved [22182716/22182716]

pi@rpi4-2:~/.local/lib/nodejs $ ls
node-v20.10.0-linux-armv7l.tar.xz
pi@rpi4-2:~/.local/lib/nodejs $ tar xJf node-v20.10.0-linux-armv7l.tar.xz    # J is for .xz, z is for .gz
pi@rpi4-2:~/.local/lib/nodejs $ cd ~/.local/bin
pi@rpi4-2:~/.local/bin $ ln -sf ~/.local/lib/nodejs/node-v20.10.0-linux-armv7l/bin/* .
pi@rpi4-2:~/.local/bin $ ls -l
total 16
lrwxrwxrwx 1 pi pi 66 Dec 11 12:48 corepack -> /home/pi/.local/lib/nodejs/node-v20.10.0-linux-armv7l/bin/corepack
lrwxrwxrwx 1 pi pi 62 Dec 11 12:48 node -> /home/pi/.local/lib/nodejs/node-v20.10.0-linux-armv7l/bin/node
lrwxrwxrwx 1 pi pi 61 Dec 11 12:48 npm -> /home/pi/.local/lib/nodejs/node-v20.10.0-linux-armv7l/bin/npm
lrwxrwxrwx 1 pi pi 61 Dec 11 12:48 npx -> /home/pi/.local/lib/nodejs/node-v20.10.0-linux-armv7l/bin/npx
pi@rpi4-2:~/.local/bin $

Reading these steps, the first five commands create the structure of the .local subdirectory needed, with subdirectories bin, lib and lib/nodejs within just like we would have in /usr/local. At this point, the current directory is ~/.local/lib/nodejs, and the wget command is used to download the nodejs package we want (change version and architecture to what you need if you are actually doing this). Then we un-tar it right where we are. After un-taring it, we change directory to our own ~/.local/bin and make the symbolic links for the nodejs commands in the package directory, just like we did for /usr/local.

The last step is again, and this may be default, is making sure this user's PATH environment variable includes our new local ~/.local/bin directory. Check it with echo $PATH to see if you need to make the addition. If so, the easiest way to do that is to add the following snipped to ~/.profile, which is the per-user equivalent to /etc/profile (the per-user script runs after the /etc script at login) to set up the shell environment. My Raspbian Buster (Debian 10) systems seem to have this done already as a default, so before making any changes, check your PATH and/or ~/.profile.

PATH=$HOME/.local/bin:$PATH
export PATH

Log out and back in, and then a which node should give you ~/.local/bin/node (or /home/username/.local/bin/node). You then have a per-user custom version of nodejs that doesn't interfere with any system versions.

Now, for running Reactor as a service, you need to be aware of where you are running it, and as who. It's generally not recommended to run Reactor as root on any bare-metal system (it runs as root on docker containers, but that's OK because the container is a constrained virtual environment). So when setting up, for example, your systemd profile for Reactor, you would want to make sure the profile contains both a User directory to set the correct (non-root) user for the running process, and a full path to the correct node executable in ExecStart (e.g. ExecStart=/home/pi/.local/bin/node app -p).

One last point: nodejs is so well-written that its entire dependencies exist entirely within its package structure, so whatever node command runs knows automatically where to find everything else it needs — you don't need to set a bunch of other environment variables to make each version work properly. That means you could, for example, run /usr/bin/node and see a perfectly working v10.24.0 as I showed above, or /usr/local/bin/node and get a perfectly-working v18.18.2 if that's what is installed there, and ~/.local/bin/node to get a fine v20.9.0 environment if that's what's installed there. Not all packages are this tight, and we can be grateful for it.

Tip: don't stop at which node when you are trying to figure out what is running. nodejs itself makes extensive use of symbolic links, so doing which node is only the tip of the iceberg. You need to then do ls -l /that-path and see if it's a symbolic link to somewhere else, as I showed in the output above. Keep following those links until you find the "real deal" and you'll find which version is actually running and where it actually lives.

I hope some of this clarifies how nodejs is installed, and highlights how you can actually have multiple versions of it installed on the system and they co-exist quite well, you just have to be careful.

[CatmanV2]

@toggledbits fabulous post, there. Thanks. 18.19.0 in /usr/local/bin for Reactor and, apparently working fine

Any particular reason not to go to 21?

Cheers

C

[toggledbits]

@CatmanV2 said in Help updating node.js for bare metal install:

Any particular reason not to go to 21?

Yes! Only even-numbered versions of nodejs are LTS (long-term support). For the longest life of the system between upgrades like this, use the LTS versions only. That's why I recommend 18 and 20.

Here's their release and support calendar: https://nodejs.github.io/nodejs.dev/en/about/releases/

Notice that 21 will be EOL far sooner than either 18 or 20.

[CatmanV2]

Doh! Thanks for that!

C

[wmarcolin]

@toggledbits

Super thanks for the master class!!!!