openLuup email server

[akbooer]

@rafale77 said in luaJIT:

I also found this:

lua-http

Yes, seen that. The difficulty here is likely the dependency on the Posix cqueues library.

What problem are you actually trying to fix?

[rafale77]

You are right... this library won't work on non POSIX OS so it may not be the best choice.
My setup works now but I am a bit off a tangent having to use an ffi library to fix a digest call.
I was just trying to bring it back to something which can be used more broadly in case others run into this problem:
on lua5.1... works as is with the old http-digest
on luajit... need to switch to luajit-request which depends on ffi-libcurl.

cqueues is what that library uses for... async calls. I am wondering how it compares to your async implementation.

[akbooer]

IIRC, the openLuup implementation of luup.inet.wget() implements digest headers.

[rafale77]

It does, I tried it. It works with the interpreter. It is inspired from the http-digest I had been using and does not work on luajit... and I don't know why.

[akbooer]

Well, it has a dependence on the md5 library, does that work under LuaJIT?

If not, there are other ways of achieving that.

[rafale77]

This was my first suspicion too so I manually installed the md5 library through luarocks and made sure I could use it in luaJIT and it seems fine. It did not help http-digest though.

[rafale77]

So... I modified the client.lua file on openLuup to switch to the luajit-request libcurl wrapper I posted above just to see what it does... it dropped my cpu utilization further down to 1.1% from 1.5% at idle while not really impacting the memory load. interesting... My openLuup instance does rely on a lot of http calls, local ones are being handled by the servlet which has not been changed so only changing the ones it is making to the cloud and other local machines seems to be having an impact in spite of their relatively low frequency...

Edit: Well I kidded myself... The drop was actually due to updating the z-way bridge this morning and missing the fact that my openLuup json file was no longer there as I had manually switched everything to rapidjson, essentially killing part of the zway bridge. After correcting back to rapidjson the drop is really within noise (1.5% vs. 1.3%)

[rafale77]

Reviving this thread... Now I am tempted to give Pallene a go just to see if it would have any problem with openLuup.
An interesting read here:

eklausmeier  /  May 14, 2020

Performance Comparison Pallene vs. Lua 5.1, 5.2, 5.3, 5.4 vs. C

This post has moved to eklausmeier.goip.de/blog/2020/05-14-performance-comparison-pallene-vs-lua-5-1-5-2-5-3-5-4-vs-c. Installing Pallene is described in the previous post: Installing Pallene Compi…

Given the amount of array handling openLuup does, I wonder if there is any gain Vs. LuaJIT.

@akbooer, thoughts about a lua5.3/5.4 version of openLuup?

[akbooer]

@rafale77

I considered this in depth a while ago (Aug 2018) here:

Aug 1, 2018

Will Vera ever move beyond Lua 5.1.5?

I think you have to ask “why”? Lua is a tiny language which, from the outset was meticulously designed to be concise, compact, transportable, and easy to use. It’s a huge credit to the implementers that the things which make it so attractive have changed so little since its inception. If you...

...but to bring the discussion onto this forum, here it is:

=======

I think you have to ask “why”?

Lua is a tiny language which, from the outset was meticulously designed to be concise, compact, transportable, and easy to use. It’s a huge credit to the implementers that the things which make it so attractive have changed so little since its inception.

If you look at the development history, outlined here: https://www.lua.org/versions.html, then you’ll see:

5.2

Its main new features are yieldable pcall and metamethods, new lexical scheme for globals, ephemeron tables, new library for bitwise operations, light C functions, emergency garbage collector, goto statement, and finalizers for tables.

5.3

Its main new features are integers, bitwise operators, a basic utf-8 library, and support for both 64-bit and 32-bit platforms.

5.4

Work on Lua 5.4 has begun but there is no release date yet nor a roadmap.

• new generational mode for garbage collection
• userdata can have multiple user values
• debug information about function arguments and returns
• new implementation for math.random

I don’t believe that most casual users would care much about the 5.2 additions, or even know what they meant. 5.1 already has metamethods, coroutines, ephemeral tables, etc. It seems to me that it’s really rather few problems which actually need more than this.

In 5.3, integers are more of an internal thing, there are already bit libraries, but I would grant that utf-8 support might be desirable for systems development. However, just for home automation logic…?

I must admit that I am a huge fan of Lua. I have to thank Vera for introducing me to the language. I chose 5.1 as the implementation language for openLuup, simply to be compatible with Vera. Whilst I’m confident that I could upgrade to 5.2 and beyond (significant changes in function environments due to the lexical scheme for globals, would be the main task) the concept of Vera/MiOS doing so fills me with dread… ever had a problem with one of their firmware upgrades…?

=======

...now, I may have moved on since I wrote this over two years ago, so I'm up for discussion.

Pallene is a somewhat different matter, and I'd be very keen to try this in a couple of areas (specifically, my Prolog-based HA logic engine.) There are benchmarks comparing it to JIT and it shines in a few places, but I don't see a pressing need.

[rafale77]

Ahh bringing back memories... Yeah there is no urgency to do it. It is just an area of interest I just looked at as I was updating my luaJIT build and as I am always looking for further optimization... being an engineer at heart.

[RHCPNG]

This is interesting, but I'm a little bit confused by the detours in the thread. Care to give me a recap of how to move openluup to JIT? I'm running openluup in docker and would like to ask the developer to integrate it in the docker.

[akbooer]

I think that the underlying theme is probably speed/efficiency and minimising load on the host machine.

[RHCPNG]

Yes, that's clear to me. But how to implement it?

[rafale77]

@rhcpng

Sure:
To install LuaJIT, you first need to have lua5.1 installed. Instructions are in the openLuup installation docs so I will skip that.
Then you will need to:

1. download the source code:

git clone https://github.com/openresty/luajit2.git

I used this specific version though it is not the official one but has some additional features I liked. Ohh and also you need to have git installed ("sudo apt-get install git" if not)
Official one is here: https://github.com/LuaJIT/LuaJIT
so you would run

git clone https://github.com/LuaJIT/LuaJIT

2. Compile the code

cd LuaJIT or cd luajit2
make && sudo make install

then in your openLuup installation, edit your "openLuup_reload" file and replace lua5.1 with luajit. And you are done!

[RHCPNG]

@rafale77 Thanks, I hope it will be integrated in the docker.

[vwout]

The latest version of the Docker image vwout/openluup includes LuaJIT 2.1.0-beta3.
OpenLuup is not (yet) automatically run using LuaJIT. In order to use it, you need to manually change openLuup_reload_for_docker and changes the lines lua5.1 openLuup/init.lua $@ to luajit openLuup/init.lua $@, especially if you are using volumes or a mount as source for /etc/cmh-ludl.

[rafale77]

Some things I found quite fascinating from my testing;:

1. Running luajit interpreter only, meaning with the JIT compiler disabled is giving me about the same speed, with a 30% smaller memory footprint and much more consistent performance as the JIT engine brings in a lot of variability. I think the equivalent performance is due to my exclusive use of rapidjson which is already compiled in C as I have completely eliminated DKjson from my openLuup installation for quite some time so JIT compiling is really not helping.
2. More specific to my VM but I noticed that the luajit/OpenLuup CPU utilization doubles after I take a snapshot of the VM. It's very strange because none of the other programs in the same VM seem to be affected...

[rafale77]

Something I recently noticed: CPU utilization drop even under heavy I/O http call situations. I haven't updated anything on my setup since April besides the regular linux kernel updates inside the VM running openLuup/HKbridge/Z-way-server. Where I used to see spikes to 5-7% CPU utilization, they now spike to 1.6%. I usually get a spike when one of my cameras detect a movement and I get homeassistant to send http calls to openLuup to update whether it is seeing a person/ a dog / a car etc... Not sure what changed in the linux kernel but it is a pretty big drop.
Very noticeable on my system because it is the first time I see my grafana display show such a scale.

Edit: All of a sudden, this is making me less motivated to explore MQTT...