luaJIT

[rafale77]

@akbooer,

Have you ever considered running openLuup under LuaJIT instead of Lua5.1 interpreter?

The LuaJIT Project

For kicks, I switched my installation to the latest LuaJIT2.1 beta3 dating from 2017 and being a lua5.1 interpreter it is a drop in replacement. I had to move the lfs library from one folder to another likely because of an include path difference but I am seeing a further speed/efficiency improvement vs. lua5.1.
My CPU load has dropped another 25% to now hover around 2.2% down from 2.8%. I have yet to make sure that everything else works but so far everything appears to be the same.
The luajit site claims up to 5x speed increase through the interpreter and 120x improvement through the compiler vs the standard lua5.1.

My initial interest was from being able to access some ffi libraries to handle video streams...

PS: Wow it is pretty impressive under load... browsing through ALTUI which used to cause spikes of CPU utilization from 2.8% to 4.8% now only go from 2.3% to 2.6%. It almost looks like noise. I am running it on a pretty fast x64 thread so it may not be that significant but this could make a big difference on ARM.

[akbooer]

@rafale77 said in Luajit:

Have you ever considered running openLuup under LuaJIT instead of Lua5.1 interpreter

I run under LuaJIT every day, as a matter of course in my development system. It’s hard to tell the performance improvement without direct metrics, but it’s certainly faster.

With cpu rates at around a few percent, does it really matter? I’d remind you that Vera typically ran at 20% when doing absolutely nothing!

But it’s certainly a viable option if you want (need?) to squeeze the last bit of performance.

[rafale77]

Thank you for confirming! I had seen this name around but never really knew what it was and never investigated it. I just ran into it, as I said, finding out that it opens to some additional extensions I was interested in so the perfomance is just a side benefit.
That being said, more efficient system means more scalability and less power consumption and heat... even if it is a mere few percent.

For others who may be interested, here is some reading references on how it opens the door to extending lua

luapower - The LuaJIT distribution for Windows, Linux and OS X

Performance: x86/x64

Extensions

Including this blog post from an 20 year old coder

A Deep Introduction to JIT Compilers: JITs are not very Just-in-time | kipply's blog

As an example, one can now use libcurl directly within lua!

[rafale77]

And this is what got my attention:

GitHub - anibali/lua-ffmpeg-ffi: LuaJIT FFI bindings to FFmpeg libraries

LuaJIT FFI bindings to FFmpeg libraries. Contribute to anibali/lua-ffmpeg-ffi development by creating an account on GitHub.

Ability to process video streams within lua.

[rafale77]

I just compiled and installed the OpenResty branch of luaJIT along with updating libc.

GitHub - openresty/luajit2: OpenResty's Branch of LuaJIT 2

OpenResty's Branch of LuaJIT 2. Contribute to openresty/luajit2 development by creating an account on GitHub.

After reboot, the cpu load is now... ~1.5%

You can see the steps below:

On the left it is using the lua5.1 interpreter hovering around 2.8%, the middle plateau is using a 2017 beta version of luaJIT: ~2.2%, the rightmost plateau at ~1.5% is after the latest update...
As a reminder I previously reduced the cpu load by swapping the json module from cjson to rapidjson which only lowered the "idle" cpu load baseline from 3.2% to 2.8% but was most impactful when the dkjson module was in use (i.e ALTUI page loading) which reduced the spike from >10% down to <5%.... to now not even showing a peak. It remains <2% (they are the little peaks on the right side going up to ~1.8%). Overall these steps reduced idle CPU load by ~50% and ALTUI browsing load by ~95%.

[akbooer]

Really interesting (and quite new) paper from the same stable as Lua, discussing, amongst other things, JIT in the context of high performance computing and Lua.

http://www.inf.puc-rio.br/~hgualandi/papers/Gualandi-2020-SCP.pdf

[DesT]

guys....

so we can just install luajit package and instead of calling

lua5.1 openLuup/init.lua

we can use luajit openLuup/init.lua ?

[akbooer]

That’s effectively what I do.

In actual fact, I develop within the ZeroBrane Studio IDE, and simply select LuaJIT as the runtime environment. Not sure if you need anything in the way of libraries compiled that way too, but should be OK.

[akbooer]

Pallene seems real enough, in active development:

GitHub - pallene-lang/pallene: Pallene Compiler

Pallene Compiler. Contribute to pallene-lang/pallene development by creating an account on GitHub.

Go on, @rafale77, give it a go!

[rafale77]

@DesT said in luaJIT:

guys....

so we can just install luajit package and instead of calling

lua5.1 openLuup/init.lua

we can use luajit openLuup/init.lua ?

Yup Drop-in replacement. It even reuses all the lua5.1 library folders.

@akbooer said in luaJIT:

Pallene seems real enough, in active development:

GitHub - pallene-lang/pallene: Pallene Compiler

Pallene Compiler. Contribute to pallene-lang/pallene development by creating an account on GitHub.

Go on, @rafale77, give it a go!

Only thing which makes me hesitant is lua5.3... Are there deprecation issues?

[akbooer]

@rafale77 said in luaJIT:

Only thing which makes me hesitant is lua5.3... Are there deprecation issues?

Oh, absolutely. The whole arrangement for function environments changed in v5.2.

I was thinking you might just want to fire it up on a stand-alone program, but I'll ask again when I have a v5.3 version of openLuup. It's actually 'only' a matter of changing a few places in the loader and scheduler modules. I've never done it because I felt it necessary to retain as much compatibility with Vera as possible. But now, it really doesn't matter – aside from installing Lua in the first place, there should anyway be no apparent change for any plugin or from the user POV.

I actually have more need of this in my other current project, so perhaps I'll try it there first.

Anyway, as with your investigations for LuaJIT, it's nice to have an option for future speed-ups as our HA needs get more sophisticated.

[rafale77]

Looking forward a lua5.3 openLuup!!

I have been spending my "dev" time on optimizing facial detection... on python. It is a pretty convoluted collage of libraries and coordination of threads to distribute between cpu and gpu... so much to learn...

[rafale77]

Well I finally found something which does not work on luajit: http-digest.
I will be experimenting with some other http libraries...

edit: I am thinking about switching my one plugin requiring digest authentication to cURL:

GitHub - Lua-cURL/Lua-cURLv3: Lua binding to libcurl

Lua binding to libcurl. Contribute to Lua-cURL/Lua-cURLv3 development by creating an account on GitHub.

GitHub - LPGhatguy/luajit-request: Simple HTTPS for LuaJIT!

Simple HTTPS for LuaJIT! Contribute to LPGhatguy/luajit-request development by creating an account on GitHub.

Wondering if it could be a good alternative to luasocket for openluup's http construct...

[akbooer]

I’ve looked at Lua-cURL previously to replace the ‘temporary’ fix using os.execute("curl ...") in the AltAppStore plugin (to get around some LuaSocket issues.). It’s a possibility.

[rafale77]

I have fixed my digest problem using the luajit-request library which indeed relies on the libcurl integration in luaJIT's ffi.
I am not quite sure why it broke the http-digest library though but overall cURL may be a better supported/more robust solution than luasocket for http...

I also found this:

lua-http

[akbooer]

@rafale77 said in luaJIT:

I also found this:

lua-http

Yes, seen that. The difficulty here is likely the dependency on the Posix cqueues library.

What problem are you actually trying to fix?

[rafale77]

You are right... this library won't work on non POSIX OS so it may not be the best choice.
My setup works now but I am a bit off a tangent having to use an ffi library to fix a digest call.
I was just trying to bring it back to something which can be used more broadly in case others run into this problem:
on lua5.1... works as is with the old http-digest
on luajit... need to switch to luajit-request which depends on ffi-libcurl.

cqueues is what that library uses for... async calls. I am wondering how it compares to your async implementation.

[akbooer]

IIRC, the openLuup implementation of luup.inet.wget() implements digest headers.

[rafale77]

It does, I tried it. It works with the interpreter. It is inspired from the http-digest I had been using and does not work on luajit... and I don't know why.

[akbooer]

Well, it has a dependence on the md5 library, does that work under LuaJIT?

If not, there are other ways of achieving that.