openLuup email server

[akbooer]

@rafale77 said in luaJIT:

Only thing which makes me hesitant is lua5.3... Are there deprecation issues?

Oh, absolutely. The whole arrangement for function environments changed in v5.2.

I was thinking you might just want to fire it up on a stand-alone program, but I'll ask again when I have a v5.3 version of openLuup. It's actually 'only' a matter of changing a few places in the loader and scheduler modules. I've never done it because I felt it necessary to retain as much compatibility with Vera as possible. But now, it really doesn't matter – aside from installing Lua in the first place, there should anyway be no apparent change for any plugin or from the user POV.

I actually have more need of this in my other current project, so perhaps I'll try it there first.

Anyway, as with your investigations for LuaJIT, it's nice to have an option for future speed-ups as our HA needs get more sophisticated.

[rafale77]

Looking forward a lua5.3 openLuup!!

I have been spending my "dev" time on optimizing facial detection... on python. It is a pretty convoluted collage of libraries and coordination of threads to distribute between cpu and gpu... so much to learn...

[rafale77]

Well I finally found something which does not work on luajit: http-digest.
I will be experimenting with some other http libraries...

edit: I am thinking about switching my one plugin requiring digest authentication to cURL:

GitHub - Lua-cURL/Lua-cURLv3: Lua binding to libcurl

Lua binding to libcurl. Contribute to Lua-cURL/Lua-cURLv3 development by creating an account on GitHub.

GitHub - LPGhatguy/luajit-request: Simple HTTPS for LuaJIT!

Simple HTTPS for LuaJIT! Contribute to LPGhatguy/luajit-request development by creating an account on GitHub.

Wondering if it could be a good alternative to luasocket for openluup's http construct...

[akbooer]

I’ve looked at Lua-cURL previously to replace the ‘temporary’ fix using os.execute("curl ...") in the AltAppStore plugin (to get around some LuaSocket issues.). It’s a possibility.

[rafale77]

I have fixed my digest problem using the luajit-request library which indeed relies on the libcurl integration in luaJIT's ffi.
I am not quite sure why it broke the http-digest library though but overall cURL may be a better supported/more robust solution than luasocket for http...

I also found this:

lua-http

[akbooer]

@rafale77 said in luaJIT:

I also found this:

lua-http

Yes, seen that. The difficulty here is likely the dependency on the Posix cqueues library.

What problem are you actually trying to fix?

[rafale77]

You are right... this library won't work on non POSIX OS so it may not be the best choice.
My setup works now but I am a bit off a tangent having to use an ffi library to fix a digest call.
I was just trying to bring it back to something which can be used more broadly in case others run into this problem:
on lua5.1... works as is with the old http-digest
on luajit... need to switch to luajit-request which depends on ffi-libcurl.

cqueues is what that library uses for... async calls. I am wondering how it compares to your async implementation.

[akbooer]

IIRC, the openLuup implementation of luup.inet.wget() implements digest headers.

[rafale77]

It does, I tried it. It works with the interpreter. It is inspired from the http-digest I had been using and does not work on luajit... and I don't know why.

[akbooer]

Well, it has a dependence on the md5 library, does that work under LuaJIT?

If not, there are other ways of achieving that.

[rafale77]

This was my first suspicion too so I manually installed the md5 library through luarocks and made sure I could use it in luaJIT and it seems fine. It did not help http-digest though.

[rafale77]

So... I modified the client.lua file on openLuup to switch to the luajit-request libcurl wrapper I posted above just to see what it does... it dropped my cpu utilization further down to 1.1% from 1.5% at idle while not really impacting the memory load. interesting... My openLuup instance does rely on a lot of http calls, local ones are being handled by the servlet which has not been changed so only changing the ones it is making to the cloud and other local machines seems to be having an impact in spite of their relatively low frequency...

Edit: Well I kidded myself... The drop was actually due to updating the z-way bridge this morning and missing the fact that my openLuup json file was no longer there as I had manually switched everything to rapidjson, essentially killing part of the zway bridge. After correcting back to rapidjson the drop is really within noise (1.5% vs. 1.3%)

[rafale77]

Reviving this thread... Now I am tempted to give Pallene a go just to see if it would have any problem with openLuup.
An interesting read here:

eklausmeier  /  May 14, 2020

Performance Comparison Pallene vs. Lua 5.1, 5.2, 5.3, 5.4 vs. C

This post has moved to eklausmeier.goip.de/blog/2020/05-14-performance-comparison-pallene-vs-lua-5-1-5-2-5-3-5-4-vs-c. Installing Pallene is described in the previous post: Installing Pallene Compi…

Given the amount of array handling openLuup does, I wonder if there is any gain Vs. LuaJIT.

@akbooer, thoughts about a lua5.3/5.4 version of openLuup?

[akbooer]

@rafale77

I considered this in depth a while ago (Aug 2018) here:

Aug 1, 2018

Will Vera ever move beyond Lua 5.1.5?

I think you have to ask “why”? Lua is a tiny language which, from the outset was meticulously designed to be concise, compact, transportable, and easy to use. It’s a huge credit to the implementers that the things which make it so attractive have changed so little since its inception. If you...

...but to bring the discussion onto this forum, here it is:

=======

I think you have to ask “why”?

Lua is a tiny language which, from the outset was meticulously designed to be concise, compact, transportable, and easy to use. It’s a huge credit to the implementers that the things which make it so attractive have changed so little since its inception.

If you look at the development history, outlined here: https://www.lua.org/versions.html, then you’ll see:

5.2

Its main new features are yieldable pcall and metamethods, new lexical scheme for globals, ephemeron tables, new library for bitwise operations, light C functions, emergency garbage collector, goto statement, and finalizers for tables.

5.3

Its main new features are integers, bitwise operators, a basic utf-8 library, and support for both 64-bit and 32-bit platforms.

5.4

Work on Lua 5.4 has begun but there is no release date yet nor a roadmap.

• new generational mode for garbage collection
• userdata can have multiple user values
• debug information about function arguments and returns
• new implementation for math.random

I don’t believe that most casual users would care much about the 5.2 additions, or even know what they meant. 5.1 already has metamethods, coroutines, ephemeral tables, etc. It seems to me that it’s really rather few problems which actually need more than this.

In 5.3, integers are more of an internal thing, there are already bit libraries, but I would grant that utf-8 support might be desirable for systems development. However, just for home automation logic…?

I must admit that I am a huge fan of Lua. I have to thank Vera for introducing me to the language. I chose 5.1 as the implementation language for openLuup, simply to be compatible with Vera. Whilst I’m confident that I could upgrade to 5.2 and beyond (significant changes in function environments due to the lexical scheme for globals, would be the main task) the concept of Vera/MiOS doing so fills me with dread… ever had a problem with one of their firmware upgrades…?

=======

...now, I may have moved on since I wrote this over two years ago, so I'm up for discussion.

Pallene is a somewhat different matter, and I'd be very keen to try this in a couple of areas (specifically, my Prolog-based HA logic engine.) There are benchmarks comparing it to JIT and it shines in a few places, but I don't see a pressing need.

[rafale77]

Ahh bringing back memories... Yeah there is no urgency to do it. It is just an area of interest I just looked at as I was updating my luaJIT build and as I am always looking for further optimization... being an engineer at heart.

[RHCPNG]

This is interesting, but I'm a little bit confused by the detours in the thread. Care to give me a recap of how to move openluup to JIT? I'm running openluup in docker and would like to ask the developer to integrate it in the docker.

[akbooer]

I think that the underlying theme is probably speed/efficiency and minimising load on the host machine.

[RHCPNG]

Yes, that's clear to me. But how to implement it?

[rafale77]

@rhcpng

Sure:
To install LuaJIT, you first need to have lua5.1 installed. Instructions are in the openLuup installation docs so I will skip that.
Then you will need to:

1. download the source code:

git clone https://github.com/openresty/luajit2.git

I used this specific version though it is not the official one but has some additional features I liked. Ohh and also you need to have git installed ("sudo apt-get install git" if not)
Official one is here: https://github.com/LuaJIT/LuaJIT
so you would run

git clone https://github.com/LuaJIT/LuaJIT

2. Compile the code

cd LuaJIT or cd luajit2
make && sudo make install

then in your openLuup installation, edit your "openLuup_reload" file and replace lua5.1 with luajit. And you are done!

[RHCPNG]

@rafale77 Thanks, I hope it will be integrated in the docker.