openLuup email server

[skogen75]

I'm working on a very slow migration from DataYours to Historian and am wondering if there is a list of cache devices that can be modified to include some that are missing, specifically several voltage variable monitors in service EnergyMetering1.

Just checked again and a couple showed up. It could be that these populate slowly after a Vera reboot?

I would also like to modify the storage-aggergation, it is possible to edit that .conf file directly and reboot openLuup?

[akbooer]

@skogen75

Glad that you're moving to Historian – hope the experience is a good one.

The definition of Historian files and archives is in a sort-of halfway house at the moment, witness the comment in openLuup/servertables.lua:

-- TODO: add retentions, xFF and aggregation method, and deprecate indirect reference to .conf files

but you can do what you want to achieve. It's currently a two-step process:

• In the archive_rules table in the above file, you find multiple rules which match the pattern of required variables: deviceNo.shortServiceId.variable, along with a schema name.
• the first-matched rule provides the schema name which is defined in the storage-schemas.conf file (following the Graphite standard) to be found either physically in the Historians history folder or in the openLuup/virtualfilesystem.lua file.

...you can see why I wanted to simplify this!

If this isn't clear, please ask again and I'll try and clarify. From the archive rules, you should be able to see clearly which types of variables are stored. Specifically for EnergyMetering1 the rule is:

{
  schema   = "every_20m", 
  patterns = {"*.*EnergyMetering*.{KWH,Watts,kWh24}"},
}

and in the storage-schemas.conf file you'll find:

[every_20m]
pattern = \.20m$
retentions = 20m:30d,3h:1y,1d:10y

Bear in mind that the Historian only archives numeric variables.

Also recall that the Historian can access your old DataYours files directly from its Whisper folder.

[skogen75]

Great, thanks for pointing me in the right direction. After a restart and waiting a bit for the voltage variable to update, I can see the following...

If I try to put a check mark in the box it won't stick. After I reload the page it is unchecked again. Any thoughts on how to make it stick?

Next, I opened the file you spoke of; question: Would I update the pattern as such...basically just adding the "Voltage" variable?

      schema   = "every_20m", 
      patterns = {"*.*EnergyMetering*.{KWH,Watts,kWh24,Voltage}"},

Second, is this what will be archived in a file or...what is actually going on here with regard to in memory vs. archived in a file? Finally, is the pattern a regex? It seems to be a different format that what what I've seen before, the options in the {} don't pattern match with regex (I'm no expert though).

Last thing, let's say I wanted to change the schema to "every_5m", would I place this pattern under the current every_5m schema or have two every_5m schemas? There is already one present as follows...

      schema   = "every_5m", 
      patterns = {"*.*{openLuup,DataYours,EventWatcher}*.*"},

Thanks for your help on this. Been having some errors pop up with DataYours on a Vera...something about address already in use on the DataCache page. This is stimulating the migration.

[akbooer]

@skogen75 said in Historian:

If I try to put a check mark in the box it won't stick. After I reload the page it is unchecked again.

Yes, this is obviously the way we'd all like it to work, but I haven't implemented that yet. The pop-up comment over the title variable (archived if checked) says:

note that the checkbox field is currently READONLY

@skogen75 said in Historian:

Would I update the pattern as such...basically just adding the "Voltage" variable?

Yes, exactly so. Actually, I'll add this to the next version so that you won't have to change it again. Do we need Current too, I wonder?

@skogen75 said in Historian:

what is actually going on here with regard to in memory vs. archived in a file?

There is an in-memory write-thru cache which keeps up to the last 1024 points since restart, but this is really just an implementation detail, the Historian stores everything in Whisper files (and can also forward to InfluxDB, Grafana, or DataYours.) It does need to be enabled in the Startup Lua with the command

luup.attr_set ("openLuup.Historian.Directory", "history/")

@skogen75 said in Historian:

Finally, is the pattern a regex? It seems to be a different format that what what I've seen before, the options in the {} don't pattern match with regex (I'm no expert though).

No, it's not. This is the language of the Graphite API (it's what Grafana used to extract data for plotting.) See the section on Paths and Wildcards in the Graphite API docs, starting here:

HTTP API — Graphite-API 1.1.3 documentation

@skogen75 said in Historian:

let's say I wanted to change the schema to "every_5m", would I place this pattern under the current every_5m schema or have two every_5m schemas?

No, these patterns have to be unique. You can call the schema what you like, as long as it doesn't conflict with anything else, and you reference that in the archive_rules. If you name it sensibly, I'm happy to include it in future releases too. A point to note is that these are only referenced on file creation – they do not act retrospectively.

[Donato]

Hi akbooer,

will datayours plugin be supported in the future ?

I've several systems using DY.

tnks

donato

[akbooer]

Hi Donato

Not quite sure about the meaning of your question. DataYours, as you know, runs just fine under openLuup, although I don’t plan further development for it. The openLuup Historian is able to render DataYours files through the Graphite API (implemented as a CGI) so you don’t need the DataGraph daemon.

As of the latest development version, the Historian is now able to create multiple CarbonCache repositories, any of which, effectively, have the functionality of the DataCache daemon.

The only things missing from Historian which are in DataYours, AFAIK, are:

• some of the CarbonRelay functionality which allows renaming/duplicating metrics (I wrote this for you)
• a UDP listener to receive data from remote machines (IIRC you use this.)

I think that’s all, but remind me if you use anything else. Maybe this wasn’t what you were asking, so ask again if this didn’t help!

AK

[Donato]

Hi Akbooer,

you remember very well.
I've a central DY receiving data from many remote DY : for this function I use the things you've listed.

Of course I use "storage-aggregation.conf" and "storage-schemas.conf".

I don't use DataGraph but I've implemented a web application to graphing centralized data.

tnks

[akbooer]

Ah yes, the .conf files are supported by the multiple, optional, CarbonCache instances now available in openLuup, although the Historian itself doesn’t need them since it has its own set of rules (and a console page which displays them.)

You should be absolutely fine sticking with what you have and I don’t plan to make it obsolete. The internals of openLuup are still evolving, but the classic Luup interface is frozen and will remain there for legacy plugins.

[ArcherS]

On my new Pi OpenLuup server I cannot get the on-disk archiving to work for some reason.
I have added luup.attr_set ("openLuup.Historian.Directory","history/") to the Lua Startup and reloaded luup but it still does not work.

This is how it looks in the console:

In the cache tab nothing is checked either.

Any idea on what could be the problem? I may have missed something obvious. The server is a Pi 4 with a standard installation of Raspberry Pi OS.

I checked the rights and they are:

drwxrwxrwx 13 pi   pi   4096 jun 11 13:57 cmh-ludl

drwxrwxrwx 2 pi   pi     4096 apr 13 07:48 history

What I did notice is that the ownership in the cmh-luld folder is mixed between "root" and "pi", should it be that?

[CatmanV2]

I don't think given that all your files are rwxrwxrwx that will make any difference. (The owners I mean)

I know nothing about Historian, but do you need to create the directory, or is that attribute meant to indicate it's in the root of the openLuup folder?

C

[ArcherS]

@catmanv2 said in Historian:

I know nothing about Historian, but do you need to create the directory, or is that attribute meant to indicate it's in the root of the openLuup folder?

I think the History directory was created at install from the date.

[CatmanV2]

Ahh right. So the question is 'Does the location of that history folder match the location that you're specifying in the attribute?' I have no idea if you are meant to specify a relative path (and if so, relative to what) or an absolute...

So snap googling gives me this:

The on-disk archive is enabled by the single LuaStartup line luup.attr_set (“openLuup.Historian.Directory”,“history/”), where ‘history/’ is the path (including trailing ‘/’) relative to cmh-ludl/ where you want the data to reside.

So assuming that your history folder is in cmh-ludl (which is appears to be) I'm pretty much out of ideas, other than to check to see if there's anything in the history folder?

C

[ArcherS]

@catmanv2 thanks for the ideas! Yes the path should be correct, it is as stated in the manual (and also what you found with google). I have the same path on my other OpenLuup server and there it works.
The folder is empty so nothing is written to it. At the moment I am out of ideas.

[akbooer]

• what version of openLuup?
• what does the Startup log show?
• what does the regular log show (just after reload)?

I’ve not seen this one before.

[ArcherS]

@akbooer I am on OpenLuup v21.5.23.

The startup log gives no errors, it says this for historian:
2021-06-11 20:46:26.338 openLuup.historian:: version 2021.05.22 @akbooer

The regular log shows this related to historian after a reload:

2021-06-12 10:15:05.253   openLuup.historian:: starting data historian
2021-06-12 10:15:05.253   openLuup.historian:: using on-disk archive: history/
2021-06-12 10:15:05.253   openLuup.historian:: disk archive storage rule sets: 9
2021-06-12 10:15:05.253   openLuup.historian:: using memory cache size (per-variable): 1024

Also no errors in the regular log that could be related.

No idea what it could be, any ideas appreciated.

I thought it could be a rights/ownership issue for the OpenLuup folders, but this could be a false track.
The reason for this is that I had some problems copying some files into it when updating a plugin (the virtual http devices plugin if I remember correctly) a few months ago.
I am not that good at Linux but it looks ok in what I showed above (I think). What is a bit strange is that the ownership varies in the folder, some are owned by "pi" and some by "root", unclear if it should be so or not to be honest.

[CatmanV2]

@archers it really shouldn't matter. The Owners are what they are, but the permissions are read write execute for everyone.

But let's be double clear and ask what the permissions and owner are for the history folder, as I can't see it in any of your screenshots

C

[ArcherS]

@catmanv2 said in Historian:

permissions and owner are for the history folder

They are drwxrwxrwx 2 pi pi 4096 apr 13 07:48 history so they should be ok I think.

[CatmanV2]

@archers Agreed, can't see any issues there.

If you wanted to be absolute sure you could su to pi and touch a file in that folder, but that would be very odd.

C

[ArcherS]

@catmanv2 I copied a file into the history (/home/pi/cmh-ludl/history) folder on the Pi with the file manager via remote desktop and that worked. So it seems ok.

[CatmanV2]

@archers I mean technically unless you were logged in as the same user as Historian is operating as...
....but I'm very very sure permissions and ownership is not your problem

C