openLuup email server

[a-lurker]

Currently I have some Whisper files used by DataYours that been working well for ages and do what I want.

One of the files is called Watts_L1.d.wsp and uses this retention from "storage_schemas_conf" in openLuup file virtualfilesystem.lua:

[day]
pattern = \.d$
retentions = 1m:1d

Inside the actual "Watts_L1.d.wsp" file is a header like so:

         1,      86400,          0,          1
         84,         60,       1440

The 1, 86400 is one minute & one day (in minutes) as per the retention listed above. As a side issue I would like to know what the other header values mean ie what's the syntax here?

New challenge: I now have three Shelly variables named:

em1/0/act_power
em1/1/act_power
em1/2/act_power

with a device ID of "10006" and a SID of "shellypro3em"

And I would like to plot them using the Historian, just like I do with Watts_L1.d.wsp in DataYours. So I need a file in the history directory for the data. So I looked at doing this:

local whisper = require "openLuup.whisper"

-- Syntax:  history/0.deviceNumber.shortServiceId.variableName
local filename = "history/0.10006.shellypro3em.em1/0/act_power.wsp"

local archives = "1m:1d"

whisper.create (filename,archives,0)

Problem is that the variable names contains forward slashes, which are invalid filename characters. What to do?

Also should the retentions now be (to suit the latest openLuup software)?:

local archives = "1m:1d,10m:7d,1h:30d,3h:1y,1d:10y"

Also "shellypro3em" is not a "shortServiceID" as per those listed in "servertables.lua". So can "shellypro3em" be used instead? ie can both short and long service IDs be used in the above call to whisper.create?

[akbooer]

@a-lurker said in openLuup charting and forward slashes in variable names:

As a side issue I would like to know what the other header values mean ie what's the syntax here?

Easily answered: Whisper Database Format

For the rest, I need to do some homework!

[akbooer]

You should absolutely not create any extra files in the Historian folder. You could, however, create what you like in the Whisper folder (or anywhere else.) The Historian can plot files found in the Whisper folder.

You can use any (valid) archive structure you like, but the more archives, the more space used (not that this should be an issue.)

IIRC, the Historian substitutes / in the file names it creates. So I guess that your problem is that these variables aren’t automatically archived in the Historian. In fact, this form of variable is mainly created to make it easier for me to develop the various Shelly device instances. What I should perhaps do is create child devices which are genuine power meters with Luup variables which the Historian will handle.

I need to dig into the code to verify what’s stopping the Historian from handling those variables. It will probably only need another rule added for the cache and archive to change this, but I didn’t really want all those diagnostic variables to be saved as a matter of course.

[a-lurker]

Easily answered: Whisper Database Format

OK

You should absolutely not create any extra files in the Historian folder.

OK - assume it's all meant to be private to openLuup. Best not to mess with its own little world.

IIRC, the Historian substitutes / in the file names it creates.

However whisper.create (filename,archives,0) does not. The string

"whisper/0.10006.shellypro3em.em1/0/act_power.wsp"

is used as is.

Ultimately I would like to have openLuup save the data for any variable I nominate and it gets saved no matter what the variable is called. I imagime forward slashes could be replaced with say dashes (underscores are already in use by Shelly).

The openLuup instance where I have the em1/0/act_power.wsp variable does not have DataYours installed. You have said however that any files in a whisper directory will get updated. Is that right? I though the whisper directory had be specified in DataYours?

So a little confused on how "em1/0/act_power.wsp" or any other variable with slashes can be charted. I can see how having children set up for each device could be set to do this but ultimately you need to be able to plot any variable of ones choosing.

For example the Shelly Pro 4pm also measures power (very useful) but the variable is "switch/0/apower" ie a completely different layout.

Variables don't have to be picked off a list - they could just be manually set up by ruuning a snippet of code such as "whisper.create" as seen above.

[akbooer]

@a-lurker said in openLuup charting and forward slashes in variable names:

Ultimately I would like to have openLuup save the data for any variable I nominate and it gets saved no matter what the variable is called.

That's a very straight-forward way to describe your requirement. Rather than just fixing the power meter issue (especially since, as you point out, the Shelly naming is done in many and various ways) I will take a look at doing this.

@a-lurker said in openLuup charting and forward slashes in variable names:

You have said however that any files in a whisper directory will get updated. Is that right? I though the whisper directory had be specified in DataYours?

I think I said "The Historian can plot files found in the Whisper folder", which isn't, in fact, quite correct. What I should have said is that the Historian back end, based on the industry standard Graphite API, is able to retrieve Whisper data from the DataYours directory (usually whisper/). But you don't have DataYours installed, nor do you need it, and anyway I don't think this is going to be of much help.

Let me address the more general problem that you have now set.

[akbooer]

Closer inspection of the code and my test system shows files can be created and updated with names like

history/543779.6.shellypro3em.em1.0.act_power.wsp

if there is an archive rule defined in the file: servertables.lua that looks something like this:

{
      patterns = {"*.shellypro3em.*"},
      retentions = "1m:1d, 5m:7d",
}

This would then enable all numeric variables with that service, in any device, to be recorded.

Can you try adding such a rule and seeing that those files are created and updated?

The only issue is that they are not recognised by the plotting code...

[akbooer]

Rather than edit the servertables.lua file (which would anyway be overwritten on the next version update), you can simply add some customisation code to Lua Startup...

For example:

do -- bespoke Historian archive rules
  local rules = require "openLuup.servertables" .archive_rules
  rules[#rules+1] = {
      patterns = {"*.shellypro*.*"},
      retentions = "1m:1d, 5m:7d"
    }
end

...will enable archiving of any numeric shellypro device variables.

[a-lurker]

Tried out the above start up code and it has done the job. Now have lots of Shelly based files that can now be plotted using Grafana. They all have the same retentions but so far that's not a problem.

Great - thanks very much.

[akbooer]

Excellent. Yes, it all works fine with Grafana, but not the internal plotting (which, frankly, is pretty minimal.)

Easy to change the rules with the sophisticated wildcard syntax described here, with multiple patterns allowed, as per the servertables.lua defaults.