Disaster recovery and virtualisation

[CatmanV2]

Hi all
Ran into an issue last month where my Open Luup box ran out of disk space in /var/log. It's Debian with a separate partition so everything carried on working but it had some very interesting effects on my network.

Culprit was the z-way server log which was at about 5G....

So I thought I'd set up logrotate on it. Dead simple. And so it seems. Log is rotate just fine but z-way-server carries on writing to the old log. Any ideas how to fix that (apart from re-starting the server every time we rotate?).

Entry from logrotate.d is here:

catman@openluup:/etc/logrotate.d$ cat z-way-server 
/var/log/z-way-server.log {
	weekly	
	rotate 4
	compress
	delaycompress
	missingok
	notifempty
	create 644 root root
}

z-way-server config:

<config>
    <automation-dir>automation</automation-dir>
    <log-file>/var/log/z-way-server.log</log-file>
    <log-level>2</log-level>
    <debug-port></debug-port>
</config>
config.xml (END)

TIA

C

[akbooer]

I wonder what happens if you move/delete the log file? Does it just carry on with a new one?

[CatmanV2]

@akbooer said in Logrotate advice:

I wonder what happens if you move/delete the log file? Does it just carry on with a new one?

No. Carries on writing to the old one. I've amended it the config to use copytruncate. In theory this copies the existing file to $file.log.1 and then truncates the original file (and the app keeps writing to it)
There is a timeslice where entries may be lost but I'm not concerned. I'm just waiting for the next rotate now

Cheers

C

[toggledbits]

@catmanv2 said in Logrotate advice:

No. Carries on writing to the old one.

That's because the inode (basically a structure that represents allocation of space on disk) still exists in the file system and the service still has the file descriptor open on that inode. Changing the name or location of a file does not change the inode itself, but just a field on it (like renaming a device on Vera doesn't change the device number or ZWave node ID).

In fact, if you delete the file, that just marks it deleted on the inode, but does not delete the inode or free up the space associated with it unless and until all file descriptors open on the inode are closed. Those open file descriptors can continue to write (and consume disk space) now to a file you can't actually see in ls, so the services that use them are still writing to the now-invisible file. Only when that open descriptor count is 0 is the file actually deleted and the inode and its allocated blocks free'd.

Most services need to be notified that the log file has changed, so they can close their file descriptors and open new ones, and you can see this expressed in a variety of ways in the various configs for the system services that are rotated. It's not unusual to see a service configured for restart at the time of log rotation, and this is why.

[CatmanV2]

Yeah, I copied the config for something else. Only a few have a restart and none use copytruncate. I don't really want to restart z-way-server every time though....

C

[toggledbits]

Have you seen this thread?

https://smarthome.community/topic/112/logrotate-for-z-way-server

[CatmanV2]

@toggledbits said in Logrotate advice:

Have you seen this thread?

https://smarthome.community/topic/112/logrotate-for-z-way-server

I actually contributed to it! my memory really is going.

I'll see how the truncate works

C

[CatmanV2]

Worked perfectly

C