You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@logging.apache.org by Gary Gregory <ga...@gmail.com> on 2021/12/18 15:07:31 UTC
Zero-copy rolling files
Hi All:
And now for something completely different.
I wonder why we do not do file rollovers like below, and if we should:
- Create the file with the target rolled over a name like applog-2021.txt
- Create a symlink for the constant name like applog.txt to point to
applog-2021.txt
- When it's rollover time, start writing to the new file
applog-2022.txt and change the symlink to point to it.
Zero copy.
Thoughts?
Gary
Re: Zero-copy rolling files
Posted by Tim Perry <ti...@gmail.com>.
Junctions are nice, but I think they are limited to pointing to directories on local file systems. Symlinks can point to remote files and directories on local or remote file systems (including using UNC paths).
I didn’t bring up the windows permissions issue with symlinks because I think it is a stopper, I brought it up because we’ll need to include information about the minimal permissions needed to use the feature. It’s normal to request fine grained permissions for an application in a windows environment; I don’t think this would give anyone pause.
Tim
> On Dec 19, 2021, at 2:16 PM, Matt Sicker <bo...@gmail.com> wrote:
> I think the NIO API for symlinks on Windows correspond to the ones that require admin permissions to create. There are also file junctions that are a feature of NTFS, though I’m not sure if there’s any way to create them besides invoking cmd and running a mklink command from there (which, as it might sound, requires a few layers of encoding to properly invoke). For more info, take a look at https://github.com/jenkinsci/jenkins/blob/master/core/src/main/java/hudson/os/WindowsUtil.java <https://github.com/jenkinsci/jenkins/blob/master/core/src/main/java/hudson/os/WindowsUtil.java> where I first encountered this filesystem madness due to an old security vulnerability we fixed in the Jenkins project years ago and ended up writing most of the code I’m talking about.
> --
> Matt Sicker
>
>> On Dec 19, 2021, at 08:33, Gary Gregory <ga...@gmail.com> wrote:
>>
>>> On Sun, Dec 19, 2021 at 9:03 AM Jochen Wiedmann
>>> <jo...@gmail.com> wrote:
>>> Having worked with symbolic links on Windows a lot, I find that
>>> privileges are present, in most cases. However, there is the technical
>>> question "How do I create them?"
>>
>> java.nio.file.Files.createSymbolicLink(Path, Path, FileAttribute<?>...)
>>
>> The API is documented as an optional operation so we might need a set
>> of OS-specific calls to Runtime.exec(String).
>>
>> Gary
>>
>>> The best solution, that I have found so far is letting "cmd" do the
>>> job for me. (The mklink command is not a separate executable, but
>>> build into cmd.)
>>> https://github.com/jochenw/afw/blob/master/afw-core/src/main/java/com/github/jochenw/afw/core/components/WindowsCmdSymbolicLinksHandler.java
>>> Jochen
>>> On Sat, Dec 18, 2021 at 7:43 PM Tim Perry <ti...@gmail.com> wrote:
>>>> I like this idea, but I think it would require non-default permissions for the account the application runs under on windows. However, it could be feature that can be switched on.
>>>> https://docs.microsoft.com/en-us/windows/security/threat-protection/security-policy-settings/create-symbolic-links
>>>> Maybe I read the docs from MS incorrectly.
>>>> Tim
>>>>> On Dec 18, 2021, at 7:07 AM, Gary Gregory <ga...@gmail.com> wrote:
>>>>> Hi All:
>>>>> And now for something completely different.
>>>>> I wonder why we do not do file rollovers like below, and if we should:
>>>>> - Create the file with the target rolled over a name like applog-2021.txt
>>>>> - Create a symlink for the constant name like applog.txt to point to
>>>>> applog-2021.txt
>>>>> - When it's rollover time, start writing to the new file
>>>>> applog-2022.txt and change the symlink to point to it.
>>>>> Zero copy.
>>>>> Thoughts?
>>>>> Gary
>>> --
>>> Philosophy is useless, theology is worse. (Industrial Desease, Dire Straits)
Re: Zero-copy rolling files
Posted by Matt Sicker <bo...@gmail.com>.
I think the NIO API for symlinks on Windows correspond to the ones that require admin permissions to create. There are also file junctions that are a feature of NTFS, though I’m not sure if there’s any way to create them besides invoking cmd and running a mklink command from there (which, as it might sound, requires a few layers of encoding to properly invoke). For more info, take a look at https://github.com/jenkinsci/jenkins/blob/master/core/src/main/java/hudson/os/WindowsUtil.java <https://github.com/jenkinsci/jenkins/blob/master/core/src/main/java/hudson/os/WindowsUtil.java> where I first encountered this filesystem madness due to an old security vulnerability we fixed in the Jenkins project years ago and ended up writing most of the code I’m talking about.
--
Matt Sicker
> On Dec 19, 2021, at 08:33, Gary Gregory <ga...@gmail.com> wrote:
>
> On Sun, Dec 19, 2021 at 9:03 AM Jochen Wiedmann
> <jo...@gmail.com> wrote:
>>
>> Having worked with symbolic links on Windows a lot, I find that
>> privileges are present, in most cases. However, there is the technical
>> question "How do I create them?"
>
> java.nio.file.Files.createSymbolicLink(Path, Path, FileAttribute<?>...)
>
> The API is documented as an optional operation so we might need a set
> of OS-specific calls to Runtime.exec(String).
>
> Gary
>
>>
>> The best solution, that I have found so far is letting "cmd" do the
>> job for me. (The mklink command is not a separate executable, but
>> build into cmd.)
>>
>> https://github.com/jochenw/afw/blob/master/afw-core/src/main/java/com/github/jochenw/afw/core/components/WindowsCmdSymbolicLinksHandler.java
>>
>> Jochen
>>
>>
>> On Sat, Dec 18, 2021 at 7:43 PM Tim Perry <ti...@gmail.com> wrote:
>>>
>>> I like this idea, but I think it would require non-default permissions for the account the application runs under on windows. However, it could be feature that can be switched on.
>>>
>>> https://docs.microsoft.com/en-us/windows/security/threat-protection/security-policy-settings/create-symbolic-links
>>>
>>> Maybe I read the docs from MS incorrectly.
>>>
>>> Tim
>>>
>>>
>>>> On Dec 18, 2021, at 7:07 AM, Gary Gregory <ga...@gmail.com> wrote:
>>>>
>>>> Hi All:
>>>>
>>>> And now for something completely different.
>>>>
>>>> I wonder why we do not do file rollovers like below, and if we should:
>>>> - Create the file with the target rolled over a name like applog-2021.txt
>>>> - Create a symlink for the constant name like applog.txt to point to
>>>> applog-2021.txt
>>>> - When it's rollover time, start writing to the new file
>>>> applog-2022.txt and change the symlink to point to it.
>>>>
>>>> Zero copy.
>>>>
>>>> Thoughts?
>>>>
>>>> Gary
>>
>>
>>
>> --
>> Philosophy is useless, theology is worse. (Industrial Desease, Dire Straits)
Re: Zero-copy rolling files
Posted by Ralph Goers <ra...@dslextreme.com>.
I think this would need to be a different rollover strategy. I am guessing it would be another implementation of DirectFileRolloverStrategy.
Ralph
> On Dec 19, 2021, at 7:33 AM, Gary Gregory <ga...@gmail.com> wrote:
>
> On Sun, Dec 19, 2021 at 9:03 AM Jochen Wiedmann
> <jo...@gmail.com> wrote:
>>
>> Having worked with symbolic links on Windows a lot, I find that
>> privileges are present, in most cases. However, there is the technical
>> question "How do I create them?"
>
> java.nio.file.Files.createSymbolicLink(Path, Path, FileAttribute<?>...)
>
> The API is documented as an optional operation so we might need a set
> of OS-specific calls to Runtime.exec(String).
>
> Gary
>
>>
>> The best solution, that I have found so far is letting "cmd" do the
>> job for me. (The mklink command is not a separate executable, but
>> build into cmd.)
>>
>> https://github.com/jochenw/afw/blob/master/afw-core/src/main/java/com/github/jochenw/afw/core/components/WindowsCmdSymbolicLinksHandler.java
>>
>> Jochen
>>
>>
>> On Sat, Dec 18, 2021 at 7:43 PM Tim Perry <ti...@gmail.com> wrote:
>>>
>>> I like this idea, but I think it would require non-default permissions for the account the application runs under on windows. However, it could be feature that can be switched on.
>>>
>>> https://docs.microsoft.com/en-us/windows/security/threat-protection/security-policy-settings/create-symbolic-links
>>>
>>> Maybe I read the docs from MS incorrectly.
>>>
>>> Tim
>>>
>>>
>>>> On Dec 18, 2021, at 7:07 AM, Gary Gregory <ga...@gmail.com> wrote:
>>>>
>>>> Hi All:
>>>>
>>>> And now for something completely different.
>>>>
>>>> I wonder why we do not do file rollovers like below, and if we should:
>>>> - Create the file with the target rolled over a name like applog-2021.txt
>>>> - Create a symlink for the constant name like applog.txt to point to
>>>> applog-2021.txt
>>>> - When it's rollover time, start writing to the new file
>>>> applog-2022.txt and change the symlink to point to it.
>>>>
>>>> Zero copy.
>>>>
>>>> Thoughts?
>>>>
>>>> Gary
>>
>>
>>
>> --
>> Philosophy is useless, theology is worse. (Industrial Desease, Dire Straits)
>
Re: Zero-copy rolling files
Posted by Gary Gregory <ga...@gmail.com>.
On Sun, Dec 19, 2021 at 9:03 AM Jochen Wiedmann
<jo...@gmail.com> wrote:
>
> Having worked with symbolic links on Windows a lot, I find that
> privileges are present, in most cases. However, there is the technical
> question "How do I create them?"
java.nio.file.Files.createSymbolicLink(Path, Path, FileAttribute<?>...)
The API is documented as an optional operation so we might need a set
of OS-specific calls to Runtime.exec(String).
Gary
>
> The best solution, that I have found so far is letting "cmd" do the
> job for me. (The mklink command is not a separate executable, but
> build into cmd.)
>
> https://github.com/jochenw/afw/blob/master/afw-core/src/main/java/com/github/jochenw/afw/core/components/WindowsCmdSymbolicLinksHandler.java
>
> Jochen
>
>
> On Sat, Dec 18, 2021 at 7:43 PM Tim Perry <ti...@gmail.com> wrote:
> >
> > I like this idea, but I think it would require non-default permissions for the account the application runs under on windows. However, it could be feature that can be switched on.
> >
> > https://docs.microsoft.com/en-us/windows/security/threat-protection/security-policy-settings/create-symbolic-links
> >
> > Maybe I read the docs from MS incorrectly.
> >
> > Tim
> >
> >
> > > On Dec 18, 2021, at 7:07 AM, Gary Gregory <ga...@gmail.com> wrote:
> > >
> > > Hi All:
> > >
> > > And now for something completely different.
> > >
> > > I wonder why we do not do file rollovers like below, and if we should:
> > > - Create the file with the target rolled over a name like applog-2021.txt
> > > - Create a symlink for the constant name like applog.txt to point to
> > > applog-2021.txt
> > > - When it's rollover time, start writing to the new file
> > > applog-2022.txt and change the symlink to point to it.
> > >
> > > Zero copy.
> > >
> > > Thoughts?
> > >
> > > Gary
>
>
>
> --
> Philosophy is useless, theology is worse. (Industrial Desease, Dire Straits)
Re: Zero-copy rolling files
Posted by Jochen Wiedmann <jo...@gmail.com>.
Having worked with symbolic links on Windows a lot, I find that
privileges are present, in most cases. However, there is the technical
question "How do I create them?"
The best solution, that I have found so far is letting "cmd" do the
job for me. (The mklink command is not a separate executable, but
build into cmd.)
https://github.com/jochenw/afw/blob/master/afw-core/src/main/java/com/github/jochenw/afw/core/components/WindowsCmdSymbolicLinksHandler.java
Jochen
On Sat, Dec 18, 2021 at 7:43 PM Tim Perry <ti...@gmail.com> wrote:
>
> I like this idea, but I think it would require non-default permissions for the account the application runs under on windows. However, it could be feature that can be switched on.
>
> https://docs.microsoft.com/en-us/windows/security/threat-protection/security-policy-settings/create-symbolic-links
>
> Maybe I read the docs from MS incorrectly.
>
> Tim
>
>
> > On Dec 18, 2021, at 7:07 AM, Gary Gregory <ga...@gmail.com> wrote:
> >
> > Hi All:
> >
> > And now for something completely different.
> >
> > I wonder why we do not do file rollovers like below, and if we should:
> > - Create the file with the target rolled over a name like applog-2021.txt
> > - Create a symlink for the constant name like applog.txt to point to
> > applog-2021.txt
> > - When it's rollover time, start writing to the new file
> > applog-2022.txt and change the symlink to point to it.
> >
> > Zero copy.
> >
> > Thoughts?
> >
> > Gary
--
Philosophy is useless, theology is worse. (Industrial Desease, Dire Straits)
Re: Zero-copy rolling files
Posted by Tim Perry <ti...@gmail.com>.
I like this idea, but I think it would require non-default permissions for the account the application runs under on windows. However, it could be feature that can be switched on.
https://docs.microsoft.com/en-us/windows/security/threat-protection/security-policy-settings/create-symbolic-links
Maybe I read the docs from MS incorrectly.
Tim
> On Dec 18, 2021, at 7:07 AM, Gary Gregory <ga...@gmail.com> wrote:
>
> Hi All:
>
> And now for something completely different.
>
> I wonder why we do not do file rollovers like below, and if we should:
> - Create the file with the target rolled over a name like applog-2021.txt
> - Create a symlink for the constant name like applog.txt to point to
> applog-2021.txt
> - When it's rollover time, start writing to the new file
> applog-2022.txt and change the symlink to point to it.
>
> Zero copy.
>
> Thoughts?
>
> Gary
Re: Zero-copy rolling files
Posted by Volkan Yazıcı <vo...@yazi.ci>.
*[I am Log4j rollover illiterate, hence apologies in advance if I am saying
something stupid.]*
Why don't we simply rename files and create new ones?
That is, `mv applog.txt applog-2021.txt` and `touch applog.txt`?
I use this in my RotatingFileOutputStream
<https://github.com/vy/rotating-fos> and there are plenty of happy
customers, even on Windows.
On Sat, Dec 18, 2021 at 4:07 PM Gary Gregory <ga...@gmail.com> wrote:
> Hi All:
>
> And now for something completely different.
>
> I wonder why we do not do file rollovers like below, and if we should:
> - Create the file with the target rolled over a name like applog-2021.txt
> - Create a symlink for the constant name like applog.txt to point to
> applog-2021.txt
> - When it's rollover time, start writing to the new file
> applog-2022.txt and change the symlink to point to it.
>
> Zero copy.
>
> Thoughts?
>
> Gary
>