You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Dawid Weiss <da...@gmail.com> on 2014/05/22 08:14:15 UTC
Read only access to the filesystem outside temp. dirs enforced in tests.
This is just a heads up that I will be committing in LUCENE-5650 that
Ryan and I have been working on for some time now. This patch adds
security manager rules that effectively prevent the JVM from writing
anything outside its temporary folder (including the CWD!).
While it may sound like an insane restriction, we have found many
places in the code where writes were happening to source code
locations or were not needed at all (a separate appender creating
velocity.log, for example). If you need a writeable location, call
createTempDir() or createTempFile(); it should be available to any
test that inherits from LuceneTestCase. There are other benefits of
doing this -- temporary files will be cleaned up automatically after
the test completes successfully, the path of each temporary file
contains the test's name and seed to map files back to what created
them, etc.
In general, I think the benefits outweigh the (subjectively perceived
:) additional burden of calling createTempDir().
Dawid
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
RE: Read only access to the filesystem outside temp. dirs enforced in tests.
Posted by Uwe Schindler <uw...@thetaphi.de>.
> Policeman Uwe at his lookout post! :) Thanks Uwe.
>
> > We should maybe sometimes run the tests without SecurityManager (like
> > Elasticsearch does) and without asserts enabled (no "-ea")
>
> You'd have to modify the test rule that currently requires assertions to be
> enabled, but otherwise I think it's doable.
Yes, the idea was to never do this while normal test runs. The idea is to have some special "Jenkins mode" that’s allowed to be configured from the outside like "ant test -Dtests.productionMode=true" to enable "production mode", so this disabled asserts and SecurityManager. Of course it that case the test rule would just ignore the missing assertions on this sysprop. We could then run the Lucene Core tests in this special Jenkins mode.
In fact, if this mode is enabled, ant should also print a warning that you are running in some "unsafe" test environment.
Uwe
> On Thu, May 22, 2014 at 8:58 AM, Uwe Schindler <uw...@thetaphi.de> wrote:
> > Hi Dawid,
> >
> > thanks for more cleanups!
> >
> > In fact, writing to source folder was not allowed before, too. I reviewed the
> patches: Lots of tests in Solr were indeed not properly creating temporary
> Solr instances, but as those tests never did anything on the fake solr folder,
> the SecurityException was just swallowed in the logs. The other failures in
> DIH were not writing to source folder, those tests were simply writing to
> CWD without requesting a temp dir. This was previously allowed, but is now
> it's disallowed, so the tests had to be fixed.
> >
> > In any case, we should fix the remaining Solr tests that swallow
> SecurityExceptions... The reason for this is: We should maybe sometimes run
> the tests without SecurityManager (like Elasticsearch does) and without
> asserts enabled (no "-ea"), to find hotspot bugs that don't happen, because
> the code is different when running in test framework than in production. But
> this is a thing todo in Lucene Core only (maybe). Without the security
> manager, those tests would create files in source folder...
> >
> > Uwe
> >
> > -----
> > Uwe Schindler
> > H.-H.-Meier-Allee 63, D-28213 Bremen
> > http://www.thetaphi.de
> > eMail: uwe@thetaphi.de
> >
> >
> >> -----Original Message-----
> >> From: Dawid Weiss [mailto:dawid.weiss@gmail.com]
> >> Sent: Thursday, May 22, 2014 8:14 AM
> >> To: dev@lucene.apache.org
> >> Subject: Read only access to the filesystem outside temp. dirs
> >> enforced in tests.
> >>
> >> This is just a heads up that I will be committing in LUCENE-5650 that
> >> Ryan and I have been working on for some time now. This patch adds
> >> security manager rules that effectively prevent the JVM from writing
> >> anything outside its temporary folder (including the CWD!).
> >>
> >> While it may sound like an insane restriction, we have found many
> >> places in the code where writes were happening to source code
> >> locations or were not needed at all (a separate appender creating
> >> velocity.log, for example). If you need a writeable location, call
> >> createTempDir() or createTempFile(); it should be available to any
> >> test that inherits from LuceneTestCase. There are other benefits of
> >> doing this -- temporary files will be cleaned up automatically after
> >> the test completes successfully, the path of each temporary file
> >> contains the test's name and seed to map files back to what created
> them, etc.
> >>
> >> In general, I think the benefits outweigh the (subjectively perceived
> >> :) additional burden of calling createTempDir().
> >>
> >> Dawid
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org For
> >> additional commands, e-mail: dev-help@lucene.apache.org
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org For
> > additional commands, e-mail: dev-help@lucene.apache.org
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org For additional
> commands, e-mail: dev-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: Read only access to the filesystem outside temp. dirs enforced in tests.
Posted by Dawid Weiss <da...@cs.put.poznan.pl>.
Policeman Uwe at his lookout post! :) Thanks Uwe.
> We should maybe sometimes run the tests without SecurityManager (like Elasticsearch does) and without asserts enabled (no "-ea")
You'd have to modify the test rule that currently requires assertions
to be enabled, but otherwise I think it's doable.
Dawid
On Thu, May 22, 2014 at 8:58 AM, Uwe Schindler <uw...@thetaphi.de> wrote:
> Hi Dawid,
>
> thanks for more cleanups!
>
> In fact, writing to source folder was not allowed before, too. I reviewed the patches: Lots of tests in Solr were indeed not properly creating temporary Solr instances, but as those tests never did anything on the fake solr folder, the SecurityException was just swallowed in the logs. The other failures in DIH were not writing to source folder, those tests were simply writing to CWD without requesting a temp dir. This was previously allowed, but is now it's disallowed, so the tests had to be fixed.
>
> In any case, we should fix the remaining Solr tests that swallow SecurityExceptions... The reason for this is: We should maybe sometimes run the tests without SecurityManager (like Elasticsearch does) and without asserts enabled (no "-ea"), to find hotspot bugs that don't happen, because the code is different when running in test framework than in production. But this is a thing todo in Lucene Core only (maybe). Without the security manager, those tests would create files in source folder...
>
> Uwe
>
> -----
> Uwe Schindler
> H.-H.-Meier-Allee 63, D-28213 Bremen
> http://www.thetaphi.de
> eMail: uwe@thetaphi.de
>
>
>> -----Original Message-----
>> From: Dawid Weiss [mailto:dawid.weiss@gmail.com]
>> Sent: Thursday, May 22, 2014 8:14 AM
>> To: dev@lucene.apache.org
>> Subject: Read only access to the filesystem outside temp. dirs enforced in
>> tests.
>>
>> This is just a heads up that I will be committing in LUCENE-5650 that Ryan and
>> I have been working on for some time now. This patch adds security manager
>> rules that effectively prevent the JVM from writing anything outside its
>> temporary folder (including the CWD!).
>>
>> While it may sound like an insane restriction, we have found many places in
>> the code where writes were happening to source code locations or were not
>> needed at all (a separate appender creating velocity.log, for example). If you
>> need a writeable location, call
>> createTempDir() or createTempFile(); it should be available to any test that
>> inherits from LuceneTestCase. There are other benefits of doing this --
>> temporary files will be cleaned up automatically after the test completes
>> successfully, the path of each temporary file contains the test's name and
>> seed to map files back to what created them, etc.
>>
>> In general, I think the benefits outweigh the (subjectively perceived
>> :) additional burden of calling createTempDir().
>>
>> Dawid
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org For additional
>> commands, e-mail: dev-help@lucene.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
RE: Read only access to the filesystem outside temp. dirs enforced in tests.
Posted by Uwe Schindler <uw...@thetaphi.de>.
Hi Dawid,
thanks for more cleanups!
In fact, writing to source folder was not allowed before, too. I reviewed the patches: Lots of tests in Solr were indeed not properly creating temporary Solr instances, but as those tests never did anything on the fake solr folder, the SecurityException was just swallowed in the logs. The other failures in DIH were not writing to source folder, those tests were simply writing to CWD without requesting a temp dir. This was previously allowed, but is now it's disallowed, so the tests had to be fixed.
In any case, we should fix the remaining Solr tests that swallow SecurityExceptions... The reason for this is: We should maybe sometimes run the tests without SecurityManager (like Elasticsearch does) and without asserts enabled (no "-ea"), to find hotspot bugs that don't happen, because the code is different when running in test framework than in production. But this is a thing todo in Lucene Core only (maybe). Without the security manager, those tests would create files in source folder...
Uwe
-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe@thetaphi.de
> -----Original Message-----
> From: Dawid Weiss [mailto:dawid.weiss@gmail.com]
> Sent: Thursday, May 22, 2014 8:14 AM
> To: dev@lucene.apache.org
> Subject: Read only access to the filesystem outside temp. dirs enforced in
> tests.
>
> This is just a heads up that I will be committing in LUCENE-5650 that Ryan and
> I have been working on for some time now. This patch adds security manager
> rules that effectively prevent the JVM from writing anything outside its
> temporary folder (including the CWD!).
>
> While it may sound like an insane restriction, we have found many places in
> the code where writes were happening to source code locations or were not
> needed at all (a separate appender creating velocity.log, for example). If you
> need a writeable location, call
> createTempDir() or createTempFile(); it should be available to any test that
> inherits from LuceneTestCase. There are other benefits of doing this --
> temporary files will be cleaned up automatically after the test completes
> successfully, the path of each temporary file contains the test's name and
> seed to map files back to what created them, etc.
>
> In general, I think the benefits outweigh the (subjectively perceived
> :) additional burden of calling createTempDir().
>
> Dawid
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org For additional
> commands, e-mail: dev-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org