You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by David Smiley <da...@gmail.com> on 2017/01/20 02:22:38 UTC
Test passes, build fails with "Could not remove temporary path"
I wrote a test in Solr extending SolrTestCaseJ4 that sets up Solr a bit
differently than some tests. Essentially instead of writing yet another
test schema.xml file, I decided to use some stock configs that are already
in a managed-schema mode that allow me to manipulate the schema via test
code -- but not persist it since I don't want/need that. I could share the
code if it helps. Any way, my test passes when run via IntelliJ; the test
JVM ends normally.
But when I run via "ant test-core -Dtestcase=mytestclassname" the test
itself passes but the build fails. Right before the "5 slowest tests"
printout is the following curious message:
Could not remove temporary path:
/SmileyDev/Search/lucene-solr_6x/solr/build/solr-core/test/J0
(/SmileyDev/Search/lucene-solr_6x/solr/build/solr-core/test/J0)
I can't find the string "Could not remove temporary path" in our codebase;
maybe it's in randomized-testing? (CC Dawid) I'm not sure how to debug
this... maybe Solr wasn't closed properly? Although this doesn't happen
when run via IntelliJ; maybe there's a race.
Here's my @BeforeClass:
@BeforeClass
private static void initManagedSchemaCore() throws Exception {
// This testing approach means no new solrconfig or schema file or
per-test temp solr-home!
System.setProperty("managed.schema.mutable", "true");
System.setProperty("managed.schema.resourceName",
"schema-one-field-no-dynamic-field-unique-key.xml");
System.setProperty("enable.update.log", "false");
initCore("solrconfig-managed-schema.xml", "ignoredSchemaName?");
IndexSchema oldSchema = h.getCore().getLatestSchema();
HeatmapSpatialField fieldType = new HeatmapSpatialField();
Map<String, String> ftInitMap = new HashMap<>();
ftInitMap.put("prefixTree", "packedQuad");
ftInitMap.put("square", "true");
ftInitMap.put("minDistErr", "1000");
ftInitMap.put("maxDistErr", "50");
ftInitMap.put("distanceUnits", "kilometers");
fieldType.init(oldSchema, ftInitMap);
fieldType.setTypeName("heatmapType");
SchemaField schemaField = new SchemaField(HM_FLD, fieldType,
SchemaField.STORED | SchemaField.INDEXED, null);
boolean persist = false; // don't write to test resource dir
IndexSchema newSchema = oldSchema.addField(schemaField, persist);
h.getCore().setLatestSchema(newSchema);
strategy = fieldType.getStrategy(schemaField.getName());
ctx = strategy.getSpatialContext();
}
Or maybe most of that info is a red herring and the problem is something
else. Any way, it'd be nice if someone else who ran into this problem
before could share some advice.
Thanks in advance,
~ David
--
Lucene/Solr Search Committer, Consultant, Developer, Author, Speaker
LinkedIn: http://linkedin.com/in/davidwsmiley | Book:
http://www.solrenterprisesearchserver.com
Re: Test passes, build fails with "Could not remove temporary path"
Posted by David Smiley <da...@gmail.com>.
Check this out: (FYI this is my test; doesn't exist upstream yet):
> ant test -Dtestcase=HeatmapSpatialFieldTest
BUILD SUCCESSFUL, 1 minute 40 seconds
> ant test-core -Dtestcase=HeatmapSpatialFieldTest
BUILD FAILED, 25 seconds
<target name="test" description="Validate, then run core, solrj, and
contrib unit tests."
depends="-init-totals, test-core, test-contrib, -check-totals"/>
<target name="test-core" description="Runs the core and solrj unit tests."
depends="-clover.load, test-solr-core, test-solrj"/>
So it turns out the "-init-totals" task sets up the temp dir along with a
deleteonexit.
Just to verify, I temporarily modified test-core to first invoke
"-init-totals", and it passed! Perhaps all the test-* tasks should
simply have an "-init-totals" and "-check-totals" then?
~ David
On Tue, Jan 24, 2017 at 6:06 AM Dawid Weiss <da...@gmail.com> wrote:
> It's exactly that, actually. We place java.io.tmpdir under ./, so this
> directory always remains after the tests are done. I filed this issue:
>
> https://github.com/randomizedtesting/randomizedtesting/issues/247
>
> But I honestly don't know what the "right" way to fix it is. The
> runner assumes cwd should be left clean -- perhaps this should be a
> switch too (with similar wipe|ignore|warn options, defaulting to warn
> for backcompat).
>
> Note that LuceneTestCase already has leftover file detection facility
> it manages internally anyway (TestRuleTemporaryFilesCleanup).
>
> Dawid
>
> On Mon, Jan 23, 2017 at 7:22 PM, Dawid Weiss <da...@gmail.com>
> wrote:
> > No problem at all. I wonder if we (in Lucene) don't point the temp
> > folder under cwd -- we probably do... If so then this is something I
> > didn't give much thought to... special case which should probably be
> > allowed. Check common-build and confirm if this is the case.
> >
> > Dawid
> >
> > On Mon, Jan 23, 2017 at 3:41 PM, David Smiley <da...@gmail.com>
> wrote:
> >> Thanks very much Dawid. So indeed, the directory in question isn't
> quite
> >> empty; it contains a "temp" directory (that is empty). Off to the next
> >> thing to debug....
> >>
> >> Thanks again.
> >> ~ David
> >>
> >> On Mon, Jan 23, 2017 at 7:40 AM Dawid Weiss <da...@gmail.com>
> wrote:
> >>>
> >>> I've committed LUCENE-7653 which should help you diagnose the problem,
> >>> David. First, it'll clean the cwd of a forked process before the tests
> >>> start (something that wasn't done before). Second, it'll report what
> >>> files remained uncleaned after a run.
> >>>
> >>> Hope it'll help.
> >>>
> >>> Dawid
> >>>
> >>> On Fri, Jan 20, 2017 at 8:57 AM, Dawid Weiss <da...@gmail.com>
> >>> wrote:
> >>> > Hi David!
> >>> >
> >>> >> I can't find the string "Could not remove temporary path" in our
> >>> >> codebase;
> >>> >> maybe it's in randomized-testing? (CC Dawid) I'm not sure how to
> >>> >> debug
> >>> >> this... maybe Solr wasn't closed properly? Although this doesn't
> >>> >> happen
> >>> >
> >>> > Yes, this message has a source in ANT's unit test runner code, here:
> >>> >
> >>> >
> >>> >
> https://github.com/randomizedtesting/randomizedtesting/blob/master/junit4-ant/src/main/java/com/carrotsearch/ant/tasks/junit4/JUnit4.java#L1031-L1041
> >>> >
> >>> > Specifically, it couldn't delete the temporary folder -- most likely
> >>> > it wasn't empty (there were some files inside the folder). I think
> the
> >>> > message here should be improved -- I'll do that -- but in the mean
> >>> > time make sure the test's folder is empty; if it isn't, the build
> will
> >>> > fail.
> >>> >
> >>> > Dawid
> >>
> >> --
> >> Lucene/Solr Search Committer, Consultant, Developer, Author, Speaker
> >> LinkedIn: http://linkedin.com/in/davidwsmiley | Book:
> >> http://www.solrenterprisesearchserver.com
>
--
Lucene/Solr Search Committer, Consultant, Developer, Author, Speaker
LinkedIn: http://linkedin.com/in/davidwsmiley | Book:
http://www.solrenterprisesearchserver.com
Re: Test passes, build fails with "Could not remove temporary path"
Posted by Uwe Schindler <uw...@thetaphi.de>.
Hi,
The problem is also the security manager and it's policy file. This is why we have the temp dir in current folder. As there is no easy way to have an absolute path in the policy file, we have it like this. Another way would be to pass a sys prop for the temp directory and add a policy file rule.
Uwe
Am 24. Januar 2017 12:06:12 MEZ schrieb Dawid Weiss <da...@gmail.com>:
>It's exactly that, actually. We place java.io.tmpdir under ./, so this
>directory always remains after the tests are done. I filed this issue:
>
>https://github.com/randomizedtesting/randomizedtesting/issues/247
>
>But I honestly don't know what the "right" way to fix it is. The
>runner assumes cwd should be left clean -- perhaps this should be a
>switch too (with similar wipe|ignore|warn options, defaulting to warn
>for backcompat).
>
>Note that LuceneTestCase already has leftover file detection facility
>it manages internally anyway (TestRuleTemporaryFilesCleanup).
>
>Dawid
>
>On Mon, Jan 23, 2017 at 7:22 PM, Dawid Weiss <da...@gmail.com>
>wrote:
>> No problem at all. I wonder if we (in Lucene) don't point the temp
>> folder under cwd -- we probably do... If so then this is something I
>> didn't give much thought to... special case which should probably be
>> allowed. Check common-build and confirm if this is the case.
>>
>> Dawid
>>
>> On Mon, Jan 23, 2017 at 3:41 PM, David Smiley
><da...@gmail.com> wrote:
>>> Thanks very much Dawid. So indeed, the directory in question isn't
>quite
>>> empty; it contains a "temp" directory (that is empty). Off to the
>next
>>> thing to debug....
>>>
>>> Thanks again.
>>> ~ David
>>>
>>> On Mon, Jan 23, 2017 at 7:40 AM Dawid Weiss <da...@gmail.com>
>wrote:
>>>>
>>>> I've committed LUCENE-7653 which should help you diagnose the
>problem,
>>>> David. First, it'll clean the cwd of a forked process before the
>tests
>>>> start (something that wasn't done before). Second, it'll report
>what
>>>> files remained uncleaned after a run.
>>>>
>>>> Hope it'll help.
>>>>
>>>> Dawid
>>>>
>>>> On Fri, Jan 20, 2017 at 8:57 AM, Dawid Weiss
><da...@gmail.com>
>>>> wrote:
>>>> > Hi David!
>>>> >
>>>> >> I can't find the string "Could not remove temporary path" in our
>>>> >> codebase;
>>>> >> maybe it's in randomized-testing? (CC Dawid) I'm not sure how
>to
>>>> >> debug
>>>> >> this... maybe Solr wasn't closed properly? Although this
>doesn't
>>>> >> happen
>>>> >
>>>> > Yes, this message has a source in ANT's unit test runner code,
>here:
>>>> >
>>>> >
>>>> >
>https://github.com/randomizedtesting/randomizedtesting/blob/master/junit4-ant/src/main/java/com/carrotsearch/ant/tasks/junit4/JUnit4.java#L1031-L1041
>>>> >
>>>> > Specifically, it couldn't delete the temporary folder -- most
>likely
>>>> > it wasn't empty (there were some files inside the folder). I
>think the
>>>> > message here should be improved -- I'll do that -- but in the
>mean
>>>> > time make sure the test's folder is empty; if it isn't, the build
>will
>>>> > fail.
>>>> >
>>>> > Dawid
>>>
>>> --
>>> Lucene/Solr Search Committer, Consultant, Developer, Author, Speaker
>>> LinkedIn: http://linkedin.com/in/davidwsmiley | Book:
>>> http://www.solrenterprisesearchserver.com
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>For additional commands, e-mail: dev-help@lucene.apache.org
--
Uwe Schindler
Achterdiek 19, 28357 Bremen
https://www.thetaphi.de
Re: Test passes, build fails with "Could not remove temporary path"
Posted by Dawid Weiss <da...@gmail.com>.
It's exactly that, actually. We place java.io.tmpdir under ./, so this
directory always remains after the tests are done. I filed this issue:
https://github.com/randomizedtesting/randomizedtesting/issues/247
But I honestly don't know what the "right" way to fix it is. The
runner assumes cwd should be left clean -- perhaps this should be a
switch too (with similar wipe|ignore|warn options, defaulting to warn
for backcompat).
Note that LuceneTestCase already has leftover file detection facility
it manages internally anyway (TestRuleTemporaryFilesCleanup).
Dawid
On Mon, Jan 23, 2017 at 7:22 PM, Dawid Weiss <da...@gmail.com> wrote:
> No problem at all. I wonder if we (in Lucene) don't point the temp
> folder under cwd -- we probably do... If so then this is something I
> didn't give much thought to... special case which should probably be
> allowed. Check common-build and confirm if this is the case.
>
> Dawid
>
> On Mon, Jan 23, 2017 at 3:41 PM, David Smiley <da...@gmail.com> wrote:
>> Thanks very much Dawid. So indeed, the directory in question isn't quite
>> empty; it contains a "temp" directory (that is empty). Off to the next
>> thing to debug....
>>
>> Thanks again.
>> ~ David
>>
>> On Mon, Jan 23, 2017 at 7:40 AM Dawid Weiss <da...@gmail.com> wrote:
>>>
>>> I've committed LUCENE-7653 which should help you diagnose the problem,
>>> David. First, it'll clean the cwd of a forked process before the tests
>>> start (something that wasn't done before). Second, it'll report what
>>> files remained uncleaned after a run.
>>>
>>> Hope it'll help.
>>>
>>> Dawid
>>>
>>> On Fri, Jan 20, 2017 at 8:57 AM, Dawid Weiss <da...@gmail.com>
>>> wrote:
>>> > Hi David!
>>> >
>>> >> I can't find the string "Could not remove temporary path" in our
>>> >> codebase;
>>> >> maybe it's in randomized-testing? (CC Dawid) I'm not sure how to
>>> >> debug
>>> >> this... maybe Solr wasn't closed properly? Although this doesn't
>>> >> happen
>>> >
>>> > Yes, this message has a source in ANT's unit test runner code, here:
>>> >
>>> >
>>> > https://github.com/randomizedtesting/randomizedtesting/blob/master/junit4-ant/src/main/java/com/carrotsearch/ant/tasks/junit4/JUnit4.java#L1031-L1041
>>> >
>>> > Specifically, it couldn't delete the temporary folder -- most likely
>>> > it wasn't empty (there were some files inside the folder). I think the
>>> > message here should be improved -- I'll do that -- but in the mean
>>> > time make sure the test's folder is empty; if it isn't, the build will
>>> > fail.
>>> >
>>> > Dawid
>>
>> --
>> Lucene/Solr Search Committer, Consultant, Developer, Author, Speaker
>> LinkedIn: http://linkedin.com/in/davidwsmiley | Book:
>> http://www.solrenterprisesearchserver.com
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: Test passes, build fails with "Could not remove temporary path"
Posted by Dawid Weiss <da...@gmail.com>.
No problem at all. I wonder if we (in Lucene) don't point the temp
folder under cwd -- we probably do... If so then this is something I
didn't give much thought to... special case which should probably be
allowed. Check common-build and confirm if this is the case.
Dawid
On Mon, Jan 23, 2017 at 3:41 PM, David Smiley <da...@gmail.com> wrote:
> Thanks very much Dawid. So indeed, the directory in question isn't quite
> empty; it contains a "temp" directory (that is empty). Off to the next
> thing to debug....
>
> Thanks again.
> ~ David
>
> On Mon, Jan 23, 2017 at 7:40 AM Dawid Weiss <da...@gmail.com> wrote:
>>
>> I've committed LUCENE-7653 which should help you diagnose the problem,
>> David. First, it'll clean the cwd of a forked process before the tests
>> start (something that wasn't done before). Second, it'll report what
>> files remained uncleaned after a run.
>>
>> Hope it'll help.
>>
>> Dawid
>>
>> On Fri, Jan 20, 2017 at 8:57 AM, Dawid Weiss <da...@gmail.com>
>> wrote:
>> > Hi David!
>> >
>> >> I can't find the string "Could not remove temporary path" in our
>> >> codebase;
>> >> maybe it's in randomized-testing? (CC Dawid) I'm not sure how to
>> >> debug
>> >> this... maybe Solr wasn't closed properly? Although this doesn't
>> >> happen
>> >
>> > Yes, this message has a source in ANT's unit test runner code, here:
>> >
>> >
>> > https://github.com/randomizedtesting/randomizedtesting/blob/master/junit4-ant/src/main/java/com/carrotsearch/ant/tasks/junit4/JUnit4.java#L1031-L1041
>> >
>> > Specifically, it couldn't delete the temporary folder -- most likely
>> > it wasn't empty (there were some files inside the folder). I think the
>> > message here should be improved -- I'll do that -- but in the mean
>> > time make sure the test's folder is empty; if it isn't, the build will
>> > fail.
>> >
>> > Dawid
>
> --
> Lucene/Solr Search Committer, Consultant, Developer, Author, Speaker
> LinkedIn: http://linkedin.com/in/davidwsmiley | Book:
> http://www.solrenterprisesearchserver.com
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: Test passes, build fails with "Could not remove temporary path"
Posted by David Smiley <da...@gmail.com>.
Thanks very much Dawid. So indeed, the directory in question isn't quite
empty; it contains a "temp" directory (that is empty). Off to the next
thing to debug....
Thanks again.
~ David
On Mon, Jan 23, 2017 at 7:40 AM Dawid Weiss <da...@gmail.com> wrote:
> I've committed LUCENE-7653 which should help you diagnose the problem,
> David. First, it'll clean the cwd of a forked process before the tests
> start (something that wasn't done before). Second, it'll report what
> files remained uncleaned after a run.
>
> Hope it'll help.
>
> Dawid
>
> On Fri, Jan 20, 2017 at 8:57 AM, Dawid Weiss <da...@gmail.com>
> wrote:
> > Hi David!
> >
> >> I can't find the string "Could not remove temporary path" in our
> codebase;
> >> maybe it's in randomized-testing? (CC Dawid) I'm not sure how to debug
> >> this... maybe Solr wasn't closed properly? Although this doesn't happen
> >
> > Yes, this message has a source in ANT's unit test runner code, here:
> >
> >
> https://github.com/randomizedtesting/randomizedtesting/blob/master/junit4-ant/src/main/java/com/carrotsearch/ant/tasks/junit4/JUnit4.java#L1031-L1041
> >
> > Specifically, it couldn't delete the temporary folder -- most likely
> > it wasn't empty (there were some files inside the folder). I think the
> > message here should be improved -- I'll do that -- but in the mean
> > time make sure the test's folder is empty; if it isn't, the build will
> > fail.
> >
> > Dawid
>
--
Lucene/Solr Search Committer, Consultant, Developer, Author, Speaker
LinkedIn: http://linkedin.com/in/davidwsmiley | Book:
http://www.solrenterprisesearchserver.com
Re: Test passes, build fails with "Could not remove temporary path"
Posted by Dawid Weiss <da...@gmail.com>.
I've committed LUCENE-7653 which should help you diagnose the problem,
David. First, it'll clean the cwd of a forked process before the tests
start (something that wasn't done before). Second, it'll report what
files remained uncleaned after a run.
Hope it'll help.
Dawid
On Fri, Jan 20, 2017 at 8:57 AM, Dawid Weiss <da...@gmail.com> wrote:
> Hi David!
>
>> I can't find the string "Could not remove temporary path" in our codebase;
>> maybe it's in randomized-testing? (CC Dawid) I'm not sure how to debug
>> this... maybe Solr wasn't closed properly? Although this doesn't happen
>
> Yes, this message has a source in ANT's unit test runner code, here:
>
> https://github.com/randomizedtesting/randomizedtesting/blob/master/junit4-ant/src/main/java/com/carrotsearch/ant/tasks/junit4/JUnit4.java#L1031-L1041
>
> Specifically, it couldn't delete the temporary folder -- most likely
> it wasn't empty (there were some files inside the folder). I think the
> message here should be improved -- I'll do that -- but in the mean
> time make sure the test's folder is empty; if it isn't, the build will
> fail.
>
> Dawid
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: Test passes, build fails with "Could not remove temporary path"
Posted by Dawid Weiss <da...@gmail.com>.
Hi David!
> I can't find the string "Could not remove temporary path" in our codebase;
> maybe it's in randomized-testing? (CC Dawid) I'm not sure how to debug
> this... maybe Solr wasn't closed properly? Although this doesn't happen
Yes, this message has a source in ANT's unit test runner code, here:
https://github.com/randomizedtesting/randomizedtesting/blob/master/junit4-ant/src/main/java/com/carrotsearch/ant/tasks/junit4/JUnit4.java#L1031-L1041
Specifically, it couldn't delete the temporary folder -- most likely
it wasn't empty (there were some files inside the folder). I think the
message here should be improved -- I'll do that -- but in the mean
time make sure the test's folder is empty; if it isn't, the build will
fail.
Dawid
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org