[jira] [Commented] (LUCENE-6829) OfflineSorter should use Directory API

["Robert Muir (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/LUCENE-6829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14949598#comment-14949598 ] 

Robert Muir commented on LUCENE-6829:
-------------------------------------

{quote}
Second, this is an API break for things that secretly used
OfflineSorter (suggesters, Hunspell Dictionary, etc.), but I think
this is good because it makes it clear to the caller that disk space
is going to be used by this method and lets the call control where, vs
running the risk of e.g. filling up your tmp partition.

Some things still wanted access to "the default temp dir", so I moved
it from OfflineSorter to IOUtils, and kept the test infra initializing
that to the mock fs wrapped version. Hmm I think maybe it's just
Hunspell, maybe I can remove that from IOUtils and just put it
(privately) in Hunspell. I think ideally nothing in Lucene should be
secretly using /tmp anymore.
{quote}

Yeah, +1 for fixing that, I think the break is ok. It was a trap before.

{quote}
I also think somehow we should extend the "retry the file delete" that
IW/IFD provides "down" to things like OfflineSorter
{quote}

Its more than that right? (with the current patch) Its also inflateGens too? :)

Looks like something went horribly wrong with the formatting of TestRandomChains?

{quote}
I think to fix this more safely we need to add a new required
(abstract) method to Directory:
{quote}

So would FSDirectory implement this with the obvious Files.createTempFile() ? That would be my vote. I guess some could argue its no longer technically write-once, since methods like that rely on the atomicity of creating a (zero-byte) file, but we are just going to append to that file after, once. And its a separate method, solely as a temporary file facility, so I think this is how it should be implemented? Deserves some thought I guess either way.


> OfflineSorter should use Directory API
> --------------------------------------
>
>                 Key: LUCENE-6829
>                 URL: https://issues.apache.org/jira/browse/LUCENE-6829
>             Project: Lucene - Core
>          Issue Type: Improvement
>            Reporter: Michael McCandless
>            Assignee: Michael McCandless
>             Fix For: Trunk, 5.4
>
>         Attachments: LUCENE-6829.patch, LUCENE-6829.patch
>
>
> I think this is a blocker for LUCENE-6825, because the block KD-tree makes heavy use of OfflineSorter and we don't want to fill up tmp space ...
> This should be a straightforward cutover, but there are some challenges, e.g. the test was failing because virus checker blocked deleting of files.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org