[jira] [Commented] (DRILL-2730) Use different paths for ExternalSort spill files

["Venki Korukanti (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/DRILL-2730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14493365#comment-14493365 ] 

Venki Korukanti commented on DRILL-2730:
----------------------------------------

Looks good. In utility method, we try to create the same formatted string {{queryId/major_fragment_1/minor_fragment_3/operator_4}} for every spill file. I think it is better to create once in  constructor and use it like {{new Path(new Path(dirs.next(), formattedStringCreatedInCtor), spillNumber)}}. We can get rid of the utility method getFileName().

> Use different paths for ExternalSort spill files
> ------------------------------------------------
>
>                 Key: DRILL-2730
>                 URL: https://issues.apache.org/jira/browse/DRILL-2730
>             Project: Apache Drill
>          Issue Type: Bug
>          Components: Execution - Relational Operators
>            Reporter: Steven Phillips
>            Assignee: Venki Korukanti
>             Fix For: 0.9.0
>
>         Attachments: DRILL-2730.patch
>
>
> Currently, all of the spill files for a drillbit end up in the same directory (or directories if multiple are configured). There is no separation based on query or fragment. This makes it very difficult to explore the spill files via the filesystem.
> We should create a directory structure that will organize the files more effectively, create a directory tree with query id, major fragment id, minor fragment id, and operator id.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)