[jira] [Issue Comment Deleted] (DRILL-5274) Exception thrown in Drillbit shutdown in UDF cleanup code

["Paul Rogers (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/DRILL-5274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Paul Rogers updated DRILL-5274:
-------------------------------
    Comment: was deleted

(was: Example case. Query:

{code}
select * from (select * from dfs.`/drill/testdata/250wide.tbl`
    order by columns[0])d where d.columns[0] = 'ljdfhwuehnoiueyf';
{code}

The data file is a large file consisting of a single column 250 characters wide.

Batch as seen by the external sort:

{code}
ExternalSortBatch - Actual batch schema & sizes {
  T0¦¦columns(std col. size: 54, actual col. size: 0, total size: 263942, vector size: 0, data size: 0, row capacity: 1023, density: 0)
  EXPR$1(std col. size: 54, actual col. size: 250, total size: 282624, vector size: 278528, data size: 255750, row capacity: 4095, density: 92)
  Records: 1023, Total size: 548614, Row width:538, Density:92}
{code}

The result is that the sort must buffer two copies of the data. Since this is an 18 GB input file, the sort must buffer (then spill) 36 GB of data. Note that there is actually no column alias or expression on the column.)

> Exception thrown in Drillbit shutdown in UDF cleanup code
> ---------------------------------------------------------
>
>                 Key: DRILL-5274
>                 URL: https://issues.apache.org/jira/browse/DRILL-5274
>             Project: Apache Drill
>          Issue Type: Bug
>    Affects Versions: 1.10
>            Reporter: Paul Rogers
>            Assignee: Arina Ielchiieva
>            Priority: Minor
>
> I ran a very simple query: a single-line text file in an embedded Drillbit. The UDF directory was placed in /tmp. During the run, the directory was deleted. On Drillbit shutdown, the following occurred:
> {code}
> 25328 DEBUG [main] [org.apache.drill.exec.server.Drillbit] - Shutdown begun.
> 26344 INFO [pool-1-thread-2] [org.apache.drill.exec.rpc.data.DataServer] - closed eventLoopGroup io.netty.channel.nio.NioEventLoopGroup@7d1c0d85 in 1007 ms
> 26345 INFO [pool-1-thread-1] [org.apache.drill.exec.rpc.user.UserServer] - closed eventLoopGroup io.netty.channel.nio.NioEventLoopGroup@7cdb3b56 in 1008 ms
> 26345 INFO [pool-1-thread-1] [org.apache.drill.exec.service.ServiceEngine] - closed userServer in 1009 ms
> 26345 INFO [pool-1-thread-2] [org.apache.drill.exec.service.ServiceEngine] - closed dataPool in 1009 ms
> 26356 WARN [main] [org.apache.drill.exec.server.Drillbit] - Failure on close()
> java.lang.IllegalArgumentException: /tmp/drill/udf/udf/local does not exist
> 	at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:1637) ~[commons-io-2.4.jar:2.4]
> 	at org.apache.drill.exec.expr.fn.FunctionImplementationRegistry.close(FunctionImplementationRegistry.java:469) ~[classes/:na]
> 	at org.apache.drill.exec.server.DrillbitContext.close(DrillbitContext.java:209) ~[classes/:na]
> 	at org.apache.drill.exec.work.WorkManager.close(WorkManager.java:152) ~[classes/:na]
> 	at org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:76) ~[classes/:na]
> 	at org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:64) ~[classes/:na]
> 	at org.apache.drill.exec.server.Drillbit.close(Drillbit.java:171) ~[classes/:na]
> ...
> {code}
> The following patch makes the problem go away, but I'm not sure if the above is an indication of deeper problems.
> {code}
> public class FunctionImplementationRegistry implements FunctionLookupContext, AutoCloseable {
>   ...
>   public void close() {
>     if (deleteTmpDir) {
>       ...
>     } else {
>       try {
>         File dir = new File(localUdfDir.toUri().getPath());
>         if (dir.exists()) {
>           FileUtils.cleanDirectory(dir);
>         }
>       ...
>     }
>   }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)