[jira] [Commented] (DRILL-5255) Drill requires that dfs.tmp be set up a start time, even if CTTAS not used

["Paul Rogers (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/DRILL-5255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15862268#comment-15862268 ] 

Paul Rogers commented on DRILL-5255:
------------------------------------

The situation is a bit more subtle than the description suggests. The standard {{bootstrap-storage-plugins.json}} does contain a definition for {{dfs.tmp}}.

Tracking through a bit more, it turns out that my machine has created a local storage for plugins at: {{/tmp/drill/sys.storage_plugins/dfs.sys.drill}}. In that, the tmp directory is defined as:

{code}
    "tmp" : {
      "location" : "/tmp",
      "writable" : false,
      "defaultInputFormat" : null
    },
{code}

This seems to be how things get persisted when the following standard test setting is configured in the bootstrap options:

{code}
      put(ExecConstants.SYS_STORE_PROVIDER_LOCAL_ENABLE_WRITE, false);
{code}

Somehow the {{dfs.tmp}} has got to be marked writable (the default configuration) even when the system store local write is not enabled.

Digging even deeper, it turns out that the tests themselves cause the problems:

{code}
package org.apache.drill.exec.util;

public class TestUtilities {
  ...
  /**
   * Make the dfs.tmp schema immutable, so that tests writers don't use the dfs.tmp to create views.
   * Schema "dfs.tmp" added as part of the default bootstrap plugins file that comes with drill-java-exec jar
   */
  public static void makeDfsTmpSchemaImmutable(final StoragePluginRegistry pluginRegistry) throws ExecutionSetupException {
{code}

Executing this test code causes Drillbit startup to fail.

This code is called from the following test framework location:

{code}
package org.apache.drill;

public class BaseTestQuery extends ExecTest {
  private static void openClient() throws Exception {
    ..
      TestUtilities.makeDfsTmpSchemaImmutable(pluginRegistry);
{code}

How do the tests pass? Via magic at startup time:

{code}
  public static final String TEMP_SCHEMA = "dfs_test.tmp";
      ..
      put(ExecConstants.DEFAULT_TEMPORARY_WORKSPACE, TEMP_SCHEMA);
{code}

All of this seems very fragile. A much better solution is to postpone the check until executing a query that does an actual CTTAS statement.

The workaround is to add the same setup logic to the new test framework to work around the work around for dfs.tmp.

> Drill requires that dfs.tmp be set up a start time, even if CTTAS not used
> --------------------------------------------------------------------------
>
>                 Key: DRILL-5255
>                 URL: https://issues.apache.org/jira/browse/DRILL-5255
>             Project: Apache Drill
>          Issue Type: Bug
>    Affects Versions: 1.10
>            Reporter: Paul Rogers
>            Assignee: Arina Ielchiieva
>             Fix For: 1.10
>
>
> Drill can operate in embedded mode. In this mode, no storage plugin definitions other than the defaults may be present. In particular, when using the Drill test framework, only those storage plugins defined in the Drill code are available.
> Yet, Drill checks for the existence of the dfs.tmp plugin definition (as named by the {{drill.exec.default_temporary_workspace}} parameter. Because this plugin is not defined, an exception occurs:
> {code}
> org.apache.drill.common.exceptions.UserException: PARSE ERROR: Unable to create or drop tables/views. Schema [dfs.tmp] is immutable.
> [Error Id: 792d4e5d-3f31-4f38-8bb4-d108f1a808f6 ]
> 	at org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:544)
> 	at org.apache.drill.exec.planner.sql.SchemaUtilites.resolveToMutableDrillSchema(SchemaUtilites.java:184)
> 	at org.apache.drill.exec.planner.sql.SchemaUtilites.getTemporaryWorkspace(SchemaUtilites.java:201)
> 	at org.apache.drill.exec.server.Drillbit.validateTemporaryWorkspace(Drillbit.java:264)
> 	at org.apache.drill.exec.server.Drillbit.run(Drillbit.java:135)
> 	at org.apache.drill.test.ClusterFixture.startDrillbits(ClusterFixture.java:207)
> 	...
> {code}
> Expected that either a configuration would exist that would use the default /tmp/drill location, or that the check for {{drill.tmp}} would be deferred until it is actually required (such as when executing a CTTAS statement.)
> It seemed that the test framework must be altered to work around this problem by defining the necessary workspace. Unfortunately, the Drillbit must start before we can define the workspace needed for the Drillbit to start. So, this workaround is not possible.
> Further, users of the embedded Drillbit may not know to do this configuration.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)