[jira] [Commented] (TAJO-1493) Add a method to get partition directories with filter conditions.

["Tajo QA (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/TAJO-1493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14727157#comment-14727157 ] 

Tajo QA commented on TAJO-1493:
-------------------------------

{color:red}*-1 overall.*{color}  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12753733/TAJO-1493.patch
  against master revision release-0.9.0-rc0-436-gace0480.

    {color:green}+1 @author.{color}  The patch does not contain any @author tags.

    {color:green}+1 tests included.{color}  The patch appears to include 3 new or modified test files.

    {color:green}+1 javac.{color}  The applied patch does not increase the total number of javac compiler warnings.

    {color:green}+1 javadoc.{color}  The applied patch does not increase the total number of javadoc warnings.

    {color:green}+1 checkstyle.{color}  The patch generated 0 code style errors.

    {color:red}-1 findbugs.{color}  The patch appears to cause Findbugs (version 2.0.3) to fail.

    {color:green}+1 release audit.{color}  The applied patch does not increase the total number of release audit warnings.

    {color:green}+1 core tests.{color}  The patch passed unit tests in tajo-catalog/tajo-catalog-client tajo-catalog/tajo-catalog-common tajo-catalog/tajo-catalog-drivers/tajo-hive tajo-catalog/tajo-catalog-server tajo-common tajo-core tajo-core-tests tajo-plan.

Test results: https://builds.apache.org/job/PreCommit-TAJO-Build/840//testReport/
Findbugs results: https://builds.apache.org/job/PreCommit-TAJO-Build/840//findbugsResult
Console output: https://builds.apache.org/job/PreCommit-TAJO-Build/840//console

This message is automatically generated.

> Add a method to get partition directories with filter conditions.
> -----------------------------------------------------------------
>
>                 Key: TAJO-1493
>                 URL: https://issues.apache.org/jira/browse/TAJO-1493
>             Project: Tajo
>          Issue Type: Sub-task
>          Components: Catalog
>            Reporter: Jaehwa Jung
>            Assignee: Jaehwa Jung
>             Fix For: 0.11.0, 0.12.0
>
>         Attachments: TAJO-1493.patch
>
>
> Currently, PartitionedTableRewriter take a look into partition directories for rewriting filter conditions. It get all sub directories of table path because catalog doesn’t provide partition directories. But if there are lots of sub directories on HDFS, such as, more than 10,000 directories, it might be cause overload to NameNode. Thus, CatalogStore need to provide partition directories for specified filter conditions. I designed new method to CatalogStore as follows:
> * method name: getPartitionsWithConditionFilters
> * first parameter: database name
> * second parameter: table name
> * third parameter: where clause (included target column name and partition value)
> * return values: List<org.apache.tajo.catalog.proto.CatalogProtos.TablePartitionProto>
> * description: It scan right partition directories on CatalogStore with where caluse. 
>   For examples, users set parameters as following:
> ** first parameter: default
> ** second parameter: table1
> ** third parameter: COLUMN_NAME = 'col1' AND PARTITION_VALUE = '3
> In the previous cases, this method will create select clause as follows.
> {code:xml}
> SELECT DISTINCT A.PATH
> FROM PARTITIONS A, (
>   SELECT B.PARTITION_ID
>   FROM PARTITION_KEYS B
>   WHERE B.PARTITION_ID > 0 
>   AND (
>     COLUMN_NAME = 'col1' AND PARTITION_VALUE = '3'
>   )
> ) B
> WHERE A.PARTITION_ID > 0
> AND A.TID = ${table_id}
> AND A.PARTITION_ID = B.PARTITION_ID
> {code}
> At the first time, I considered to use EvalNode instead of where clause. But I can’t use it because of recursive related problems between tajo-catalog module and tajo-plan module. So, I’ll implement utility class to convert EvalNode to SQL.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)