You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@drill.apache.org by "Chun Chang (JIRA)" <ji...@apache.org> on 2016/02/17 22:37:18 UTC
[jira] [Comment Edited] (DRILL-4392) CTAS with partition writes an
internal field into generated parquet files
[ https://issues.apache.org/jira/browse/DRILL-4392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15151214#comment-15151214 ]
Chun Chang edited comment on DRILL-4392 at 2/17/16 9:36 PM:
------------------------------------------------------------
Our nightly automation run also hit this:
{noformat}
Summary
----------------------------------------------------------------------------------------------------------------
Execution Failures:
Verification Failures:
/root/drillAutomation/framework-master/framework/resources/Functional/interpreted_partition_pruning/ctas_auto_partition/hierarchical/data/textSelectStartFromPartition.q
/root/drillAutomation/framework-master/framework/resources/Functional/ctas/ctas_auto_partition/general/data/q2.q
/root/drillAutomation/framework-master/framework/resources/Functional/ctas/ctas_auto_partition/general/data/q7.q
/root/drillAutomation/framework-master/framework/resources/Functional/ctas/ctas_auto_partition/existing_partition_pruning/parquet/data/drill3410.q
/root/drillAutomation/framework-master/framework/resources/Functional/ctas/ctas_auto_partition/general/data/q4.q
/root/drillAutomation/framework-master/framework/resources/Functional/ctas/ctas_auto_partition/general/data/q1.q
/root/drillAutomation/framework-master/framework/resources/Functional/ctas/ctas_auto_partition/general/data/q5.q
/root/drillAutomation/framework-master/framework/resources/Functional/ctas/ctas_auto_partition/existing_partition_pruning/json/data/jsonselectStarFromPartition.q
/root/drillAutomation/framework-master/framework/resources/Functional/ctas/ctas_auto_partition/general/data/drill3361.q
/root/drillAutomation/framework-master/framework/resources/Functional/ctas/ctas_auto_partition/general/data/q9.q
/root/drillAutomation/framework-master/framework/resources/Functional/ctas/ctas_auto_partition/general/data/q11.q
/root/drillAutomation/framework-master/framework/resources/Functional/ctas/ctas_auto_partition/general/data/q8.q
/root/drillAutomation/framework-master/framework/resources/Functional/ctas/ctas_auto_partition/general/data/q6.q
/root/drillAutomation/framework-master/framework/resources/Functional/ctas/ctas_auto_partition/existing_partition_pruning/hierarchical/data/textSelectStartFromPartition.q
/root/drillAutomation/framework-master/framework/resources/Functional/ctas/ctas_auto_partition/existing_partition_pruning/parquet/data/parquetselectStarFromPartition.q
/root/drillAutomation/framework-master/framework/resources/Functional/ctas/ctas_auto_partition/general/data/q10.q
/root/drillAutomation/framework-master/framework/resources/Functional/ctas/ctas_auto_partition/general/data/q3.q
Timeout Failures:
----------------------------------------------------------------------------------------------------------------
Passing tests: 4979
Execution Failures: 0
VerificationFailures: 17
Timeouts: 0
Canceled: 0
{noformat}
was (Author: cchang@maprtech.com):
Our night automation run also hit this:
{noformat}
Summary
----------------------------------------------------------------------------------------------------------------
Execution Failures:
Verification Failures:
/root/drillAutomation/framework-master/framework/resources/Functional/interpreted_partition_pruning/ctas_auto_partition/hierarchical/data/textSelectStartFromPartition.q
/root/drillAutomation/framework-master/framework/resources/Functional/ctas/ctas_auto_partition/general/data/q2.q
/root/drillAutomation/framework-master/framework/resources/Functional/ctas/ctas_auto_partition/general/data/q7.q
/root/drillAutomation/framework-master/framework/resources/Functional/ctas/ctas_auto_partition/existing_partition_pruning/parquet/data/drill3410.q
/root/drillAutomation/framework-master/framework/resources/Functional/ctas/ctas_auto_partition/general/data/q4.q
/root/drillAutomation/framework-master/framework/resources/Functional/ctas/ctas_auto_partition/general/data/q1.q
/root/drillAutomation/framework-master/framework/resources/Functional/ctas/ctas_auto_partition/general/data/q5.q
/root/drillAutomation/framework-master/framework/resources/Functional/ctas/ctas_auto_partition/existing_partition_pruning/json/data/jsonselectStarFromPartition.q
/root/drillAutomation/framework-master/framework/resources/Functional/ctas/ctas_auto_partition/general/data/drill3361.q
/root/drillAutomation/framework-master/framework/resources/Functional/ctas/ctas_auto_partition/general/data/q9.q
/root/drillAutomation/framework-master/framework/resources/Functional/ctas/ctas_auto_partition/general/data/q11.q
/root/drillAutomation/framework-master/framework/resources/Functional/ctas/ctas_auto_partition/general/data/q8.q
/root/drillAutomation/framework-master/framework/resources/Functional/ctas/ctas_auto_partition/general/data/q6.q
/root/drillAutomation/framework-master/framework/resources/Functional/ctas/ctas_auto_partition/existing_partition_pruning/hierarchical/data/textSelectStartFromPartition.q
/root/drillAutomation/framework-master/framework/resources/Functional/ctas/ctas_auto_partition/existing_partition_pruning/parquet/data/parquetselectStarFromPartition.q
/root/drillAutomation/framework-master/framework/resources/Functional/ctas/ctas_auto_partition/general/data/q10.q
/root/drillAutomation/framework-master/framework/resources/Functional/ctas/ctas_auto_partition/general/data/q3.q
Timeout Failures:
----------------------------------------------------------------------------------------------------------------
Passing tests: 4979
Execution Failures: 0
VerificationFailures: 17
Timeouts: 0
Canceled: 0
{noformat}
> CTAS with partition writes an internal field into generated parquet files
> -------------------------------------------------------------------------
>
> Key: DRILL-4392
> URL: https://issues.apache.org/jira/browse/DRILL-4392
> Project: Apache Drill
> Issue Type: Bug
> Reporter: Jinfeng Ni
> Assignee: Steven Phillips
> Priority: Blocker
>
> On today's master branch:
> {code}
> select * from sys.version;
> +-----------------+-------------------------------------------+---------------------------------------------------------------------+----------------------------+-----------------+----------------------------+
> | version | commit_id | commit_message | commit_time | build_email | build_time |
> +-----------------+-------------------------------------------+---------------------------------------------------------------------+----------------------------+-----------------+----------------------------+
> | 1.5.0-SNAPSHOT | 9a3a5c4ff670a50a49f61f97dd838da59a12f976 | DRILL-4382: Remove dependency on drill-logical from vector package | 16.02.2016 @ 11:58:48 PST | jni@apache.org | 16.02.2016 @ 17:40:44 PST |
> +-----------------+-------------------------------------------+---------------------------------------------------------------------+----------------------------+-----------------
> {code}
> Parquet table created by Drill's CTAS statement has one internal field "P_A_R_T_I_T_I_O_N_C_O_M_P_A_R_A_T_O_R". This additional field would not impact non-star query, but would cause incorrect result for star query.
> {code}
> use dfs.tmp;
> create table nation_ctas partition by (n_regionkey) as select * from cp.`tpch/nation.parquet`;
> select * from dfs.tmp.nation_ctas limit 6;
> +--------------+----------------+--------------+-----------------------------------------------------------------------------------------------------------------+----------------------------------------+
> | n_nationkey | n_name | n_regionkey | n_comment | P_A_R_T_I_T_I_O_N_C_O_M_P_A_R_A_T_O_R |
> +--------------+----------------+--------------+-----------------------------------------------------------------------------------------------------------------+----------------------------------------+
> | 5 | ETHIOPIA | 0 | ven packages wake quickly. regu | true |
> | 15 | MOROCCO | 0 | rns. blithely bold courts among the closely regular packages use furiously bold platelets? | false |
> | 14 | KENYA | 0 | pending excuses haggle furiously deposits. pending, express pinto beans wake fluffily past t | false |
> | 0 | ALGERIA | 0 | haggle. carefully final deposits detect slyly agai | false |
> | 16 | MOZAMBIQUE | 0 | s. ironic, unusual asymptotes wake blithely r | false |
> | 24 | UNITED STATES | 1 | y final packages. slow foxes cajole quickly. quickly silent platelets breach ironic accounts. unusual pinto be | true
> {code}
> This basically breaks all the parquet files created by Drill's CTAS with partition support.
> Also, it will also fail one of the Pre-commit functional test [1]
> [1] https://github.com/mapr/drill-test-framework/blob/master/framework/resources/Functional/ctas/ctas_auto_partition/general/data/drill3361.q
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)