You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "ASF subversion and git services (JIRA)" <ji...@apache.org> on 2019/04/14 21:35:00 UTC
[jira] [Commented] (IMPALA-8346) Testcase builder for query planner
introduced a data load regression on remote clusters
[ https://issues.apache.org/jira/browse/IMPALA-8346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16817453#comment-16817453 ]
ASF subversion and git services commented on IMPALA-8346:
---------------------------------------------------------
Commit df2d9f1333967e166895c13361d99f6554022eec in impala's branch refs/heads/master from David Knupp
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=df2d9f1 ]
IMPALA-8346: Don't create FE testcase files unless testing locally
The same test data setup scripts get called when loading data for
mini-cluster testing and testing against a real deployed cluster.
Unfortunately, we're seeing more and more that not all set up steps
apply equally in both situations.
This patch avoids one such example. It skips the creation of TPCDS
testcase files that are used by the FE java tests. These tests don't
run against deployed clusters.
Change-Id: Ibe11d7cb50d9e2657152c94f8defcbc69ca7e1ba
Reviewed-on: http://gerrit.cloudera.org:8080/12958
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
> Testcase builder for query planner introduced a data load regression on remote clusters
> ---------------------------------------------------------------------------------------
>
> Key: IMPALA-8346
> URL: https://issues.apache.org/jira/browse/IMPALA-8346
> Project: IMPALA
> Issue Type: Bug
> Affects Versions: Impala 3.2.0, Impala 3.3.0
> Reporter: David Knupp
> Assignee: bharath v
> Priority: Blocker
>
> The patch to address IMPALA-5872 introduced a new script into our data load process. This script has been tested against the single-node mini-cluster, but doesn't appear to run against actual (remote) clusters.
> {noformat}
> Starting Impala Shell without Kerberos authentication
> Opened TCP connection to remote-coordinator-node.mycompany.com:21000
> Connected to remote-coordinator-node.mycompany.com:21000
> Server version: impalad version 3.2.0-cdh6.x-SNAPSHOT RELEASE (build 2e55383eb86de20499e2f9327cd5bcbda6788e50)
> Query: use `tpcds`
> Query: use `tpcds`
> Query: COPY TESTCASE TO '/test-warehouse/tpcds-testcase-data' -- start query 1 in stream 0 using template query11.tpl
> with year_total as (
> select c_customer_id customer_id
> ,c_first_name customer_first_name
> ,c_last_name customer_last_name
> ,c_preferred_cust_flag customer_preferred_cust_flag
> ,c_birth_country customer_birth_country
> ,c_login customer_login
> ,c_email_address customer_email_address
> ,d_year dyear
> ,sum(ss_ext_list_price-ss_ext_discount_amt) year_total
> ,'s' sale_type
> from customer
> ,store_sales
> ,date_dim
> where c_customer_sk = ss_customer_sk
> and ss_sold_date_sk = d_date_sk
> group by c_customer_id
> ,c_first_name
> ,c_last_name
> ,c_preferred_cust_flag
> ,c_birth_country
> ,c_login
> ,c_email_address
> ,d_year
> union all
> select c_customer_id customer_id
> ,c_first_name customer_first_name
> ,c_last_name customer_last_name
> ,c_preferred_cust_flag customer_preferred_cust_flag
> ,c_birth_country customer_birth_country
> ,c_login customer_login
> ,c_email_address customer_email_address
> ,d_year dyear
> ,sum(ws_ext_list_price-ws_ext_discount_amt) year_total
> ,'w' sale_type
> from customer
> ,web_sales
> ,date_dim
> where c_customer_sk = ws_bill_customer_sk
> and ws_sold_date_sk = d_date_sk
> group by c_customer_id
> ,c_first_name
> ,c_last_name
> ,c_preferred_cust_flag
> ,c_birth_country
> ,c_login
> ,c_email_address
> ,d_year
> )
> select
> t_s_secyear.customer_id
> ,t_s_secyear.customer_first_name
> ,t_s_secyear.customer_last_name
> ,t_s_secyear.customer_email_address
> from year_total t_s_firstyear
> ,year_total t_s_secyear
> ,year_total t_w_firstyear
> ,year_total t_w_secyear
> where t_s_secyear.customer_id = t_s_firstyear.customer_id
> and t_s_firstyear.customer_id = t_w_secyear.customer_id
> and t_s_firstyear.customer_id = t_w_firstyear.customer_id
> and t_s_firstyear.sale_type = 's'
> and t_w_firstyear.sale_type = 'w'
> and t_s_secyear.sale_type = 's'
> and t_w_secyear.sale_type = 'w'
> and t_s_firstyear.dyear = 2001
> and t_s_secyear.dyear = 2001+1
> and t_w_firstyear.dyear = 2001
> and t_w_secyear.dyear = 2001+1
> and t_s_firstyear.year_total > 0
> and t_w_firstyear.year_total > 0
> and case when t_w_firstyear.year_total > 0 then t_w_secyear.year_total / t_w_firstyear.year_total else 0.0 end
> > case when t_s_firstyear.year_total > 0 then t_s_secyear.year_total / t_s_firstyear.year_total else 0.0 end
> order by t_s_secyear.customer_id
> ,t_s_secyear.customer_first_name
> ,t_s_secyear.customer_last_name
> ,t_s_secyear.customer_email_address
> limit 100
> Query submitted at: 2019-03-23 23:40:12 (Coordinator: http://remote-coordinator-node.mycompany.com:25000)
> ERROR: ImpalaRuntimeException: Error writing test case output to file: hdfs://namenode.mycompany.com:8020/test-warehouse/tpcds-testcase-data/impala-testcase-data-6430bc87-5337-4e65-b6aa-d059088f3a4b
> CAUSED BY: AccessControlException: Permission denied: user=impala, access=WRITE, inode="/test-warehouse/tpcds-testcase-data":hdfs:hdfs:drwxr-xr-x
> [...]
> Could not execute command: COPY TESTCASE TO '/test-warehouse/tpcds-testcase-data' -- start query 1 in stream 0 using template query11.tpl
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org