[jira] [Created] (IMPALA-9970) create-load-data.sh seems flaky when loading TPC-DS data via Hive

["Fang-Yu Rao (Jira)" <ji...@apache.org>]

Fang-Yu Rao created IMPALA-9970:
-----------------------------------

             Summary: create-load-data.sh seems flaky when loading TPC-DS data via Hive
                 Key: IMPALA-9970
                 URL: https://issues.apache.org/jira/browse/IMPALA-9970
             Project: IMPALA
          Issue Type: Bug
          Components: Infrastructure
            Reporter: Fang-Yu Rao
            Assignee: Csaba Ringhofer
         Attachments: load-tpcds-core-hive-generated-orc-def-block.sql, load-tpcds-core-hive-generated-orc-def-block.sql.log

We saw the following error in a recent build when [create-load-data.sh|https://github.com/apache/impala/blame/master/testdata/bin/create-load-data.sh] was loading the TPC-DS  data into the table of {{store_sales}} under the database of {{tpcds_orc_def}} via Hive.
{code:java}
ERROR : FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.MoveTask. java.io.IOException: Fail to get checksum, since file /test-warehouse/managed/tpcds.store_sales_orc_def/ss_sold_date_sk=2451141/base_0000002/_orc_acid_version is under construction.
INFO  : Completed executing command(queryId=jenkins_20200716163033_4fd19a3b-47b2-4ceb-a7ec-f4781b5c68b2); Time taken: 80.229 seconds
INFO  : OK
Error: Error while compiling statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.MoveTask. java.io.IOException: Fail to get checksum, since file /test-warehouse/managed/tpcds.store_sales_orc_def/ss_sold_date_sk=2451141/base_0000002/_orc_acid_version is under construction. (state=08S01,code=1)
java.sql.SQLException: Error while compiling statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.MoveTask. java.io.IOException: Fail to get checksum, since file /test-warehouse/managed/tpcds.store_sales_orc_def/ss_sold_date_sk=2451141/base_0000002/_orc_acid_version is under construction.
        at org.apache.hive.jdbc.HiveStatement.waitForOperationToComplete(HiveStatement.java:401)
        at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:266)
        at org.apache.hive.beeline.Commands.executeInternal(Commands.java:1007)
        at org.apache.hive.beeline.Commands.execute(Commands.java:1217)
        at org.apache.hive.beeline.Commands.sql(Commands.java:1146)
        at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:1497)
        at org.apache.hive.beeline.BeeLine.execute(BeeLine.java:1355)
        at org.apache.hive.beeline.BeeLine.executeFile(BeeLine.java:1329)
        at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:1127)
        at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:1082)
        at org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:546)
        at org.apache.hive.beeline.BeeLine.main(BeeLine.java:528)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hadoop.util.RunJar.run(RunJar.java:318)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:232)
Closing: 0: jdbc:hive2://localhost:11050/default;auth=none
{code}
It looks like there was some problem when the following statement was executed.
{code:java}
INSERT OVERWRITE TABLE tpcds_orc_def.store_sales PARTITION (ss_sold_date_sk)
 SELECT ss_sold_time_sk,
  ss_item_sk,
  ss_customer_sk,
  ss_cdemo_sk,
  ss_hdemo_sk,
  ss_addr_sk,
  ss_store_sk,
  ss_promo_sk,
  ss_ticket_number,
  ss_quantity,
  ss_wholesale_cost,
  ss_list_price,
  ss_sales_price,
  ss_ext_discount_amt,
  ss_ext_sales_price,
  ss_ext_wholesale_cost,
  ss_ext_list_price,
  ss_ext_tax,
  ss_coupon_amt,
  ss_net_paid,
  ss_net_paid_inc_tax,
  ss_net_profit,
  ss_sold_date_sk
FROM tpcds.store_sales
WHERE ss_sold_date_sk IS NULL;
{code}
For easy reference, [^load-tpcds-core-hive-generated-orc-def-block.sql.log] and [^load-tpcds-core-hive-generated-orc-def-block.sql] are also provided.

Since [~csringhofer] and [~tarmstrong@cloudera.com] might be more familiar with the data loading, maybe you could provide some insight into it. Assign the JIRA to [~csringhofer] for now but please feel free to reassign this JIRA if you see appropriate. Thanks!

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)