[jira] [Created] (IMPALA-15437) Dataload can fail on Hive Iceberg

["Michael Smith (Jira)" <ji...@apache.org>]

Michael Smith created IMPALA-15437:
--------------------------------------

             Summary: Dataload can fail on Hive Iceberg
                 Key: IMPALA-15437
                 URL: https://issues.apache.org/jira/browse/IMPALA-15437
             Project: IMPALA
          Issue Type: Task
          Components: Test
    Affects Versions: Impala 5.0.0
            Reporter: Michael Smith
            Assignee: Fang-Yu Rao


On Ubuntu 22.04 (x86_64), I started running into an error with dataload on the master branch that hadn't been there before
{code:java}
ERROR : FAILED: Execution Error, return code 40000 from org.apache.hadoop.hive.ql.exec.MoveTask. Error committing job: job_17902705519450_0022 for table: functional_parquet.iceberg_avro_format
INFO  : Completed executing command(queryId=michael_20260924103245_79db4f29-1da2-4fc6-9adf-63bc6565fcf7); Time taken: 29.937 seconds
INFO  : OK
Error: Error while compiling statement: FAILED: Execution Error, return code 40000 from org.apache.hadoop.hive.ql.exec.MoveTask. Error committing job: job_17902705519450_0022 for table: functional_parquet.iceberg_avro_format; Query ID: michael_20260924103245_79db4f29-1da2-4fc6-9adf-63bc6565fcf7 (state=08S01,code=40000)
java.sql.SQLException: Error while compiling statement: FAILED: Execution Error, return code 40000 from org.apache.hadoop.hive.ql.exec.MoveTask. Error committing job: job_17902705519450_0022 for table: functional_parquet.iceberg_avro_format; Query ID: michael_20260924103245_79db4f29-1da2-4fc6-9adf-63bc6565fcf7
	at org.apache.hive.jdbc.HiveStatement.waitForOperationToComplete(HiveStatement.java:472)
	at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:334)
	at org.apache.hive.beeline.Commands.executeInternal(Commands.java:1008)
	at org.apache.hive.beeline.Commands.execute(Commands.java:1218)
	at org.apache.hive.beeline.Commands.sql(Commands.java:1147)
	at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:1498)
	at org.apache.hive.beeline.BeeLine.execute(BeeLine.java:1356)
	at org.apache.hive.beeline.BeeLine.executeFile(BeeLine.java:1330)
	at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:1128)
	at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:1083)
	at org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:541)
	at org.apache.hive.beeline.BeeLine.main(BeeLine.java:523)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:569)
	at org.apache.hadoop.util.RunJar.run(RunJar.java:333)
	at org.apache.hadoop.util.RunJar.main(RunJar.java:254) {code}
I suspect I redownloaded the CDP Hive tarball and the jar ordering on the filesystem changed. It seems to have exposed mismatched implementations of {{org.apache.iceberg.BaseFile}} in {{hive-iceberg-handler-\*.jar}} / {{hive-iceberg-shading-\*.jar}} and {{patched-iceberg-core-*.jar}}.

I was able to work around it with a patch to {{run-hive-server.sh}}, but it should really be fixed in the CDP Hive build
{code}
# Hive's lib/ jars are added in unsorted 'find' order, and patched-iceberg-core has an
# incompatible org.apache.iceberg.BaseFile (serialVersionUID) from the one in
# hive-iceberg-handler that Tez tasks use to serialize commit files. Put the handler
# first via TEZ_CONF_DIR, the only entry bin/hive places ahead of lib/. The "/./" keeps
# Hadoop's classpath de-duplication from dropping it in favor of the later lib/ entry.
for jar in "${HIVE_HOME}"/lib/hive-iceberg-handler-*.jar; do
  if [[ -f "$jar" ]]; then
    export TEZ_CONF_DIR="${TEZ_CONF_DIR:-/etc/tez/conf}:${HIVE_HOME}/lib/./$(basename "$jar")"
  fi
done
{code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)