You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "YUBI LEE (Jira)" <ji...@apache.org> on 2023/10/12 23:37:00 UTC
[jira] [Commented] (IMPALA-10871) Add MetastoreShim to support Apache Hive 3.x
[ https://issues.apache.org/jira/browse/IMPALA-10871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17774714#comment-17774714 ]
YUBI LEE commented on IMPALA-10871:
-----------------------------------
Why {{CommitTxnEvent}} is not handled on {{fe/src/compat-apache-hive-3/java/org/apache/impala/compat/MetastoreShim.java
}}?
{code}
13 Oct 2023 00:14:02 INFO o.a.i.c.e.MetastoreEventsProcessor [MetastoreEventsProcessor] - Received 2 events. Start event id : 2467173
13 Oct 2023 00:14:02 ERROR o.a.i.c.e.MetastoreEventsProcessor [MetastoreEventsProcessor] - Unexpected exception received while processing event
java.lang.UnsupportedOperationException: CommitTxnEvent is not supported.
at org.apache.impala.compat.MetastoreShim$CommitTxnEvent.<init>(MetastoreShim.java:594)
at org.apache.impala.catalog.events.MetastoreEvents$MetastoreEventFactory.get(MetastoreEvents.java:190)
at org.apache.impala.catalog.events.MetastoreEvents$MetastoreEventFactory.getFilteredEvents(MetastoreEvents.java:247)
at org.apache.impala.catalog.events.MetastoreEventsProcessor.processEvents(MetastoreEventsProcessor.java:921)
at org.apache.impala.catalog.events.MetastoreEventsProcessor.processEvents(MetastoreEventsProcessor.java:833)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
{code}
After receiving {{CommitTxnEvent}}, {{events-processor.status}} for catalogd changes to "ERROR" and event processing stops.
> Add MetastoreShim to support Apache Hive 3.x
> --------------------------------------------
>
> Key: IMPALA-10871
> URL: https://issues.apache.org/jira/browse/IMPALA-10871
> Project: IMPALA
> Issue Type: Task
> Reporter: Quanlong Huang
> Assignee: Fucun Chu
> Priority: Major
>
> Impala 3.x supports both Hive 2 and 3 by a MetastoreShim that is chose at compilation time:
> {code:bash}
> fe/src/compat-hive-2
> └── java/org/apache
> ├── hadoop/hive/common/ValidWriteIdList.java
> └── impala/compat/MetastoreShim.java
> fe/src/compat-hive-3
> └── java/org/apache/impala/compat
> ├── HiveMetadataFormatUtils.java
> └── MetastoreShim.java {code}
> In pom.xml:
> {code:xml}
> <plugin>
> <groupId>org.codehaus.mojo</groupId>
> <artifactId>build-helper-maven-plugin</artifactId>
> <version>1.5</version>
> <executions>
> <!-- Tell maven about our generated files -->
> <execution>
> <id>add-source</id>
> <phase>generate-sources</phase>
> <goals>
> <goal>add-source</goal>
> </goals>
> <configuration>
> <sources>
> ...
> <source>${project.basedir}/src/compat-hive-${hive.major.version}/java</source>
> </sources>
> </configuration>
> </execution>
> </executions>
> </plugin>
> {code}
> Now MetastoreShim evolves to only support CDP Hive which contains some breaking changes of Hive 4.x.
> To support Apache Hive 3.x, we need another MetastoreShim. Some codes that directly use Hive 4 APIs may need to be ignored in compilation, e.g. fe/src/main/java/org/apache/impala/catalog/metastore/*.java
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org