You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Jingsong Lee (Jira)" <ji...@apache.org> on 2019/12/12 06:18:00 UTC
[jira] [Updated] (FLINK-15185) Hive sink can not run in standalone
mode
[ https://issues.apache.org/jira/browse/FLINK-15185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jingsong Lee updated FLINK-15185:
---------------------------------
Description:
Now in hive HadoopFileSystemFactory, we use org.apache.flink.runtime.fs.hdfs.HadoopFileSystem to get FileSystem.
But it should not work after we setting default child first class loader. Because in standalone mode, the cluster has no hadoop dependency. So the solution is:
# Add `flink-hadoop-fs` dependency to hive module, not work, because classes with "org.apache.flink" prefix will always be loaded by parent class loader :(
# User add hadoop dependency to standalone cluster, it breaks out-of-the-box.
# Shade hadoop FileSystem in hive module, not complex, good.
was:
Now in hive HadoopFileSystemFactory, we use org.apache.flink.runtime.fs.hdfs.HadoopFileSystem to get FileSystem.
But it should not work after we setting default child first class loader. Because in standalone mode, the cluster has no hadoop dependency. So the solution is:
# Add `flink-hadoop-fs` dependency to hive module, not work, because classes with "org.apache.flink" prefix will always be loaded by parent class loader :(
# User add hadoop dependency to standalone cluster, it breaks out-of-the-box.
# Implement hive hadoop FileSystem, not complex, good.
> Hive sink can not run in standalone mode
> ----------------------------------------
>
> Key: FLINK-15185
> URL: https://issues.apache.org/jira/browse/FLINK-15185
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Hive
> Reporter: Jingsong Lee
> Priority: Major
> Fix For: 1.10.0
>
>
> Now in hive HadoopFileSystemFactory, we use org.apache.flink.runtime.fs.hdfs.HadoopFileSystem to get FileSystem.
> But it should not work after we setting default child first class loader. Because in standalone mode, the cluster has no hadoop dependency. So the solution is:
> # Add `flink-hadoop-fs` dependency to hive module, not work, because classes with "org.apache.flink" prefix will always be loaded by parent class loader :(
> # User add hadoop dependency to standalone cluster, it breaks out-of-the-box.
> # Shade hadoop FileSystem in hive module, not complex, good.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)