You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2021/05/31 14:28:13 UTC
[GitHub] [iceberg] liubo1022126 opened a new issue #2653: [Flink] Can not see hive udf in Flink when use iceberg hive catalog
liubo1022126 opened a new issue #2653:
URL: https://github.com/apache/iceberg/issues/2653
Flink: **flink-1.11.1-bin-scala_2.12**
Jar: ./bin/sql-client.sh embedded -j /app/flink-1.11.1/lib/**iceberg-flink-runtime-0.11.0.jar** -j /app/flink-1.11.1/**flink-sql-connector-hive-2.3.6_2.11-1.11.0.jar** shell
Hive: **2.3.8**
JDK: **1.8.0_202**
As [Flink doc](https://ci.apache.org/projects/flink/flink-docs-release-1.11/dev/table/hive/hive_functions.html#hive-user-defined-functions) say, Hive’s UDF and GenericUDF are automatically translated into Flink’s ScalarFunction.
So what I do is:
1. Create hive udf in hive like.
>create function udf1 as 'com.xxx.xxxx' using jar 'hdfs://icebergHadoop/user/deploy/udf/xxx.jar';
2. Put the udf xxx.jar in flink lib path.
3. Open flink sql-client.
>sql-client.sh embedded
4. Create hive catalog in flink sql-client and use it.
> CREATE CATALOG myhive WITH (
> 'type' = 'hive',
> 'default-database' = 'default',
> 'hive-conf-dir' = '/app/hive/conf'
> );
>
> use catalog myhive;
5. show functions and use udf1 like .
>show functions;
>
>select udf1('a', 'b');
It works Ok.
But when I use iceberg hive catalog As [iceberg hive catalog Doc](https://iceberg.apache.org/flink/#hive-catalog), I can not use hive udf.
As what I do is like:
1.Open flink sql-client with iceberg.
> export HADOOP_CLASSPATH=`$HADOOP_HOME/bin/hadoop classpath`
>
> ./bin/sql-client.sh embedded \
> -j <flink-runtime-directory>/iceberg-flink-runtime-xxx.jar \
> -j <hive-bundlded-jar-directory>/flink-sql-connector-hive-2.3.6_2.11-1.11.0.jar \
> shell
2.Create iceberg hive catalog in flink sql-client and use it.
> CREATE CATALOG hive_catalog WITH (
> 'type'='iceberg',
> 'catalog-type'='hive',
> 'uri'='thrift://localhost:9083',
> 'clients'='5',
> 'property-version'='1',
> 'warehouse'='hdfs://nn:8020/warehouse/path'
> );
3.show functions and use udf1 like **(What is not works Ok, I can not see the hive udf which is registered in hive metastore)**
>show functions;
So how can I use hive udf on iceberg table in Flink, Thx
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] StefanXiepj commented on issue #2653: Flink: Can not see hive udf in Flink when use iceberg hive catalog
Posted by GitBox <gi...@apache.org>.
StefanXiepj commented on issue #2653:
URL: https://github.com/apache/iceberg/issues/2653#issuecomment-851540632
@liubo1022126 I'll try to support this feature.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] liubo1022126 commented on issue #2653: Flink: Can not see hive udf in Flink when use iceberg hive catalog
Posted by GitBox <gi...@apache.org>.
liubo1022126 commented on issue #2653:
URL: https://github.com/apache/iceberg/issues/2653#issuecomment-851776763
thanks for @StefanXiepj ,is there any PR for this? We urgently need this function, or is there any other way to achieve this requirement ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] StefanXiepj commented on issue #2653: Flink: Can not see hive udf in Flink when use iceberg hive catalog
Posted by GitBox <gi...@apache.org>.
StefanXiepj commented on issue #2653:
URL: https://github.com/apache/iceberg/issues/2653#issuecomment-852642015
Someone think it’s like this：Iceberg catalog for hive just support loaded tables from a hive metastore, in other words, iceberg is a table format, so it only focus on the meta data about tables.
But I think we should support this feature, which is the standard defined by the Catalog interface.
@rymurr cc
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org