You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2021/01/29 10:39:30 UTC
[GitHub] [iceberg] Simmyo opened a new issue #2181: Flink: Can't use flink sql to create hive catalog
Simmyo opened a new issue #2181:
URL: https://github.com/apache/iceberg/issues/2181
I tried to use flink sql to create hive catalog, but an error was reported, Can you give me some advice?
**My environment:** flink1.11.2, iceberg0.11.0,hive2.3.6
**Sql**:
```
EnvironmentSettings settings = EnvironmentSettings.newInstance().useBlinkPlanner().inStreamingMode().build();
TableEnvironment tableEnv = TableEnvironment.create(settings);
tableEnv.executeSql("create catalog iceberg_hive_1 with ('type'='iceberg','catalog-type'='hive','uri'='thrift://xxx:9083','warehouse'='hdfs://xxx:8020/user/hive/warehouse')");
```
**error message:**
```
The following factories have been considered:
org.apache.flink.table.catalog.hive.factories.HiveCatalogFactory
org.apache.flink.table.catalog.GenericInMemoryCatalogFactory
at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:302)
at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:198)
at org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:149)
at org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:699)
at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:232)
at org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:916)
at org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:992)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1875)
at org.apache.flink.runtime.security.contexts.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:992)
Caused by: org.apache.flink.table.api.NoMatchingTableFactoryException: Could not find a suitable table factory for 'org.apache.flink.table.factories.CatalogFactory' in
the classpath.
```
**My dependences:**
```
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-java</artifactId>
<version>${flink.version}</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-connector-hive_2.12</artifactId>
<version>${flink.version}</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-streaming-scala_2.12</artifactId>
<version>${flink.version}</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-table-api-java-bridge_2.12</artifactId>
<version>${flink.version}</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-table-planner-blink_2.12</artifactId>
<version>${flink.version}</version>
</dependency>
<dependency>
<groupId>org.apache.iceberg</groupId>
<artifactId>iceberg-flink-runtime</artifactId>
<version>${iceberg.version}</version>
</dependency>
<dependency>
<groupId>org.apache.iceberg</groupId>
<artifactId>iceberg-flink</artifactId>
<version>${iceberg.version}</version>
</dependency>
<dependency>
<groupId>org.apache.iceberg</groupId>
<artifactId>iceberg-hive-metastore</artifactId>
<version>${iceberg.version}</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
<version>${hadoop.version}</version>
</dependency>
<dependency>
<groupId>org.apache.hive</groupId>
<artifactId>hive-exec</artifactId>
<version>${hive.version}</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-table-planner_2.12</artifactId>
<version>${flink.version}</version>
</dependency>
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] Simmyo commented on issue #2181: Flink: Can't use flink sql to create hive catalog
Posted by GitBox <gi...@apache.org>.
Simmyo commented on issue #2181:
URL: https://github.com/apache/iceberg/issues/2181#issuecomment-769732086
> `iceberg-flink` and `iceberg-hive-metastore` is unnecessary，you can remove it from pom, we only need `iceberg-flink-runtime-xxx.jar` and `flink-sql-connector-hive-2.3.6_2.11-1.11.0.jar`,see [doc](https://iceberg.apache.org/flink/)
I still get the same error after I remove it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] Simmyo edited a comment on issue #2181: Flink: Can't use flink sql to create hive catalog
Posted by GitBox <gi...@apache.org>.
Simmyo edited a comment on issue #2181:
URL: https://github.com/apache/iceberg/issues/2181#issuecomment-769736710
> > `iceberg-flink` and `iceberg-hive-metastore` is unnecessary，you can remove it from pom, we only need `iceberg-flink-runtime-xxx.jar` and `flink-sql-connector-hive-2.3.6_2.11-1.11.0.jar`,see [doc](https://iceberg.apache.org/flink/)
>
> I still get the same error after I remove it.
remove `iceberg-hive-metastore` , unable to connect hive metastore with flink datastream.
```
TableLoader hiveTableLoader = TableLoader.fromCatalog(
(CatalogLoader) () -> new HiveCatalog("hive", "thrift://xxx:9083",
"hdfs://xxx:8020/user/hive/warehouse", 3, hadoopConf),
TableIdentifier.of(dbName, tableName));
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] zhangjun0x01 commented on issue #2181: Flink: Can't use flink sql to create hive catalog
Posted by GitBox <gi...@apache.org>.
zhangjun0x01 commented on issue #2181:
URL: https://github.com/apache/iceberg/issues/2181#issuecomment-769621263
`iceberg-flink` and `iceberg-hive-metastore` is unnecessary，you can remove it from pom, we only need `iceberg-flink-runtime-xxx.jar` and `flink-sql-connector-hive-2.3.6_2.11-1.11.0.jar`,see [doc](https://iceberg.apache.org/flink/)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] openinx commented on issue #2181: Flink: Can't use flink sql to create hive catalog
Posted by GitBox <gi...@apache.org>.
openinx commented on issue #2181:
URL: https://github.com/apache/iceberg/issues/2181#issuecomment-769811208
@Simmyo , Yeah, I guess it's true that we need the iceberg-hive-metastore to load iceberg table inside the task managers if you are running flink data streaming job.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] Simmyo commented on issue #2181: Flink: Can't use flink sql to create hive catalog
Posted by GitBox <gi...@apache.org>.
Simmyo commented on issue #2181:
URL: https://github.com/apache/iceberg/issues/2181#issuecomment-769725473
> Have you try execute sql with command in sql-shell?
I was able to do this successfully with sql-shell
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] Simmyo commented on issue #2181: Flink: Can't use flink sql to create hive catalog
Posted by GitBox <gi...@apache.org>.
Simmyo commented on issue #2181:
URL: https://github.com/apache/iceberg/issues/2181#issuecomment-769852537
> @Simmyo , Yeah, I guess it's true that we need the iceberg-hive-metastore to load iceberg table inside the task managers if you are running flink data streaming job.
I think the official website document should be more detailed, I can't get much information
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] Simmyo commented on issue #2181: Flink: Can't use flink sql to create hive catalog
Posted by GitBox <gi...@apache.org>.
Simmyo commented on issue #2181:
URL: https://github.com/apache/iceberg/issues/2181#issuecomment-769736710
> > `iceberg-flink` and `iceberg-hive-metastore` is unnecessary，you can remove it from pom, we only need `iceberg-flink-runtime-xxx.jar` and `flink-sql-connector-hive-2.3.6_2.11-1.11.0.jar`,see [doc](https://iceberg.apache.org/flink/)
>
> I still get the same error after I remove it.
remove `iceberg-hive-metastore` , unable to connect hive metastore.
```
TableLoader hiveTableLoader = TableLoader.fromCatalog(
(CatalogLoader) () -> new HiveCatalog("hive", "thrift://xxx:9083",
"hdfs://xxx:8020/user/hive/warehouse", 3, hadoopConf),
TableIdentifier.of(dbName, tableName));
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] andrefranciscol commented on issue #2181: Flink: Can't use flink sql to create hive catalog
Posted by GitBox <gi...@apache.org>.
andrefranciscol commented on issue #2181:
URL: https://github.com/apache/iceberg/issues/2181#issuecomment-1002661397
I'm facing the same problem. same error when launching flink job, and when executing in the flink SQL shell. @Simmyo Did you find any clue about this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] simuhunluo commented on issue #2181: Flink: Can't use flink sql to create hive catalog
Posted by GitBox <gi...@apache.org>.
simuhunluo commented on issue #2181:
URL: https://github.com/apache/iceberg/issues/2181#issuecomment-769618965
Have you try execute sql with command in sql-shell?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] Simmyo closed issue #2181: Flink: Can't use flink sql to create hive catalog
Posted by GitBox <gi...@apache.org>.
Simmyo closed issue #2181:
URL: https://github.com/apache/iceberg/issues/2181
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] Simmyo commented on issue #2181: Flink: Can't use flink sql to create hive catalog
Posted by GitBox <gi...@apache.org>.
Simmyo commented on issue #2181:
URL: https://github.com/apache/iceberg/issues/2181#issuecomment-769725687
> `iceberg-flink` and `iceberg-hive-metastore` is unnecessary，you can remove it from pom, we only need `iceberg-flink-runtime-xxx.jar` and `flink-sql-connector-hive-2.3.6_2.11-1.11.0.jar`,see [doc](https://iceberg.apache.org/flink/)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org