You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "jini-lee (via GitHub)" <gi...@apache.org> on 2023/06/30 09:14:41 UTC
[GitHub] [iceberg] jini-lee opened a new issue, #7947: Can't read table on hive created by spark-sql
jini-lee opened a new issue, #7947:
URL: https://github.com/apache/iceberg/issues/7947
### Apache Iceberg version
1.3.0 (latest release)
### Query engine
Hive
### Please describe the bug 🐞
Can't read table's data on hive created by spark-sql but data exists.
How can i fix it ?
Below is the work history I tried.
Thanks.
**Version Info**
```
Spark - 3.1.2
Hive - 3.1
```
**Step**
1) I started spark-sql shell.
```
spark-sql \
--packages org.apache.iceberg:iceberg-spark-runtime-3.1_2.12:1.3.0 \
--conf spark.sql.catalog.test=org.apache.iceberg.spark.SparkCatalog \
--conf spark.sql.catalog.test.type=hadoop \
--conf spark.sql.catalog.test.warehouse=hdfs://my-hdfs/warehouse
spark-sql> add jar hdfs://my-hdfs/user/test/tmp/iceberg-hive-runtime-1.3.0.jar;
```
2) Then, create table and insert some data like below. I did check the table be created on warehouse path.
```
spark-sql> CREATE TABLE test.db.test (id bigint, data string) USING iceberg TBLPROPERTIES ('iceberg.engine.hive.enabled' = 'true' );
Time taken: 0.319 seconds
spark-sql> INSERT INTO test.db.test VALUES (1, 'a'), (2, 'b'), (3, 'c');
Time taken: 7.096 seconds
spark-sql> select * from test.db.test;
1 a
2 b
3 c
Time taken: 1.766 seconds, Fetched 3 row(s)
```
3) For query on hive, create external table on beeline.
```
0: jdbc:hive2://hiveserver > add jar hdfs://my-hdfs/user/test/tmp/iceberg-hive-runtime-1.3.0.jar;
0: jdbc:hive2://hiveserver > CREATE DATABASE test;
0: jdbc:hive2://hiveserver > use test;
0: jdbc:hive2://hiveserver > CREATE EXTERBAL TABLE test (id bigint, data string)
. . . .> STORED BY 'org.apache.iceberg.mr.hive.HiveIcebergStorageHandler'
. . . .> LOCATION 'hdfs://my-hdfs/warehouse/db/test';
0: jdbc:hive2://hiveserver > DESCRIBE FORMATTED test;
+-------------------------------+----------------------------------------------------+----------------------------------------------------+
| col_name | data_type | comment |
+-------------------------------+----------------------------------------------------+----------------------------------------------------+
| id | bigint | from deserializer |
| data | string | from deserializer |
| | NULL | NULL |
| # Detailed Table Information | NULL | NULL |
| Database: | test | NULL |
| OwnerType: | USER | NULL |
| Owner: | linead | NULL |
| CreateTime: | Fri Jun 30 18:08:06 JST 2023 | NULL |
| LastAccessTime: | UNKNOWN | NULL |
| Retention: | 0 | NULL |
| Location: | hdfs://my-hdfs/warehouse/db/test | NULL |
| Table Type: | EXTERNAL_TABLE | NULL |
| Table Parameters: | NULL | NULL |
| | EXTERNAL | TRUE |
| | bucketing_version | 2 |
| | current-schema | {\"type\":\"struct\",\"schema-id\":0,\"fields\":[{\"id\":1,\"name\":\"id\",\"required\":false,\"type\":\"long\"},{\"id\":2,\"name\":\"data\",\"required\":false,\"type\":\"string\"}]} |
| | engine.hive.enabled | true |
| | external.table.purge | TRUE |
| | metadata_location | hdfs://my-hdfs/warehouse/db/test/metadata/00000-f4e2e9d0-b7ad-4f25-b333-999e1acf0f00.metadata.json |
| | snapshot-count | 0 |
| | storage_handler | org.apache.iceberg.mr.hive.HiveIcebergStorageHandler |
| | table_type | ICEBERG |
| | transient_lastDdlTime | 1688116086 |
| | uuid | b5812155-41a1-4200-9288-953639ec4771 |
| | NULL | NULL |
| # Storage Information | NULL | NULL |
| SerDe Library: | org.apache.iceberg.mr.hive.HiveIcebergSerDe | NULL |
| InputFormat: | org.apache.iceberg.mr.hive.HiveIcebergInputFormat | NULL |
| OutputFormat: | org.apache.iceberg.mr.hive.HiveIcebergOutputFormat | NULL |
| Compressed: | No | NULL |
| Num Buckets: | 0 | NULL |
| Bucket Columns: | [] | NULL |
| Sort Columns: | [] | NULL |
+-------------------------------+----------------------------------------------------+----------------------------------------------------+
3) Result of select query on beeline
```
0: jdbc:hive2://hiveserver > SELECT * FROM test;
+----------+------------+
| test.id | test.data |
+----------+------------+
+----------+------------+
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] nastra commented on issue #7947: Can't read table on hive created by spark-sql
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on issue #7947:
URL: https://github.com/apache/iceberg/issues/7947#issuecomment-1614406240
seems to be related to https://github.com/apache/iceberg/issues/7924
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] jini-lee closed issue #7947: Can't read table on hive created by spark-sql
Posted by "jini-lee (via GitHub)" <gi...@apache.org>.
jini-lee closed issue #7947: Can't read table on hive created by spark-sql
URL: https://github.com/apache/iceberg/issues/7947
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] jini-lee commented on issue #7947: Can't read table on hive created by spark-sql
Posted by "jini-lee (via GitHub)" <gi...@apache.org>.
jini-lee commented on issue #7947:
URL: https://github.com/apache/iceberg/issues/7947#issuecomment-1619956554
This issue has been resolved by adding the parameters below.
```
SET iceberg.catalog.test.type=hadoop;
SET iceberg.catalog.test.warehouse=hdfs://my-hdfs/warehouse/db/test;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org