You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/11/16 10:44:31 UTC
[GitHub] [hudi] punish-yh opened a new issue, #7221: [SUPPORT] Spark could not read Flink created table
punish-yh opened a new issue, #7221:
URL: https://github.com/apache/hudi/issues/7221
**Describe the problem you faced**
I used flink-sql create a hudi MOR table and the metadata store in Hive metastore.
Then I use Spark with Hive metastroe to read the data with Flink inserted, but i got 0 lines data, and no error&exception log
**To Reproduce**
Steps to reproduce the behavior:
1. Flink sql create table sql:
```
create catalog hive with(
'type'='hudi',
'mode'='hms',
'hive.conf.dir'='/xxx/hive-conf'
);
CREATE TABLE if not exists hive.db.table_name(
`id` BIGINT PRIMARY KEY NOT ENFORCED,
`eid` STRING
)WITH (
'connector' = 'hudi',
'path' = 'hdfs://nameservice/user/hudi/db/table_name',
'table.type' = 'MERGE_ON_READ',
'compaction.max_memory' = '512')
```
2. select mysql source data insert into hudi table (it‘s worked, i use flink sql check result, it's correct)
3. check table format in hive
```
hive> show create table table_name;
OK
CREATE TABLE `table_name`(
)
ROW FORMAT SERDE
'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe'
STORED AS INPUTFORMAT
'org.apache.hadoop.mapred.TextInputFormat'
OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.IgnoreKeyTextOutputFormat'
LOCATION
'hdfs://nameservice/user/hive/warehouse/db.db/table_name'
TBLPROPERTIES (
'flink.compaction.max_memory'='512',
'flink.connector'='hudi',
'flink.path'='hdfs://nameservice/user/hudi/db/table_name',
'flink.schema.0.data-type'='BIGINT NOT NULL',
'flink.schema.0.name'='id',
'flink.schema.1.data-type'='VARCHAR(2147483647)',
'flink.schema.1.name'='eid',
'flink.schema.2.data-type'='VARCHAR(2147483647)',
'flink.schema.2.name'='oid',
'flink.schema.primary-key.columns'='id',
'flink.schema.primary-key.name'='PK_3386',
'flink.table.type'='MERGE_ON_READ',
'transient_lastDdlTime'='1666582177')
```
4. spark read code:
```
SparkSession spark = SparkSession
.builder()
.config("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
.config("spark.sql.extensions", "org.apache.spark.sql.hudi.HoodieSparkSessionExtension")
.config("hive.input.format","org.apache.hudi.hadoop.hive.HoodieCombineHiveInputFormat")
.config("hive.metastore.uris", "thrift://xxxx:9083")
.config("spark.sql.warehouse.dir", "hdfs://nemaservice/user/hudi/db/")
.enableHiveSupport()
.getOrCreate();
spark.sql("show databases").show();
spark.sql("use db").show();
spark.sql("show tables").show();
spark.sql("desc db.table_name").show();
spark.sql("SELECT count(1) FROM db.table_name").show();
```
**Expected behavior**
I expected the spark code can count this table num.
but just only "show databases","use db","show tables" can normal work
I could get this table format and correct count
actualy output:
+--------+---------+-------+
|col_name|data_type|comment|
+--------+---------+-------+
+--------+---------+-------+
+--------+
|count(1)|
+--------+
| 0|
+--------+
**Environment Description**
* Hudi version : 0.11.1
* Spark version : 3.2.2
* Flink version：1.14.5
* Hive version : 2.1.1-cdh6.3.2
* Hadoop version : 3.0.0-cdh6.3.2
* Storage (HDFS/S3/GCS..) : hdfs
* Running on Docker? (yes/no) : no
**Additional context**
should i must config 'hive_sync.enable'='true' in flink job then can use spark correct read table ?
**Stacktrace**
flink and spark job execute success, no error , no exception
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] KnightChess commented on issue #7221: [SUPPORT] Spark could not read Flink created table
Posted by GitBox <gi...@apache.org>.
KnightChess commented on issue #7221:
URL: https://github.com/apache/hudi/issues/7221#issuecomment-1317056231
<img width="801" alt="image" src="https://user-images.githubusercontent.com/20125927/202198281-66f1fb08-2731-4888-b2b4-f0371536af01.png">
It doesn't look like the same table
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] punish-yh commented on issue #7221: [SUPPORT] Spark could not read Flink created table
Posted by GitBox <gi...@apache.org>.
punish-yh commented on issue #7221:
URL: https://github.com/apache/hudi/issues/7221#issuecomment-1317098424
> LOCATION
`LOCATION` path is empty, but `flink.path` is true and have file that hudi can readable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] punish-yh commented on issue #7221: [SUPPORT] Spark could not read Flink created table
Posted by GitBox <gi...@apache.org>.
punish-yh commented on issue #7221:
URL: https://github.com/apache/hudi/issues/7221#issuecomment-1356805439
> you can load data like this val df = spark.read.format("hudi").load("hdfs://nameservice/user/hudi/db/table_name")
thanks ,it can be wrok.
i guess in 0.11 version, flink some configuration write to hive metastore make spark couldn't use in some scene.
and i upgarde hudi version to 0.12.1 and use hudi catalog ,spark can read table with hive metastore
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope commented on issue #7221: [SUPPORT] Spark could not read Flink created table
Posted by GitBox <gi...@apache.org>.
codope commented on issue #7221:
URL: https://github.com/apache/hudi/issues/7221#issuecomment-1348723532
@punish-yh gentle ping to check the recommended solution.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] jiangjin-f commented on issue #7221: [SUPPORT] Spark could not read Flink created table
Posted by GitBox <gi...@apache.org>.
jiangjin-f commented on issue #7221:
URL: https://github.com/apache/hudi/issues/7221#issuecomment-1318140284
you can load data like this
val df = spark.read.format("hudi").load("hdfs://nameservice/user/hudi/db/table_name")
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] punish-yh closed issue #7221: [SUPPORT] Spark could not read Flink created table
Posted by GitBox <gi...@apache.org>.
punish-yh closed issue #7221: [SUPPORT] Spark could not read Flink created table
URL: https://github.com/apache/hudi/issues/7221
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] punish-yh commented on issue #7221: [SUPPORT] Spark could not read Flink created table
Posted by GitBox <gi...@apache.org>.
punish-yh commented on issue #7221:
URL: https://github.com/apache/hudi/issues/7221#issuecomment-1317095105
> @punish-yh set `hive_sync.enable` true, and it will sync meta to metastore in next checkpoint. It will create two tables, which name are `table_name_rt` and `table_name_ro`
thank you for your reply, is it mean if i want use spark to read flink inserted data, must use `hive_sync.enable` ? is there another way?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] KnightChess commented on issue #7221: [SUPPORT] Spark could not read Flink created table
Posted by GitBox <gi...@apache.org>.
KnightChess commented on issue #7221:
URL: https://github.com/apache/hudi/issues/7221#issuecomment-1317128443
@punish-yh you can use dataFrame to load data
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan commented on issue #7221: [SUPPORT] Spark could not read Flink created table
Posted by GitBox <gi...@apache.org>.
xushiyan commented on issue #7221:
URL: https://github.com/apache/hudi/issues/7221#issuecomment-1320924188
> you can load data like this val df = spark.read.format("hudi").load("hdfs://nameservice/user/hudi/db/table_name")
@punish-yh pls close if this works in your case
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org