You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "bithw1 (via GitHub)" <gi...@apache.org> on 2024/09/26 10:14:07 UTC
[I] [SUPPORT]Looks like a bug with keygenerator class when inserting data to a table (created by flink sql) via spark sql cli [hudi]
bithw1 opened a new issue, #12013:
URL: https://github.com/apache/hudi/issues/12013
I am using Hudi 0.15.0 and Flink 1.17.1, following are the steps to reproduce the problem:
From the flink-sql cli: do the following sql statements
```
CREATE CATALOG hudi_catalog WITH (
'type' = 'hudi',
'mode' = 'hms',
'default-database' = 'default',
'hive.conf.dir' = '/home/hadoop/software/hive-3.1.3/conf',
'table.external' = 'true'
);
create database if not exists hudi_catalog.`default`;
use hudi_catalog.`default`;
CREATE TABLE test_hudi_flink_mor_2 (
a int PRIMARY KEY NOT ENFORCED,
b int,
c int
)
WITH (
'connector' = 'hudi',
'path' = '/tmp/test_hudi_flink_mor_2',
'table.type' = 'MERGE_ON_READ',
'hoodie.datasource.write.keygenerator.class' = 'org.apache.hudi.keygen.ComplexAvroKeyGenerator', --- Using ComplexAvroKeyGenerator
'hoodie.datasource.write.recordkey.field' = 'a',
'write.precombine.key'='b',
'hoodie.datasource.write.hive_style_partitioning' = 'true',
'hive_sync.conf.dir'='/home/hadoop/software/hive-3.1.3/conf'
);
insert into test_hudi_flink_mor_2 values (1,1,1),(2,2,2);
```
So far so good, then I run `insert into test_hudi_flink_mor_2 select 3,3, 3` on the spark-sql cli, an error occurred, key exception message is as follows:
```
org.apache.hudi.exception.HoodieException: Config conflict(key current value existing value):
KeyGenerator: org.apache.hudi.keygen.ComplexAvroKeyGenerator org.apache.hudi.keygen.NonpartitionedAvroKeyGenerator
at org.apache.hudi.HoodieWriterUtils$.validateTableConfig(HoodieWriterUtils.scala:229)
at org.apache.hudi.HoodieSparkSqlWriterInternal.writeInternal(HoodieSparkSqlWriter.scala:232)
at org.apache.hudi.HoodieSparkSqlWriterInternal.write(HoodieSparkSqlWriter.scala:187)
```
When I run `show create table test_hudi_flink_mor_2` on spark-sql cli, it shows that keygenerator is using ComplexAvroKeyGenerator,but when i look at the `/tmp/test_hudi_flink_mor_2/.hoodie/hoodie.properties`,
`hoodie.table.keygenerator.class=org.apache.hudi.keygen.NonpartitionedAvroKeyGenerator`, per the exception, it looks to me that there is a bug here...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT]problem when inserting data to a non-partitioned table created by flink sql via spark sql cli [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #12013:
URL: https://github.com/apache/hudi/issues/12013#issuecomment-2378209962
The non-partitioned key generator is right, did you specify the key generator on Spark side?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT]problem when inserting data to a non-partitioned table created by flink sql via spark sql cli [hudi]
Posted by "bithw1 (via GitHub)" <gi...@apache.org>.
bithw1 commented on issue #12013:
URL: https://github.com/apache/hudi/issues/12013#issuecomment-2378218107
> The non-partitioned key generator is right, did you specify the key generator on Spark side?
I don't think non-partitioned key generator is right here. When I am creating the hudi table using flink sql, I have explicitly specified the key generator as `ComplexAvroKeyGenerator`. But, it is saved in the hoodies.properties as `hoodie.table.keygenerator.class=org.apache.hudi.keygen.NonpartitionedAvroKeyGenerator`
From the spark side, I only insert one record `insert into test_hudi_flink_mor_2 select 3,3, 3`, and don't do any DDL .
What I am doing is to create an hudi table from flink sql and use it from spark sql
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT]problem when inserting data to a non-partitioned table created by flink sql via spark sql cli [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #12013:
URL: https://github.com/apache/hudi/issues/12013#issuecomment-2378221641
yeah, we do have some set up logic in `HoodieTableFactory` and `HoodieHiveCatalog`, can you dig a little bit why the non-partitioned key generator is set up regardless of the explicit set up? I guess it is located in `HoodieTableFactory`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT]problem when inserting data to a non-partitioned table created by flink sql via spark sql cli [hudi]
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #12013:
URL: https://github.com/apache/hudi/issues/12013#issuecomment-2410698868
@jonvex Any insights on this ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT]Looks like a bug with keygenerator class when inserting data to a table (created by flink sql) via spark sql cli [hudi]
Posted by "bithw1 (via GitHub)" <gi...@apache.org>.
bithw1 commented on issue #12013:
URL: https://github.com/apache/hudi/issues/12013#issuecomment-2376569659
btw, i tried a little more, when I create the hudi table as a partitioned table, then i am able to insert from spark sql cli,
the problem should be related with non-partitioned table.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org