You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "LinMingQiang (via GitHub)" <gi...@apache.org> on 2023/04/03 13:52:02 UTC
[GitHub] [hudi] LinMingQiang opened a new issue, #8371: [SUPPORT] Flink cant read metafield '_hoodie_commit_time'
LinMingQiang opened a new issue, #8371:
URL: https://github.com/apache/hudi/issues/8371
**_Tips before filing an issue_**
CREATE TABLE ITTestMetaField(
_hoodie_commit_time STRING,
id STRING PRIMARY KEY NOT ENFORCED
)
WITH (
'index.type'='BUCKET',
'payload.class'='org.apache.hudi.common.model.PartialUpdateAvroPayload',
'precombine.field'='ts',
'table.type' = 'MERGE_ON_READ'
)
**Stacktrace**
```Caused by: java.lang.NullPointerException
at org.apache.flink.formats.parquet.vector.reader.AbstractColumnReader.readToVector(AbstractColumnReader.java:160)
at org.apache.hudi.table.format.cow.vector.reader.ParquetColumnarRowSplitReader.nextBatch(ParquetColumnarRowSplitReader.java:312)
at org.apache.hudi.table.format.cow.vector.reader.ParquetColumnarRowSplitReader.ensureBatch(ParquetColumnarRowSplitReader.java:288)
at org.apache.hudi.table.format.cow.vector.reader.ParquetColumnarRowSplitReader.reachedEnd(ParquetColumnarRowSplitReader.java:267)
at org.apache.hudi.table.format.ParquetSplitRecordIterator.hasNext(ParquetSplitRecordIterator.java:42)
at org.apache.hudi.table.format.mor.MergeOnReadInputFormat$BaseFileOnlyFilteringIterator.hasNext(MergeOnReadInputFormat.java:563)
at org.apache.hudi.table.format.mor.MergeOnReadInputFormat.reachedEnd(MergeOnReadInputFormat.java:264)
at org.apache.flink.streaming.api.functions.source.InputFormatSourceFunction.run(InputFormatSourceFunction.java:89)
at org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:110)
at org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:67)
at org.apache.flink.streaming.runtime.tasks.SourceStreamTask$LegacySourceFunctionThread.run(SourceStreamTask.java:332)
```
There's no problem querying other metafields
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on issue #8371: [SUPPORT] Flink cant read metafield '_hoodie_commit_time'
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #8371:
URL: https://github.com/apache/hudi/issues/8371#issuecomment-1495432965
Seems a bug, could you fire a PR and fix it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on issue #8371: [SUPPORT] Flink cant read metafield '_hoodie_commit_time'
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #8371:
URL: https://github.com/apache/hudi/issues/8371#issuecomment-1607145021
Did you declare the `_hoodie_commit_time` as a schema field in your table?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] LinMingQiang closed issue #8371: [SUPPORT] Flink cant read metafield '_hoodie_commit_time'
Posted by "LinMingQiang (via GitHub)" <gi...@apache.org>.
LinMingQiang closed issue #8371: [SUPPORT] Flink cant read metafield '_hoodie_commit_time'
URL: https://github.com/apache/hudi/issues/8371
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] LinMingQiang commented on issue #8371: [SUPPORT] Flink cant read metafield '_hoodie_commit_time'
Posted by "LinMingQiang (via GitHub)" <gi...@apache.org>.
LinMingQiang commented on issue #8371:
URL: https://github.com/apache/hudi/issues/8371#issuecomment-1497244412
```CREATE TABLE HUDI_6032(
id STRING PRIMARY KEY NOT ENFORCED,
name STRING,
age bigint,
ts string,
`par` STRING
) PARTITIONED BY (`par`)
WITH (
'index.type'='BUCKET',
'payload.class'='org.apache.hudi.common.model.PartialUpdateAvroPayload',
'precombine.field'='ts',
'changelog.enabled'='false',
'compaction.delta_commits'='1',
'compaction.async.enabled'='true',
'write.tasks'='1',
'hoodie.bucket.index.num.buckets'='1',
'compaction.schedule.enable'='true',
'table.type' = 'COPY_ON_WRITE',
'hoodie.datasource.write.hive_style_partitioning'='true',
'hive_sync.partition_extractor_class'='org.apache.hudi.hive.HiveStylePartitionValueExtractor',
'path' = 'file:///Users/hunter/workspace/hudipr/HUDI-6032/hudi-debug/hudi-debug-flink/target/HUDI_6032',
'connector' = 'hudi'
)```
```insert into HUDI_6032(id, name,age, ts, par) values('id1','name1',1, 'ts1','par1'),('id1','name2',2, 'ts2','par1')```
```
CREATE TABLE HUDI_6032(
_hoodie_commit_time STRING,
id STRING PRIMARY KEY NOT ENFORCED
) PARTITIONED BY (id)
WITH (
...
)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] LinMingQiang commented on issue #8371: [SUPPORT] Flink cant read metafield '_hoodie_commit_time'
Posted by "LinMingQiang (via GitHub)" <gi...@apache.org>.
LinMingQiang commented on issue #8371:
URL: https://github.com/apache/hudi/issues/8371#issuecomment-1494408958
<img width="1399" alt="截屏2023-04-03 22 14 53" src="https://user-images.githubusercontent.com/14815873/229536311-28350286-0e79-47f7-b8a6-d719e00919cd.png">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] Coco0201 commented on issue #8371: [SUPPORT] Flink cant read metafield '_hoodie_commit_time'
Posted by "Coco0201 (via GitHub)" <gi...@apache.org>.
Coco0201 commented on issue #8371:
URL: https://github.com/apache/hudi/issues/8371#issuecomment-1607190582
> Did you declare the `_hoodie_commit_time` as a schema field in your table?
Yes.My java codes as follows:
`tabEnv.executeSql(“create table cdc_hudi(_hoodie_commit_time string) with (...)”)`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] Coco0201 commented on issue #8371: [SUPPORT] Flink cant read metafield '_hoodie_commit_time'
Posted by "Coco0201 (via GitHub)" <gi...@apache.org>.
Coco0201 commented on issue #8371:
URL: https://github.com/apache/hudi/issues/8371#issuecomment-1608574627
> Did you declare the `_hoodie_commit_time` as a schema field in your table?
I found the comma which is in the DDL of my flink table was forgotten.So there is no problem while reading metafields.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] Coco0201 commented on issue #8371: [SUPPORT] Flink cant read metafield '_hoodie_commit_time'
Posted by "Coco0201 (via GitHub)" <gi...@apache.org>.
Coco0201 commented on issue #8371:
URL: https://github.com/apache/hudi/issues/8371#issuecomment-1607071315
现在可以读取_hoodie_commit_time了么?flink1.13读取hudi0.13.1还是报错Caused by: org.apache.flink.client.program.ProgramInvocationException: The main method caused an error: SQL parse failed. Encountered "`_hoodie_commit_time`" @LinMingQiang
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] LinMingQiang commented on issue #8371: [SUPPORT] Flink cant read metafield '_hoodie_commit_time'
Posted by "LinMingQiang (via GitHub)" <gi...@apache.org>.
LinMingQiang commented on issue #8371:
URL: https://github.com/apache/hudi/issues/8371#issuecomment-1494410630
<img width="1393" alt="截屏2023-04-03 22 00 36" src="https://user-images.githubusercontent.com/14815873/229536537-827e14bb-7ca4-43c0-b477-8f469a885c4c.png">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] voonhous commented on issue #8371: [SUPPORT] Flink cant read metafield '_hoodie_commit_time'
Posted by "voonhous (via GitHub)" <gi...@apache.org>.
voonhous commented on issue #8371:
URL: https://github.com/apache/hudi/issues/8371#issuecomment-1496970307
We've encountered similar issues around this code recently.
we can't seem to reproduce your issue, is it possible to provide a minimal example of your table so i can trigger this bug locally to see if the issue we are encountering is the same?
i.e. possible to share your hudi table file?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org