You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2020/06/15 07:58:22 UTC
[GitHub] [hudi] cocopc opened a new issue #1735: HoodieRealtimeRecordReader can only work on HoodieRealtimeFileSplit
cocopc opened a new issue #1735:
URL: https://github.com/apache/hudi/issues/1735
MOR table, query xxx_rt view with hive
SQL： select count(1) from xxx_rt ;
Exception:
`
Caused by: java.lang.IllegalArgumentException: HoodieRealtimeRecordReader can only work on HoodieRealtimeFileSplit and not with hdfs://bj-es-001:8020/tmp/hudi/mor/user/ce6d87ad-c7ae-4a1a-a0a5-253f568cbe32-0_0-111-78_20200615141804.parquet:0+435860
at com.google.common.base.Preconditions.checkArgument(Preconditions.java:88)
at org.apache.hudi.hadoop.realtime.HoodieParquetRealtimeInputFormat.getRecordReader(HoodieParquetRealtimeInputFormat.java:247)
at org.apache.hadoop.hive.ql.io.CombineHiveRecordReader.<init>(CombineHiveRecordReader.java:67)
... 16 more
`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] cocopc commented on issue #1735: [SUPPORT]HoodieRealtimeRecordReader can only work on HoodieRealtimeFileSplit
Posted by GitBox <gi...@apache.org>.
cocopc commented on issue #1735:
URL: https://github.com/apache/hudi/issues/1735#issuecomment-644481482
set hive.input.format but not work. xxx_ro table is ok, but xxx_rt not work
yarn logs
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] bhasudha closed issue #1735: [SUPPORT]HoodieRealtimeRecordReader can only work on HoodieRealtimeFileSplit
Posted by GitBox <gi...@apache.org>.
bhasudha closed issue #1735:
URL: https://github.com/apache/hudi/issues/1735
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] bhasudha commented on issue #1735: [SUPPORT]HoodieRealtimeRecordReader can only work on HoodieRealtimeFileSplit
Posted by GitBox <gi...@apache.org>.
bhasudha commented on issue #1735:
URL: https://github.com/apache/hudi/issues/1735#issuecomment-644437264
It looks like you are using the default CombinedInputFormat in your hive session. You may want to use HoodieCombineHiveInputFormat instead of CombineInputFormat? You can set it like this - ```set hive.input.format=org.apache.hudi.hadoop.hive.HoodieCombineHiveInputFormat```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] bhasudha commented on issue #1735: [SUPPORT]HoodieRealtimeRecordReader can only work on HoodieRealtimeFileSplit
Posted by GitBox <gi...@apache.org>.
bhasudha commented on issue #1735:
URL: https://github.com/apache/hudi/issues/1735#issuecomment-644960546
thats strange. Are you getting the same error message as before - " HoodieRealtimeRecordReader can only work on HoodieRealtimeFileSplit" or a different exception. The stack trace does not show specific errors. Can you try this way- https://hudi.apache.org/docs/docker_demo.html#step-4-a-run-hive-queries ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] cocopc commented on issue #1735: [SUPPORT]HoodieRealtimeRecordReader can only work on HoodieRealtimeFileSplit
Posted by GitBox <gi...@apache.org>.
cocopc commented on issue #1735:
URL: https://github.com/apache/hudi/issues/1735#issuecomment-645141721
`set hive.input.format=org.apache.hadoop.hive.ql.io.HiveInputFormat` sloved my problem
if set hive.input.format=org.apache.hudi.hadoop.hive.HoodieCombineHiveInputFormat
HoodieParquetRealtimeInputFormat getSplits method not called.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] cocopc edited a comment on issue #1735: [SUPPORT]HoodieRealtimeRecordReader can only work on HoodieRealtimeFileSplit
Posted by GitBox <gi...@apache.org>.
cocopc edited a comment on issue #1735:
URL: https://github.com/apache/hudi/issues/1735#issuecomment-645141721
@bhasudha
`set hive.input.format=org.apache.hadoop.hive.ql.io.HiveInputFormat` sloved my problem
if set hive.input.format=org.apache.hudi.hadoop.hive.HoodieCombineHiveInputFormat
HoodieParquetRealtimeInputFormat getSplits method not called.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] bhasudha commented on issue #1735: [SUPPORT]HoodieRealtimeRecordReader can only work on HoodieRealtimeFileSplit
Posted by GitBox <gi...@apache.org>.
bhasudha commented on issue #1735:
URL: https://github.com/apache/hudi/issues/1735#issuecomment-649511288
@cocopc Closing this since we triaged this to the above Jira issue - https://issues.apache.org/jira/browse/HUDI-1036. Lets continue there.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] bhasudha commented on issue #1735: [SUPPORT]HoodieRealtimeRecordReader can only work on HoodieRealtimeFileSplit
Posted by GitBox <gi...@apache.org>.
bhasudha commented on issue #1735:
URL: https://github.com/apache/hudi/issues/1735#issuecomment-647187754
Thanks @cocopc . The HoodieCombineHiveInputFormat should work. I created a Jira ticket here - https://issues.apache.org/jira/browse/HUDI-1036 to debug this further. @n3nash do you know about this?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] cocopc edited a comment on issue #1735: [SUPPORT]HoodieRealtimeRecordReader can only work on HoodieRealtimeFileSplit
Posted by GitBox <gi...@apache.org>.
cocopc edited a comment on issue #1735:
URL: https://github.com/apache/hudi/issues/1735#issuecomment-644481482
@bhasudha
set hive.input.format but not work. xxx_ro table is ok, but xxx_rt not work
yarn logs
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org