You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2020/06/25 16:50:13 UTC
[GitHub] [hudi] RajasekarSribalan opened a new issue #1766: Hudi COW - Bulk Insert followed by Upsert via Spark streaming job
RajasekarSribalan opened a new issue #1766:
URL: https://github.com/apache/hudi/issues/1766
Hi,
I have a spark batch job which pulls data from Hive and do a bulk insert in Hudi table.This is for initial back fill.
Then I have a Spark streaming job, which reads data from Kafka and do an upsert on the same Hudi table.
Isssue is, when I query via Hive after every streaming commit(spark microbatch), the number of rows is getting doubled.
For example,
Number of rows after bulk insert in Hudi = 50,000
Number of rows after streaming commit @1 = 1,00,000
Number of rows after streaming commit @2 = 1,50,000
and so on...
When i query via Hive, it should always read the latest commit and give me the results. But it always reads entire parquet files
PLease help to understand why hive query is behaving very strange
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] RajasekarSribalan edited a comment on issue #1766: Hudi COW - Bulk Insert followed by Upsert via Spark streaming job
Posted by GitBox <gi...@apache.org>.
RajasekarSribalan edited a comment on issue #1766:
URL: https://github.com/apache/hudi/issues/1766#issuecomment-649994437
@vinothchandar @bvaradar could you please help!
And I am getting below error when querying from hive beeline.
1) In below scenario
set hive.input.format=org.apache.hudi.hadoop.hive.HoodieCombineHiveInputFormat;
Error: org.apache.hudi.hadoop.hive.HoodieCombineHiveInputFormat.pushProjectionsAndFilters(Lorg/apache/hadoop/mapred/JobConf;Ljava/lang/Class;Lorg/apache/hadoop/fs/Path;)V
2) In below scenario
set hive.input.format=org.apache.hudi.hadoop.HoodieParquetInputFormat;
Error :
2020-06-26 09:50:19,231 FATAL [IPC Server handler 4 on 36657] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Task: attempt_1592820090013_257670_m_000000_0 - exited : java.lang.IndexOutOfBoundsException: Index: 1, Size: 1
at java.util.ArrayList.rangeCheck(ArrayList.java:653)
at java.util.ArrayList.get(ArrayList.java:429)
at org.apache.hadoop.hive.ql.io.parquet.read.DataWritableReadSupport.getProjectedGroupFields(DataWritableReadSupport.java:116)
at org.apache.hadoop.hive.ql.io.parquet.read.DataWritableReadSupport.getSchemaByName(DataWritableReadSupport.java:176)
at org.apache.hadoop.hive.ql.io.parquet.read.DataWritableReadSupport.init(DataWritableReadSupport.java:242)
at org.apache.hadoop.hive.ql.io.parquet.read.ParquetRecordReaderWrapper.getSplit(ParquetRecordReaderWrapper.java:256)
at org.apache.hadoop.hive.ql.io.parquet.read.ParquetRecordReaderWrapper.<init>(ParquetRecordReaderWrapper.java:95)
at org.apache.hadoop.hive.ql.io.parquet.read.ParquetRecordReaderWrapper.<init>(ParquetRecordReaderWrapper.java:81)
at org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat.getRecordReader(MapredParquetInputFormat.java:72)
at org.apache.hudi.hadoop.HoodieParquetInputFormat.getRecordReader(HoodieParquetInputFormat.java:297)
at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.<init>(MapTask.java:169)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:438)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1917)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
2020-06-26 09:50:19,231 INFO [IPC Server handler 4 on 36657] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Diagnostics report from attempt_1592820090013_257670_m_000000_0: Error: java.lang.IndexOutOfBoundsException: Index: 1, Size: 1
at java.util.ArrayList.rangeCheck(ArrayList.java:653)
at java.util.ArrayList.get(ArrayList.java:429)
at org.apache.hadoop.hive.ql.io.parquet.read.DataWritableReadSupport.getProjectedGroupFields(DataWritableReadSupport.java:116)
at org.apache.hadoop.hive.ql.io.parquet.read.DataWritableReadSupport.getSchemaByName(DataWritableReadSupport.java:176)
at org.apache.hadoop.hive.ql.io.parquet.read.DataWritableReadSupport.init(DataWritableReadSupport.java:242)
at org.apache.hadoop.hive.ql.io.parquet.read.ParquetRecordReaderWrapper.getSplit(ParquetRecordReaderWrapper.java:256)
at org.apache.hadoop.hive.ql.io.parquet.read.ParquetRecordReaderWrapper.<init>(ParquetRecordReaderWrapper.java:95)
at org.apache.hadoop.hive.ql.io.parquet.read.ParquetRecordReaderWrapper.<init>(ParquetRecordReaderWrapper.java:81)
at org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat.getRecordReader(MapredParquetInputFormat.java:72)
at org.apache.hudi.hadoop.HoodieParquetInputFormat.getRecordReader(HoodieParquetInputFormat.java:297)
at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.<init>(MapTask.java:169)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:438)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1917)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] RajasekarSribalan commented on issue #1766: [SUPPORT] Hudi COW - Bulk Insert followed by Upsert via Spark streaming job
Posted by GitBox <gi...@apache.org>.
RajasekarSribalan commented on issue #1766:
URL: https://github.com/apache/hudi/issues/1766#issuecomment-650487308
@vinothchandar Thanks for the quick response. Much appreciated.
Version details :
Hudi : 0.5.2
Hive version : hive-1.1.0+cdh5.12.2+1218
I get these error consistently and even for non-Hudi hive tables I get these errors.
I have the list of parquets correctly in the DFS and able to read via spark.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] vinothchandar commented on issue #1766: [SUPPORT] Hudi COW - Bulk Insert followed by Upsert via Spark streaming job
Posted by GitBox <gi...@apache.org>.
vinothchandar commented on issue #1766:
URL: https://github.com/apache/hudi/issues/1766#issuecomment-650472516
@RajasekarSribalan as you may have guessed the issue seems like the right input format not getting invoked. Hudi input formats filler for the latest parquet files after each commit. So when this is not happening query ends up reading all the files resulting in duplicates.
What’s the version of hive you are trying to use? First error with combine input format seems like a jar mismatch issue
The second exception does seem to be coming from actual reading? Ie it called hoodieParquetInputFormat.getSplits properly and errors out trying to read parquet
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] bvaradar commented on issue #1766: [SUPPORT] Hudi COW - Bulk Insert followed by Upsert via Spark streaming job
Posted by GitBox <gi...@apache.org>.
bvaradar commented on issue #1766:
URL: https://github.com/apache/hudi/issues/1766#issuecomment-668659435
@RajasekarSribalan : Can you turn on debug level logs and see if hudi input format is activated when reading the data.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] RajasekarSribalan commented on issue #1766: [SUPPORT] Hudi COW - Bulk Insert followed by Upsert via Spark streaming job
Posted by GitBox <gi...@apache.org>.
RajasekarSribalan commented on issue #1766:
URL: https://github.com/apache/hudi/issues/1766#issuecomment-651759274
@vinothchandar @bhasudha could you please assist?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] bvaradar closed issue #1766: [SUPPORT] Hudi COW - Bulk Insert followed by Upsert via Spark streaming job
Posted by GitBox <gi...@apache.org>.
bvaradar closed issue #1766:
URL: https://github.com/apache/hudi/issues/1766
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] RajasekarSribalan commented on issue #1766: [SUPPORT] Hudi COW - Bulk Insert followed by Upsert via Spark streaming job
Posted by GitBox <gi...@apache.org>.
RajasekarSribalan commented on issue #1766:
URL: https://github.com/apache/hudi/issues/1766#issuecomment-650712662
Thanks for your reply.
I tried this setting as well but I get duplicate records when querying hudi
table... ideally it has to pick up only latest commit but it fetches all
parquet and returns duplicate records.
On Sun, 28 Jun 2020, 12:42 pm Bhavani Sudha Saktheeswaran, <
notifications@github.com> wrote:
> It is strange you are seeing this for Hudi and non Hudi tables. Could you
> try setting this config when querying Hive
>
> set hive.input.format=org.apache.hadoop.hive.ql.io.HiveInputFormat and
> check once?
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/apache/hudi/issues/1766#issuecomment-650708997>, or
> unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AFMO6I42ZJHBA4T2WLQF6NTRY3UNPANCNFSM4OIRVWCA>
> .
>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] RajasekarSribalan commented on issue #1766: [SUPPORT] Hudi COW - Bulk Insert followed by Upsert via Spark streaming job
Posted by GitBox <gi...@apache.org>.
RajasekarSribalan commented on issue #1766:
URL: https://github.com/apache/hudi/issues/1766#issuecomment-652851038
@vinothchandar
PLs find the details
+-----------------------------+----------------------------------------------------+----------+--+
| col_name | data_type | comment |
+-----------------------------+----------------------------------------------------+----------+--+
| _hoodie_commit_time | string | |
| _hoodie_commit_seqno | string | |
| _hoodie_record_key | string | |
| _hoodie_partition_path | string | |
| _hoodie_file_name | string | |
| id | bigint | |
| dummy_id | bigint | |
| opened_at | string | |
| pending_since | string | |
| resolved_at | string | |
| closed_at | string | |
| first_assigned_at | string | |
inputFormat:org.apache.hudi.hadoop.HoodieParquetInputFormat, outputFormat:org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat, compressed:false, numBuckets:-1, serdeInfo:SerDeInfo(name:null, serializationLib:org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe, parameters:{serialization.format=1}), bucketCols:[], sortCols:[], parameters:{}, skewedInfo:SkewedInfo(skewedColNames:[], skewedColValues:[], skewedColValueLocationMaps:{}), storedAsSubDirectories:false), partitionKeys:[], parameters:{totalSize=165348936606, EXTERNAL=TRUE, last_commit_time_sync=20200702071300, numRows=-1, rawDataSize=-1, COLUMN_STATS_ACCURATE=false, numFiles=1683, transient_lastDdlTime=1593535291}, viewOriginalText:null, viewExpandedText:null, tableType:EXTERNAL_TABLE)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] RajasekarSribalan edited a comment on issue #1766: Hudi COW - Bulk Insert followed by Upsert via Spark streaming job
Posted by GitBox <gi...@apache.org>.
RajasekarSribalan edited a comment on issue #1766:
URL: https://github.com/apache/hudi/issues/1766#issuecomment-649994437
@vinothchandar @bvaradar could you please help!
And I am getting below error when querying from hive beeline.
set hive.input.format=org.apache.hudi.hadoop.hive.HoodieCombineHiveInputFormat;
Error: org.apache.hudi.hadoop.hive.HoodieCombineHiveInputFormat.pushProjectionsAndFilters(Lorg/apache/hadoop/mapred/JobConf;Ljava/lang/Class;Lorg/apache/hadoop/fs/Path;)V
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] RajasekarSribalan commented on issue #1766: [SUPPORT] Hudi COW - Bulk Insert followed by Upsert via Spark streaming job
Posted by GitBox <gi...@apache.org>.
RajasekarSribalan commented on issue #1766:
URL: https://github.com/apache/hudi/issues/1766#issuecomment-658789885
Hello Balaji,
I still do get incorrect entries when queried via hive. In the earlier
comment,I have mentioned the hive cdh version that we are using. I m really
not sure why it actually fetches duplicate rows.
Kind regards,
Rajasekar
On Wed, 15 Jul 2020, 7:25 pm Balaji Varadarajan, <no...@github.com>
wrote:
> @RajasekarSribalan <https://github.com/RajasekarSribalan> : Sorry for the
> delayed response. Is this still a problem ? I am not seeing any issue with
> the table definition and as you can see from the docker demo, hive queries
> after upserting does work fine :
> https://hudi.apache.org/docs/docker_demo.html
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/apache/hudi/issues/1766#issuecomment-658782603>, or
> unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AFMO6IZP3NEOTL6FQFLUIPLR3WYN5ANCNFSM4OIRVWCA>
> .
>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] bhasudha commented on issue #1766: [SUPPORT] Hudi COW - Bulk Insert followed by Upsert via Spark streaming job
Posted by GitBox <gi...@apache.org>.
bhasudha commented on issue #1766:
URL: https://github.com/apache/hudi/issues/1766#issuecomment-650708997
It is strange you are seeing this for Hudi and non Hudi tables. Could you try setting this config when querying Hive
```set hive.input.format=org.apache.hadoop.hive.ql.io.HiveInputFormat``` and check once?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] bvaradar commented on issue #1766: [SUPPORT] Hudi COW - Bulk Insert followed by Upsert via Spark streaming job
Posted by GitBox <gi...@apache.org>.
bvaradar commented on issue #1766:
URL: https://github.com/apache/hudi/issues/1766#issuecomment-658782603
@RajasekarSribalan : Sorry for the delayed response. Is this still a problem ? I am not seeing any issue with the table definition and as you can see from the docker demo, hive queries after upserting does work fine : https://hudi.apache.org/docs/docker_demo.html
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] vinothchandar commented on issue #1766: [SUPPORT] Hudi COW - Bulk Insert followed by Upsert via Spark streaming job
Posted by GitBox <gi...@apache.org>.
vinothchandar commented on issue #1766:
URL: https://github.com/apache/hudi/issues/1766#issuecomment-650472589
Are you getting this error consistently
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] RajasekarSribalan edited a comment on issue #1766: [SUPPORT] Hudi COW - Bulk Insert followed by Upsert via Spark streaming job
Posted by GitBox <gi...@apache.org>.
RajasekarSribalan edited a comment on issue #1766:
URL: https://github.com/apache/hudi/issues/1766#issuecomment-650712662
Thanks for your reply @bhasudha @vinothchandar
I tried this setting as well but I get duplicate records when querying hudi
tables... ideally it has to pick up only latest commit but it fetches all
parquet and returns duplicate records.
But for non-hudi tables, below setting works fine.
set hive.input.format=org.apache.hadoop.hive.ql.io.HiveInputFormat
On Sun, 28 Jun 2020, 12:42 pm Bhavani Sudha Saktheeswaran, <
notifications@github.com> wrote:
> It is strange you are seeing this for Hudi and non Hudi tables. Could you
> try setting this config when querying Hive
>
> set hive.input.format=org.apache.hadoop.hive.ql.io.HiveInputFormat and
> check once?
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/apache/hudi/issues/1766#issuecomment-650708997>, or
> unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AFMO6I42ZJHBA4T2WLQF6NTRY3UNPANCNFSM4OIRVWCA>
> .
>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] bvaradar commented on issue #1766: [SUPPORT] Hudi COW - Bulk Insert followed by Upsert via Spark streaming job
Posted by GitBox <gi...@apache.org>.
bvaradar commented on issue #1766:
URL: https://github.com/apache/hudi/issues/1766#issuecomment-668660075
Closing this ticket as we have the jira
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] RajasekarSribalan commented on issue #1766: Hudi COW - Bulk Insert followed by Upsert via Spark streaming job
Posted by GitBox <gi...@apache.org>.
RajasekarSribalan commented on issue #1766:
URL: https://github.com/apache/hudi/issues/1766#issuecomment-649994437
@vinothchandar @bvaradar could you please help!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org