You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2020/11/24 11:06:52 UTC
[GitHub] [hudi] santas-little-helper-13 opened a new issue #2277: [SUPPORT]
santas-little-helper-13 opened a new issue #2277:
URL: https://github.com/apache/hudi/issues/2277
Hi,
I am working with hudi in AWS Glue. I have a problem with hudi updates.
So I have one Glue job that inserts data into hudi parquet files, it reads data from glue table, does some processing, gets max ID_key from already existing data and adds it to the row number in order for Id_key to be unique on the whole table level.
Now I have the other Glue job in which I read from that hudi table:
`hudiDF = spark.read.format("hudi").load('s3://prct-parquet-tgt/test_task1' + "/*")`
limit it to just one record and make changes in one column and in column upd_ind which is precombine field (all records have 0 by default as upd_ind):
`updateDF = hudiDF.limit(1).withColumn('sequence', lit('new_value')).withColumn('upd_ind', lit(1))`
then I define hudi options:
```
hoodie_write_options = {
'hoodie.datasource.write.table.type': 'COPY_ON_WRITE',
'hoodie.parquet.compression.codec': 'snappy',
'hoodie.table.name': 'test_task1',
'hoodie.datasource.write.recordkey.field': 'ID_key',
'hoodie.datasource.write.hive_style_partitioning': True,
'hoodie.datasource.write.table.name': 'test_task1',
'hoodie.datasource.write.operation': 'upsert',
'hoodie.datasource.write.precombine.field': 'upd_ind',
'hoodie.datasource.write.insert.drop.duplicates': True,
'hoodie.datasource.write.partitionpath.field': "datehour",
'hoodie.upsert.shuffle.parallelism': 8,
'hoodie.insert.shuffle.parallelism': 8,
'hoodie.datasource.write.keygenerator.class': 'org.apache.hudi.keygen.ComplexKeyGenerator',
'hoodie.parquet.small.file.limit': 0
}
```
and write the updated row:
`updateDF.write.format('hudi').options(**hoodie_write_options).mode('append').save('s3://prct-parquet-tgt/test_task1')`
The problem is that the record that gets updated is random and has no connection to the record that is shown in Glue job.
If I define specific record, then update isn’t done at all:
`updateDF = hudiDF.filter(col('ID_key')==64777).withColumn('sequence', lit('new_value')).withColumn('upd_ind', lit(1))`
I need to update the exact record that I specify. Please help.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] bvaradar commented on issue #2277: [SUPPORT]
Posted by GitBox <gi...@apache.org>.
bvaradar commented on issue #2277:
URL: https://github.com/apache/hudi/issues/2277#issuecomment-752989110
closing this due to inactivity.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] bvaradar commented on issue #2277: [SUPPORT]
Posted by GitBox <gi...@apache.org>.
bvaradar commented on issue #2277:
URL: https://github.com/apache/hudi/issues/2277#issuecomment-733305873
@umehrot2 : Can you please take a look at this ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] umehrot2 commented on issue #2277: [SUPPORT]
Posted by GitBox <gi...@apache.org>.
umehrot2 commented on issue #2277:
URL: https://github.com/apache/hudi/issues/2277#issuecomment-737136631
This seems like you are not able to do a basic upsert. I doubt it has anything to do with the fact that you are using glue. Each record is uniquely recognized by **record key + partition key**. You may want to debug, by printing out the particular row you receive when you do:
```
updateDF = hudiDF.limit(1).withColumn('sequence', lit('new_value')).withColumn('upd_ind', lit(1))
```
OR
```
updateDF = hudiDF.filter(col('ID_key')==64777).withColumn('sequence', lit('new_value')).withColumn('upd_ind', lit(1))
```
Just verify once that it indeed does have the record key, partition key and pre-combine column in it. That particular record should get updated. I don't particularly see anything wrong that stands out based on your description.
Which version of Hudi are you using ?
A couple of things that stand out:
- You are using `org.apache.hudi.keygen.ComplexKeyGenerator` which just by looking at your example doesn't seem like you need because you have only one column for record key and one columns for partition key. So you may want to skip it and let it use `SimpleKeyGenerator` which is the default.
- Another thing to give a shot may be that once you have the `updateDF` drop the hoodie metadata columns from it i.e. that columns that begin with name `_hoodie`. I have sometimes noticed weird issues happening if the update data frame has metadata columns in it. Most likely this may not be the issue in your case, but I would like to rule it out before we proceed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] bvaradar closed issue #2277: [SUPPORT]
Posted by GitBox <gi...@apache.org>.
bvaradar closed issue #2277:
URL: https://github.com/apache/hudi/issues/2277
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org