You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2021/07/21 21:47:51 UTC
[GitHub] [hudi] vinothchandar opened a new issue #3323: [SUPPORT] Trouble with Point in time, Incremental queries
vinothchandar opened a new issue #3323:
URL: https://github.com/apache/hudi/issues/3323
**Describe the problem you faced**
A clear and concise description of the problem.
**To Reproduce**
Steps to reproduce the behavior:
```
from pyspark.sql import SparkSession
spark = (
SparkSession.builder.appName("Hudi_Data_Processing_Framework")
.config("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
.config("spark.sql.hive.convertMetastoreParquet", "false")
.config(
"spark.jars.packages",
"org.apache.hudi:hudi-spark3-bundle_2.12:0.8.0,org.apache.spark:spark-avro_2.12:3.0.2"
)
.getOrCreate()
)
input_df = spark.createDataFrame(
[
(100, "2015-01-01", "2015-01-01T01:01:01.010101Z", 10),
(101, "2015-01-01", "2015-01-01T01:01:01.010101Z", 10),
(102, "2015-01-01", "2015-01-01T01:01:01.010101Z", 10),
(103, "2015-01-01", "2015-01-01T01:01:01.010101Z", 10),
(104, "2015-01-01", "2015-01-01T01:01:01.010101Z", 10),
(105, "2015-01-01", "2015-01-01T01:01:01.010101Z", 10),
],
("acctid", "date", "ts", "deposit"),
)
hudi_options = {'hoodie.table.name': 'compaction',
'hoodie.datasource.write.table.type': 'MERGE_ON_READ',
'hoodie.datasource.write.operation': 'upsert',
'hoodie.datasource.write.recordkey.field': 'acctid',
'hoodie.datasource.write.partitionpath.field': 'date',
'hoodie.datasource.write.precombine.field': 'ts',
'hoodie.datasource.write.hive_style_partitioning': 'true',
'hoodie.upsert.shuffle.parallelism': 2,
'hoodie.insert.shuffle.parallelism': 2,
'hoodie.delete.shuffle.parallelism': 2
}
# INSERT
(
input_df.write.format("org.apache.hudi")
.options(**hudi_options)
.mode("append")
.save("/tmp/hudi_incr1")
)
# UPDATE deposit to **11** for key 100
update_df = spark.createDataFrame(
[(100, "2015-01-01", "2015-01-01T11:01:01.000000Z", 11)], ("acctid", "date", "ts", "deposit"))
update_df.write.format("org.apache.hudi").options(**hudi_options).mode("append").save("/tmp/hudi_incr1")
# UPDATE deposit to **12** for key 100
update_df = spark.createDataFrame(
[(100, "2015-01-01", "2015-01-01T12:01:01.000000Z", 12)], ("acctid", "date", "ts", "deposit"))
update_df.write.format("org.apache.hudi").options(**hudi_options).mode("append").save("/tmp/hudi_incr1")
# UPDATE deposit to **13** for key 100
update_df = spark.createDataFrame(
[(100, "2015-01-01", "2015-01-01T13:01:01.000000Z", 13)], ("acctid", "date", "ts", "deposit"))
update_df.write.format("org.apache.hudi").options(**hudi_options).mode("append").save("/tmp/hudi_incr1")
output_df = spark.read.format("org.apache.hudi").load(
"/tmp/hudi_incr1/"
)
output_df.show()
first_commit = '20210721080013' # As per this particular run
output_df = (spark.read
.option("hoodie.datasource.query.type", "incremental")
.option("hoodie.datasource.read.begin.instanttime", first_commit)
# .option("hoodie.datasource.read.end.instanttime", last_commit)
.format("org.apache.hudi")
.load("/tmp/hudi_incr1/"))
output_df.show()
```
**Expected behavior**
A clear and concise description of what you expected to happen.
**Environment Description**
* Hudi version :
* Spark version :
* Hive version :
* Hadoop version :
* Storage (HDFS/S3/GCS..) :
* Running on Docker? (yes/no) :
**Additional context**
Add any other context about the problem here.
**Stacktrace**
```Add the stacktrace of the error.```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] fanxinglanyu commented on issue #3323: [SUPPORT] Trouble with Point in time, Incremental queries
Posted by GitBox <gi...@apache.org>.
fanxinglanyu commented on issue #3323:
URL: https://github.com/apache/hudi/issues/3323#issuecomment-889981072
I ran into the same problem of doing an update operation in the original database and then using Spark to do an incremental query and found that the query time was not the actual point in time. For example, the actual Parquet file is the result of the Hadoop command query（the actual:4fe43850-4be8-447f-827e-edfdba44adb
4-0_0-341-295_20210730142459.parquet）, but the incremental query result is
`java.io.FileNotFoundException: File does not exist: HDFS: / / HDP - jk - 020 / user / 1:8 hive/warehouse/be8 fe43850 test_increment_hudi9_mor 2021/07/30/4-4-447 - f - 827 - e - edfdba44adb _0-4-0 340-294_20210730142459.parquet`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #3323: [SUPPORT] Trouble with Point in time, Incremental queries
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #3323:
URL: https://github.com/apache/hudi/issues/3323#issuecomment-884638185
I delved deeper and found the root cause. Could be a bug in code. But surprised how come we have not encountered this so far.
When we construct the records back from disk (log blocks), we use [reflection to instantiate the payload](https://github.com/apache/hudi/blob/5a94b6bf54b18739da55ebde10adf93f133e3204/hudi-common/src/main/java/org/apache/hudi/common/util/SpillableMapUtils.java#L116). And we have two constructors with OverwriteWithLatestAvroPayload, one of them takes in ordering field value, while 2nd one does not and assumes natural ordering(sets 0 as preCombine value).
Hence when two records are merged, we see the discrepancy.
Incase you are wondering, how come snapshot read is giving us correct results, here is the reason.
Snapshot reads log blocks in reverse and so the latest record always gets picked.
Where as w/ incremental, we read log blocks from start to end and so first record gets picked.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #3323: [SUPPORT] Trouble with Point in time, Incremental queries
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #3323:
URL: https://github.com/apache/hudi/issues/3323#issuecomment-885243661
hmmm, I get it. spark sql support started persisted preCombine field in table props and hence it worked for me.
I am not sure if felix tried. but I tried it locally and the fix worked.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan edited a comment on issue #3323: [SUPPORT] Trouble with Point in time, Incremental queries
Posted by GitBox <gi...@apache.org>.
nsivabalan edited a comment on issue #3323:
URL: https://github.com/apache/hudi/issues/3323#issuecomment-884638185
I delved deeper and found the root cause. Could be a bug in code. But surprised how come we have not encountered this so far.
When we construct the records back from disk (log blocks), we use [reflection to instantiate the payload](https://github.com/apache/hudi/blob/5a94b6bf54b18739da55ebde10adf93f133e3204/hudi-common/src/main/java/org/apache/hudi/common/util/SpillableMapUtils.java#L116). And we have two constructors with OverwriteWithLatestAvroPayload, one of them takes in ordering field value, while 2nd one does not and assumes natural ordering(sets 0 as preCombine value).
Hence when two records are merged, we see the discrepancy.
Incase you are wondering, how come snapshot read is giving us correct results, here is the reason.
Snapshot read, reads/merges log blocks in reverse and so the latest record always gets picked.
Where as w/ incremental, we read/merge log blocks from start to end and so first record gets picked.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan closed issue #3323: [SUPPORT] Trouble with Point in time, Incremental queries
Posted by GitBox <gi...@apache.org>.
xushiyan closed issue #3323:
URL: https://github.com/apache/hudi/issues/3323
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] FelixKJose commented on issue #3323: [SUPPORT] Trouble with Point in time, Incremental queries
Posted by GitBox <gi...@apache.org>.
FelixKJose commented on issue #3323:
URL: https://github.com/apache/hudi/issues/3323#issuecomment-885249002
@vinothchandar @nsivabalan The later commit times are higher actually in my example. So why do you think it was not getting the latest?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] vinothchandar commented on issue #3323: [SUPPORT] Trouble with Point in time, Incremental queries
Posted by GitBox <gi...@apache.org>.
vinothchandar commented on issue #3323:
URL: https://github.com/apache/hudi/issues/3323#issuecomment-884521964
@nsivabalan could you please take a look at this?
Felix reported this on slack.
I am really confused on incremental query on MOR table: From all the documentation I have seen we get the records that is changed from given begin time via an incremental query. But I am getting following behavior any insights?
schema: {id: int, value = int}
Inserted 10 records at time T1
2. Updated record with id =1 to value=2 at Time T2
3. Made another update to same record id=1 to value=3 at Time T3
4. Updates same record again with value=4 at time T4
5. Now at time T5, I am making an incremental query, with hoodie.datasource.read.begin.instanttime=T1
then I get record id=1 with value=2, is that expected?
How do I get the latest value value=4 via an incremental query?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] vinothchandar edited a comment on issue #3323: [SUPPORT] Trouble with Point in time, Incremental queries
Posted by GitBox <gi...@apache.org>.
vinothchandar edited a comment on issue #3323:
URL: https://github.com/apache/hudi/issues/3323#issuecomment-885225421
@nsivabalan Not sure. I think this code is assuming later commit times imply higher precombine field. Not sure if we actually persisted the precombine field in hoodie.properties in prior releases actually. the precombine field was mostly used as a way to de-duplicate input.
What's the status of this issue? does your PR solve Felix's issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #3323: [SUPPORT] Trouble with Point in time, Incremental queries
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #3323:
URL: https://github.com/apache/hudi/issues/3323#issuecomment-901493406
Hey folks. latest master has the fix. Can you give it a try and let us know if the issue is resolved. We can close it out if things are good.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan edited a comment on issue #3323: [SUPPORT] Trouble with Point in time, Incremental queries
Posted by GitBox <gi...@apache.org>.
nsivabalan edited a comment on issue #3323:
URL: https://github.com/apache/hudi/issues/3323#issuecomment-885243661
hmmm, I get it. spark sql support started persisting preCombine field in table props I guess and hence it worked for me.
I am not sure if felix tried. but I tried it locally and the fix worked.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] FelixKJose commented on issue #3323: [SUPPORT] Trouble with Point in time, Incremental queries
Posted by GitBox <gi...@apache.org>.
FelixKJose commented on issue #3323:
URL: https://github.com/apache/hudi/issues/3323#issuecomment-885762685
@nsivabalan is this only a problem in 0.8.0 version or older version as well?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] Guanpx commented on issue #3323: [SUPPORT] Trouble with Point in time, Incremental queries
Posted by GitBox <gi...@apache.org>.
Guanpx commented on issue #3323:
URL: https://github.com/apache/hudi/issues/3323#issuecomment-889817356
> @nsivabalan is this only a problem in 0.8.0 version or older version as well? Could you please provide some insights?
master also have this problem, maybe all version as well
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] FelixKJose commented on issue #3323: [SUPPORT] Trouble with Point in time, Incremental queries
Posted by GitBox <gi...@apache.org>.
FelixKJose commented on issue #3323:
URL: https://github.com/apache/hudi/issues/3323#issuecomment-884947681
Thank you @nsivabalan. I tried the 'ts' as string and then as timestamp, both has the same issue. And from you recent message it seems as an existing issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #3323: [SUPPORT] Trouble with Point in time, Incremental queries
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #3323:
URL: https://github.com/apache/hudi/issues/3323#issuecomment-885248141
Actually another patch which has this fix already https://github.com/apache/hudi/pull/3267 in review. We can sync up sometime on this to see how to go about it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] FelixKJose commented on issue #3323: [SUPPORT] Trouble with Point in time, Incremental queries
Posted by GitBox <gi...@apache.org>.
FelixKJose commented on issue #3323:
URL: https://github.com/apache/hudi/issues/3323#issuecomment-926279759
@vinothchandar Yes, I tested and is solved. Thank you so much
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] vinothchandar commented on issue #3323: [SUPPORT] Trouble with Point in time, Incremental queries
Posted by GitBox <gi...@apache.org>.
vinothchandar commented on issue #3323:
URL: https://github.com/apache/hudi/issues/3323#issuecomment-885225421
@nsivabalan Not sure. I think this code is assuming later commit times imply higher precombine field. Not sure if we actually persisted the precombine field in hoodie.properties in prior releases actually. the precombine field was mostly used as a way to de-duplicate input. We have not had at-least 2-3 issues where users also expect it to be respected against storage.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #3323: [SUPPORT] Trouble with Point in time, Incremental queries
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #3323:
URL: https://github.com/apache/hudi/issues/3323#issuecomment-884615732
I guess its bcoz, you don't have a higher preCombine value for new commits made. I see that all your preCombine is set to same value. Can you try it out and let me know.
my local trail:
https://gist.github.com/nsivabalan/d75f3f948c5f01a0b3454b986c3b9fe6
output:
```
scala> spark.sql("select rowId, preComb, value from hudi_trips_incremental").show(false)
+-----+-------+-----+
|rowId|preComb|value|
+-----+-------+-----+
|row_3|2 |2 |
|row_2|2 |2 |
|row_1|2 |2 |
|row_4|2 |2 |
|row_5|2 |2 |
|row_6|2 |2 |
|row_7|2 |2 |
|row_8|2 |2 |
|row_9|2 |2 |
+-----+-------+-----+
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #3323: [SUPPORT] Trouble with Point in time, Incremental queries
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #3323:
URL: https://github.com/apache/hudi/issues/3323#issuecomment-884616158
Hudi uses preCombine to merge records from multiple log blocks fyi, but uses combineAndGetUpdateValue() when merging base file w/ the merged records from log.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #3323: [SUPPORT] Trouble with Point in time, Incremental queries
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #3323:
URL: https://github.com/apache/hudi/issues/3323#issuecomment-884642954
This fix worked for me locally: https://github.com/apache/hudi/pull/3325
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #3323: [SUPPORT] Trouble with Point in time, Incremental queries
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #3323:
URL: https://github.com/apache/hudi/issues/3323#issuecomment-884618704
my bad. looks like you do have a high preCombine value. Just that the datatype is different from what I have tried.
may I know what was the datatype of "ts" column in hudi table ?
I am looking to reproduce w/ spark shell.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #3323: [SUPPORT] Trouble with Point in time, Incremental queries
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #3323:
URL: https://github.com/apache/hudi/issues/3323#issuecomment-884640235
@vinothchandar : Do you know the reason why it was designed this way?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] FelixKJose edited a comment on issue #3323: [SUPPORT] Trouble with Point in time, Incremental queries
Posted by GitBox <gi...@apache.org>.
FelixKJose edited a comment on issue #3323:
URL: https://github.com/apache/hudi/issues/3323#issuecomment-885249002
@vinothchandar @nsivabalan The later commit times (precombine values)are higher actually in my example. So why do you think it was not getting the latest?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] FelixKJose edited a comment on issue #3323: [SUPPORT] Trouble with Point in time, Incremental queries
Posted by GitBox <gi...@apache.org>.
FelixKJose edited a comment on issue #3323:
URL: https://github.com/apache/hudi/issues/3323#issuecomment-885762685
@nsivabalan is this only a problem in 0.8.0 version or older version as well? Could you please provide some insights?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] vinothchandar commented on issue #3323: [SUPPORT] Trouble with Point in time, Incremental queries
Posted by GitBox <gi...@apache.org>.
vinothchandar commented on issue #3323:
URL: https://github.com/apache/hudi/issues/3323#issuecomment-926272319
@FelixKJose this is in 0.9.0. Can we close this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #3323: [SUPPORT] Trouble with Point in time, Incremental queries
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #3323:
URL: https://github.com/apache/hudi/issues/3323#issuecomment-895556687
@FelixKJose : to my knowledge, it has been like from the beginning. Glad to say that we have a [PR fix](https://github.com/apache/hudi/pull/3401) on this and will be merging in this week. So, upcoming release should have it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org