You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "abhisheksahani91 (via GitHub)" <gi...@apache.org> on 2023/07/13 12:54:33 UTC
[GitHub] [hudi] abhisheksahani91 opened a new issue, #9183: [SUPPORT] Glue 4.0 Hudi 0.12.1 PreCommit validator i.e SqlQueryEqualityPreCommitValidator is not working
abhisheksahani91 opened a new issue, #9183:
URL: https://github.com/apache/hudi/issues/9183
* Hudi version :0.12.1
* GLUE Version:4.0
**Describe the problem you faced**
I am trying to add a pre-commit validator for HudiDeltaStreamer running on AWS glue 4.0 and hudi 0.12.1 for a MOR table
Upon adding configs for the pre-commit validator, the pipeline initiates failure immediately, regardless of whether there are any incoming messages on Kafka. Even if a Kafka message is valid, the pipeline still encounters failure.
"--hoodie-conf", "hoodie.precommit.validators=org.apache.hudi.client.validator.SqlQueryEqualityPreCommitValidator",
"--hoodie-conf", "hoodie.precommit.validators.single.value.sql.queries=select count(*) from user_data_test_mor3 where updtdTm is null#0",
Stack Trace:
at org.apache.hudi.table.action.deltacommit.SparkUpsertDeltaCommitActionExecutor.execute(SparkUpsertDeltaCommitActionExecutor.java:46)
at org.apache.hudi.table.HoodieSparkMergeOnReadTable.upsert(HoodieSparkMergeOnReadTable.java:89)
at org.apache.hudi.table.HoodieSparkMergeOnReadTable.upsert(HoodieSparkMergeOnReadTable.java:76)
at org.apache.hudi.client.SparkRDDWriteClient.upsert(SparkRDDWriteClient.java:158)
at org.apache.hudi.utilities.deltastreamer.DeltaSync.writeToSink(DeltaSync.java:590)
at org.apache.hudi.utilities.deltastreamer.DeltaSync.syncOnce(DeltaSync.java:336)
at org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer$DeltaSyncService.lambda$startService$1(HoodieDeltaStreamer.java:704)
at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
Caused by: org.apache.hudi.exception.HoodieValidationException: At least one pre-commit validation failed
at org.apache.hudi.client.utils.SparkValidatorUtils.runValidators(SparkValidatorUtils.java:96)
at org.apache.hudi.table.action.commit.BaseSparkCommitActionExecutor.runPrecommitValidators(BaseSparkCommitActionExecutor.java:412)
at org.apache.hudi.table.action.commit.BaseCommitActionExecutor.commitOnAutoCommit(BaseCommitActionExecutor.java:172)
at org.apache.hudi.table.action.commit.BaseSparkCommitActionExecutor.updateIndexAndCommitIfNeeded(BaseSparkCommitActionExecutor.java:266)
at org.apache.hudi.table.action.commit.BaseSparkCommitActionExecutor.execute(BaseSparkCommitActionExecutor.java:174)
at org.apache.hudi.table.action.commit.BaseSparkCommitActionExecutor.execute(BaseSparkCommitActionExecutor.java:85)
at org.apache.hudi.table.action.commit.BaseWriteHelper.write(BaseWriteHelper.java:57)
... 11 more
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Glue 4.0 Hudi 0.12.1 PreCommit validator i.e SqlQueryEqualityPreCommitValidator is not working [hudi]
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #9183:
URL: https://github.com/apache/hudi/issues/9183#issuecomment-2289147763
Confirmed with @abhisheksahani91 on call. We are good to close this. Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] ad1happy2go commented on issue #9183: [SUPPORT] Glue 4.0 Hudi 0.12.1 PreCommit validator i.e SqlQueryEqualityPreCommitValidator is not working
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #9183:
URL: https://github.com/apache/hudi/issues/9183#issuecomment-1634363247
@abhisheksahani91 you should use SqlQuerySingleResultPreCommitValidator for this instead of SqlQueryEqualityPreCommitValidator
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] abhisheksahani91 commented on issue #9183: [SUPPORT] Glue 4.0 Hudi 0.12.1 PreCommit validator i.e SqlQueryEqualityPreCommitValidator is not working
Posted by "abhisheksahani91 (via GitHub)" <gi...@apache.org>.
abhisheksahani91 commented on issue #9183:
URL: https://github.com/apache/hudi/issues/9183#issuecomment-1634495622
Thanks @ad1happy2go
Let me try it and will confirm it here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] ad1happy2go commented on issue #9183: [SUPPORT] Glue 4.0 Hudi 0.12.1 PreCommit validator i.e SqlQueryEqualityPreCommitValidator is not working
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #9183:
URL: https://github.com/apache/hudi/issues/9183#issuecomment-1637927065
@abhisheksahani91 Please let us know if it worked.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] ad1happy2go commented on issue #9183: [SUPPORT] Glue 4.0 Hudi 0.12.1 PreCommit validator i.e SqlQueryEqualityPreCommitValidator is not working
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #9183:
URL: https://github.com/apache/hudi/issues/9183#issuecomment-1697107046
@abhisheksahani91 hoodie.deltastreamer.kafka.commit_on_errors doesn't apply to recommit validators. It's more related to committing Kafka offsets on errors.
Why you think so that your job should continue if precommit validation fails?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] soumilshah1995 commented on issue #9183: [SUPPORT] Glue 4.0 Hudi 0.12.1 PreCommit validator i.e SqlQueryEqualityPreCommitValidator is not working
Posted by "soumilshah1995 (via GitHub)" <gi...@apache.org>.
soumilshah1995 commented on issue #9183:
URL: https://github.com/apache/hudi/issues/9183#issuecomment-1636551169
Here is Video Tutorials
https://www.youtube.com/watch?v=KNzs9dj_Btc&t=73s
# Tested
```
try:
from pyspark.sql import SparkSession
import os
import sys
import uuid
from datetime import datetime
from faker import Faker
except Exception as e:
print("Error: ", e)
hudi_version = '0.13.1'
jar_file = 'hudi-spark3.3-bundle_2.12-0.14.0-SNAPSHOT.jar'
os.environ['PYSPARK_SUBMIT_ARGS'] = f"--jars {jar_file} pyspark-shell"
os.environ['PYSPARK_PYTHON'] = sys.executable
os.environ['PYSPARK_DRIVER_PYTHON'] = sys.executable
spark = SparkSession.builder \
.config('spark.serializer', 'org.apache.spark.serializer.KryoSerializer') \
.config('spark.jars', jar_file) \
.config('spark.sql.hive.convertMetastoreParquet', 'false') \
.getOrCreate()
db_name = "hudidb"
table_name = "pre_commit_validator"
recordkey = 'uuid'
precombine = 'precomb'
method = 'upsert'
table_type = "COPY_ON_WRITE"
validator_query = """SELECT COUNT(*) FROM <TABLE_NAME> WHERE message IS NULL;"""
path = f"file:///C:/tmp/{db_name}/{table_name}"
hudi_options = {
'hoodie.table.name': table_name,
'hoodie.datasource.write.recordkey.field': recordkey,
'hoodie.datasource.write.table.name': table_name,
'hoodie.datasource.write.operation': method,
'hoodie.datasource.write.precombine.field': precombine,
'hoodie.upsert.shuffle.parallelism': 2,
'hoodie.insert.shuffle.parallelism': 2,
"hoodie.precommit.validators": "org.apache.hudi.client.validator.SqlQueryEqualityPreCommitValidator",
"hoodie.precommit.validators.equality.sql.queries": validator_query
}
spark_df = spark.createDataFrame(data=[
(1, "This is APPEND 1", 111, "1"),
(2, "This is APPEND 2", 222, "2"), ],
schema=["uuid", "message", "precomb", "partition"])
spark_df.write.format("hudi").options(**hudi_options).mode("append").save(path)
spark.read.format("hudi").load(path).createOrReplaceTempView("hudi_snapshots")
spark.sql("select * from hudi_snapshots").show(truncate=False)
spark_df = spark.createDataFrame(
data=[
(4, None, 444, None),
(5, "This is APPEND 5", 555, "5"),
],
schema=["uuid", "message", "precomb", "partition"])
spark_df.show()
spark_df.write.format("hudi").options(**hudi_options).mode("append").save(path)
spark.read.format("hudi").load(path).createOrReplaceTempView("hudi_snapshots")
spark.sql("select * from hudi_snapshots").show(truncate=False)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Glue 4.0 Hudi 0.12.1 PreCommit validator i.e SqlQueryEqualityPreCommitValidator is not working [hudi]
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #9183:
URL: https://github.com/apache/hudi/issues/9183#issuecomment-1757756659
@abhisheksahani91 Do you have any more issues/doubts around this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] abhisheksahani91 commented on issue #9183: [SUPPORT] Glue 4.0 Hudi 0.12.1 PreCommit validator i.e SqlQueryEqualityPreCommitValidator is not working
Posted by "abhisheksahani91 (via GitHub)" <gi...@apache.org>.
abhisheksahani91 commented on issue #9183:
URL: https://github.com/apache/hudi/issues/9183#issuecomment-1653379103
@ad1happy2go I tried the config you pointed out validation is working correctly. But I am still facing an issue, I am trying to test that in case of any validation failure, the Kafka message for which validation failed should be ignored and the glue hudi job should continue running. But in my case, it is getting stopped.
Config used:
"--hoodie-conf", "hoodie.precommit.validators=org.apache.hudi.client.validator.SqlQuerySingleResultPreCommitValidator",
"--hoodie-conf", "hoodie.precommit.validators.single.value.sql.queries=select count(*) from <TABLE_NAME> where updtdTm is null#0",
"--hoodie-conf", "hoodie.deltastreamer.kafka.commit_on_errors=true",
"--continuous"
Stack Trace :
at org.apache.hudi.client.validator.SqlQuerySingleResultPreCommitValidator.validateUsingQuery(SqlQuerySingleResultPreCommitValidator.java:66) at org.apache.hudi.client.validator.SqlQueryPreCommitValidator.lambda$validateRecordsBeforeAndAfter$0(SqlQueryPreCommitValidator.java:70) at java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948) at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:647) at org.apache.hudi.client.validator.SqlQueryPreCommitValidator.validateRecordsBeforeAndAfter(SqlQueryPreCommitValidator.java:69) at org.apache.hudi.client.validator.SparkPreCommitValidator.validate(SparkPreCommitValidator.java:75) | at org.apache.hudi.client.validator.SqlQuerySingleResultPreCommitValidator.validateUsingQuery(SqlQuerySingleResultPreCommitValidator.java:66) at org.apache.hudi.client.validator.SqlQueryPreCommitValidator.lambda$validateRecordsBeforeAndAfter$0(SqlQueryPreCommitValidator.java:70) at java.util.Spliterators$ArraySpliterat
or.forEachRemaining(Spliterators.java:948) at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:647) at org.apache.hudi.client.validator.SqlQueryPreCommitValidator.validateRecordsBeforeAndAfter(SqlQueryPreCommitValidator.java:69) at org.apache.hudi.client.validator.SparkPreCommitValidator.validate(SparkPreCommitValidator.java:75)
-- | --
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Glue 4.0 Hudi 0.12.1 PreCommit validator i.e SqlQueryEqualityPreCommitValidator is not working [hudi]
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go closed issue #9183: [SUPPORT] Glue 4.0 Hudi 0.12.1 PreCommit validator i.e SqlQueryEqualityPreCommitValidator is not working
URL: https://github.com/apache/hudi/issues/9183
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org