You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "Gatsby-Lee (via GitHub)" <gi...@apache.org> on 2024/08/15 23:03:52 UTC
[I] [SUPPORT] Failed to use Bloom filter Indexing [hudi]
Gatsby-Lee opened a new issue, #11783:
URL: https://github.com/apache/hudi/issues/11783
**Describe the problem you faced**
* Hudi 0.14.1
* Enabled Metadata Table + Enabled Bloom filter Indexing
* When enabling "hoodie.bloom.index.use.metadata=true" to use the Bloom filter Indexing in Metadata Table, it started failing.
**To Reproduce**
Steps to reproduce the behavior:
1. Enable Metadata Table + Bloom Filter Indexing
2. Enable "hoodie.bloom.index.use.metadata=true"
**Expected behavior**
Once "hoodie.bloom.index.use.metadata=true" is enabled, it has to work with no issue
**Environment Description**
* Hudi version : 0.14.1-amazon-0 ( EMR on EKS )
* Spark version : Spark 3.5 ( EMR 7.1 )
**Additional context**
Add any other context about the problem here.
**Stacktrace**
```
2024-08-15 01:25:00
at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:101)
at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:508)
at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:651)
at org.apache.spark.serializer.KryoSerializationStream.writeObject(KryoSerializer.scala:279)
at org.apache.spark.broadcast.TorrentBroadcast$.$anonfun$blockifyObject$4(TorrentBroadcast.scala:365)
at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally(SparkErrorUtils.scala:64)
at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally$(SparkErrorUtils.scala:61)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:95)
at org.apache.spark.broadcast.TorrentBroadcast$.blockifyObject(TorrentBroadcast.scala:367)
at org.apache.spark.broadcast.TorrentBroadcast.writeBlocks(TorrentBroadcast.scala:161)
at org.apache.spark.broadcast.TorrentBroadcast.<init>(TorrentBroadcast.scala:99)
at org.apache.spark.broadcast.TorrentBroadcastFactory.newBroadcast(TorrentBroadcastFactory.scala:38)
at org.apache.spark.broadcast.BroadcastManager.newBroadcast(BroadcastManager.scala:78)
at org.apache.spark.SparkContext.broadcastInternal(SparkContext.scala:1670)
at org.apache.spark.SparkContext.broadcast(SparkContext.scala:1652)
at org.apache.spark.api.java.JavaSparkContext.broadcast(JavaSparkContext.scala:546)
at org.apache.hudi.index.bloom.SparkHoodieBloomIndexHelper.findMatchingFilesForRecordKeys(SparkHoodieBloomIndexHelper.java:113)
at org.apache.hudi.index.bloom.HoodieBloomIndex.lookupIndex(HoodieBloomIndex.java:135)
at org.apache.hudi.index.bloom.HoodieBloomIndex.tagLocation(HoodieBloomIndex.java:91)
at org.apache.hudi.table.action.commit.HoodieWriteHelper.tag(HoodieWriteHelper.java:59)
at org.apache.hudi.table.action.commit.HoodieWriteHelper.tag(HoodieWriteHelper.java:41)
at org.apache.hudi.table.action.commit.BaseWriteHelper.write(BaseWriteHelper.java:59)
... 95 more
Caused by: java.lang.IllegalArgumentException: Unable to create serializer "com.esotericsoftware.kryo.serializers.FieldSerializer" for class: java.util.concurrent.locks.ReentrantReadWriteLock
at com.esotericsoftware.kryo.factories.ReflectionSerializerFactory.makeSerializer(ReflectionSerializerFactory.java:65)
at com.esotericsoftware.kryo.factories.ReflectionSerializerFactory.makeSerializer(ReflectionSerializerFactory.java:43)
at com.esotericsoftware.kryo.Kryo.newDefaultSerializer(Kryo.java:396)
at com.twitter.chill.KryoBase.newDefaultSerializer(KryoBase.scala:62)
at com.esotericsoftware.kryo.Kryo.getDefaultSerializer(Kryo.java:380)
at com.esotericsoftware.kryo.util.DefaultClassResolver.registerImplicit(DefaultClassResolver.java:74)
at com.esotericsoftware.kryo.Kryo.getRegistration(Kryo.java:508)
at com.esotericsoftware.kryo.util.DefaultClassResolver.writeClass(DefaultClassResolver.java:97)
at com.esotericsoftware.kryo.Kryo.writeClass(Kryo.java:540)
at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:75)
... 116 more
Caused by: java.lang.reflect.InvocationTargetException
at jdk.internal.reflect.GeneratedConstructorAccessor112.newInstance(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499)
at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:480)
at com.esotericsoftware.kryo.factories.ReflectionSerializerFactory.makeSerializer(ReflectionSerializerFactory.java:51)
... 125 more
Caused by: java.lang.reflect.InaccessibleObjectException: Unable to make field private final java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock java.util.concurrent.locks.ReentrantReadWriteLock.readerLock accessible: module java.base does not "opens java.util.concurrent.locks" to unnamed module @75db5df9
at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:354)
at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:297)
at java.base/java.lang.reflect.Field.checkCanSetAccessible(Field.java:178)
at java.base/java.lang.reflect.Field.setAccessible(Field.java:172)
at com.esotericsoftware.kryo.serializers.FieldSerializer.buildValidFields(FieldSerializer.java:283)
at com.esotericsoftware.kryo.serializers.FieldSerializer.rebuildCachedFields(FieldSerializer.java:216)
at com.esotericsoftware.kryo.serializers.FieldSerializer.rebuildCachedFields(FieldSerializer.java:157)
at com.esotericsoftware.kryo.serializers.FieldSerializer.<init>(FieldSerializer.java:150)
at com.esotericsoftware.kryo.serializers.FieldSerializer.<init>(FieldSerializer.java:134)
at com.esotericsoftware.kryo.serializers.FieldSerializer.<init>(FieldSerializer.java:130)
... 130 more
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Failed to use Bloom filter Indexing [hudi]
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #11783:
URL: https://github.com/apache/hudi/issues/11783#issuecomment-2431200759
Confirmed by @Gatsby-Lee and closing the issue. Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Failed to use Bloom filter Indexing [hudi]
Posted by "Gatsby-Lee (via GitHub)" <gi...@apache.org>.
Gatsby-Lee commented on issue #11783:
URL: https://github.com/apache/hudi/issues/11783#issuecomment-2297558630
> EMR 7.1 uses Java 17 by default but older Java version should still exist there. I think you can try Java 8/11 in this case: https://docs.aws.amazon.com/emr/latest/ReleaseGuide/configuring-java8.html#configuring-java8-override-spark
Thank you. I will re-try after setting the java version overriding.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Failed to use Bloom filter Indexing [hudi]
Posted by "CTTY (via GitHub)" <gi...@apache.org>.
CTTY commented on issue #11783:
URL: https://github.com/apache/hudi/issues/11783#issuecomment-2297460657
EMR 7.1 uses Java 17 by default but older Java version should still exist there. I think you can try Java 8/11 in this case: https://docs.aws.amazon.com/emr/latest/ReleaseGuide/configuring-java8.html#configuring-java8-override-spark
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Failed to use Bloom filter Indexing [hudi]
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #11783:
URL: https://github.com/apache/hudi/issues/11783#issuecomment-2294836395
@Gatsby-Lee May be related to Java version. Which java version its using?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Failed to use Bloom filter Indexing [hudi]
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #11783:
URL: https://github.com/apache/hudi/issues/11783#issuecomment-2419157336
Sorry for the delay here @Gatsby-Lee . We will prioritise this. Any further update on this or are you still blocked here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Failed to use Bloom filter Indexing [hudi]
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #11783:
URL: https://github.com/apache/hudi/issues/11783#issuecomment-2419220387
@Gatsby-Lee I tried to use metadata table and `hoodie.bloom.index.use.metadata` and on emr-7.1.0 and didn't got any issues. Can you try below or share the code/configurations what you are using?
Code -
```
pyspark --jars /usr/lib/hudi/hudi-spark-bundle.jar --conf "spark.serializer=org.apache.spark.serializer.KryoSerializer" --conf "spark.sql.catalog.spark_catalog=org.apache.spark.sql.hudi.catalog.HoodieCatalog" --conf "spark.sql.extensions=org.apache.spark.sql.hudi.HoodieSparkSessionExtension"
```
```
schema = StructType(
[
StructField("id", IntegerType(), True),
StructField("name", StringType(), True)
]
)
data = [
Row(1, "a"),
Row(2, "a"),
Row(3, "c"),
]
hudi_configs = {
"hoodie.table.name": TABLE_NAME,
"hoodie.datasource.write.recordkey.field": "name",
"hoodie.datasource.write.precombine.field": "id",
"hoodie.datasource.write.operation":"insert_overwrite_table",
"hoodie.table.keygenerator.class": "org.apache.hudi.keygen.NonpartitionedKeyGenerator",
"hoodie.index.type" : "BLOOM",
"hoodie.metadata.index.bloom.filter.enable" : "true",
"hoodie.bloom.index.use.metadata" : "true"
}
df = spark.createDataFrame(spark.sparkContext.parallelize(data), schema)
df.write.format("org.apache.hudi").options(**hudi_configs).mode("overwrite").save(PATH)
spark.read.format("hudi").load(PATH).show()
for i in range(0,30):
df.write.format("org.apache.hudi").options(**hudi_configs).mode("append").save(PATH)
```
Also, Did you got a chance to try with OSS 0.15.0 version ?
OSS 0.14.1 doesn't officially support spark 3.5.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Failed to use Bloom filter Indexing [hudi]
Posted by "Gatsby-Lee (via GitHub)" <gi...@apache.org>.
Gatsby-Lee commented on issue #11783:
URL: https://github.com/apache/hudi/issues/11783#issuecomment-2303847637
> @Gatsby-Lee Please let us know if using Java 11/8 solves this issue, Thanks.
Thank you for following up.
I will reply back after I test it 👍
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Failed to use Bloom filter Indexing [hudi]
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go closed issue #11783: [SUPPORT] Failed to use Bloom filter Indexing
URL: https://github.com/apache/hudi/issues/11783
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Failed to use Bloom filter Indexing [hudi]
Posted by "Gatsby-Lee (via GitHub)" <gi...@apache.org>.
Gatsby-Lee commented on issue #11783:
URL: https://github.com/apache/hudi/issues/11783#issuecomment-2363232149
@ad1happy2go Do you have any updates on this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Failed to use Bloom filter Indexing [hudi]
Posted by "saadbelgi (via GitHub)" <gi...@apache.org>.
saadbelgi commented on issue #11783:
URL: https://github.com/apache/hudi/issues/11783#issuecomment-5277187533
> [@saadbelgi](https://github.com/saadbelgi) hello, I checked my previous doc, but I wasn't able to find any final conclusion. BTW, I intentionally disabled MDT feature since it doesn't help much for my use cases.
Thanks a lot for the reply! Have moved to record index for my use case, instead of using `hoodie.bloom.index.use.metadata=true` with `GLOBAL_BLOOM` index
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Failed to use Bloom filter Indexing [hudi]
Posted by "Gatsby-Lee (via GitHub)" <gi...@apache.org>.
Gatsby-Lee commented on issue #11783:
URL: https://github.com/apache/hudi/issues/11783#issuecomment-2296148272
> * EMR 7.1
checked here - https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-710-release.html
It's Java 17.
Checked Hudi 0.14.1 doesn't support Java 17
( https://hudi.apache.org/roadmap/#execution-engine-integration )
And, find the Jira task - https://issues.apache.org/jira/browse/HUDI-6506
I think you're right. It's Java version related issue.
FYI @CTTY
Would you happen to know how to fix this issue in EMR 7.1?
Maybe using Java 11?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Failed to use Bloom filter Indexing [hudi]
Posted by "Gatsby-Lee (via GitHub)" <gi...@apache.org>.
Gatsby-Lee commented on issue #11783:
URL: https://github.com/apache/hudi/issues/11783#issuecomment-5273473619
@saadbelgi
hello, I checked my previous doc, but I wasn't able to find any final conclusion.
BTW, I intentionally disabled MDT feature since it doesn't help much for my use cases.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Failed to use Bloom filter Indexing [hudi]
Posted by "Gatsby-Lee (via GitHub)" <gi...@apache.org>.
Gatsby-Lee commented on issue #11783:
URL: https://github.com/apache/hudi/issues/11783#issuecomment-2328331921
Finally completed testing and confirmed it doesn't work.
* EMR 7.2.0 Java17 ( EMR Hudi 0.14.1 ): FAILED
* EMR 7.2.0 Java11 ( EMR Hudi 0.14.1 ): FAILED
* EMR 7.2.0 Java8 ( EMR Hudi 0.14.1 ): FAILED
* EMR 7.15.0 Java8 by default ( OSS Hudi 0.14.1 ): FAILED
So, I can say it is currently broken.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Failed to use Bloom filter Indexing [hudi]
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #11783:
URL: https://github.com/apache/hudi/issues/11783#issuecomment-2303650478
@Gatsby-Lee Please let us know if using Java 11/8 solves this issue, Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Failed to use Bloom filter Indexing [hudi]
Posted by "saadbelgi (via GitHub)" <gi...@apache.org>.
saadbelgi commented on issue #11783:
URL: https://github.com/apache/hudi/issues/11783#issuecomment-5266273166
Hi @ad1happy2go @Gatsby-Lee, not sure if this is the right forum to raise this but facing the exact same issue with EMR 7.10.0 (Hudi 0.15 and Java 17) on setting hoodie.bloom.index.use.metadata to true. Without this config being enabled, I have been running Hudi jobs with this setup since a very long time, haven't faced any issues. Have also set:
--conf "spark.serializer=org.apache.spark.serializer.KryoSerializer" --conf "spark.sql.catalog.spark_catalog=org.apache.spark.sql.hudi.catalog.HoodieCatalog" --conf "spark.sql.extensions=org.apache.spark.sql.hudi.HoodieSparkSessionExtension"
Since this issue is closed and a resolution was found, is there anything specific that needs to be done to resolve this issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org