You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "Amar1404 (via GitHub)" <gi...@apache.org> on 2024/09/05 05:33:26 UTC
[I] REPARTITION In Bloom Index Causing Slow Down [hudi]
Amar1404 opened a new issue, #11888:
URL: https://github.com/apache/hudi/issues/11888
**Describe the problem you faced**
I am using bloom index where will finding the file it is repartition and sorting seen in the spark dag view causing the problem of executor failure even I have increase memory per executor to 8GB for only 10GB of data.
**To Reproduce**
Steps to reproduce the behavior:
1. Create a partition path based on year month day and hour
2.Enable Bloom Index and using HudiDeltstreamer to incrementally load data from the above path
3. The size of data is 10GB in parquet partitioned between three hours.
4. Try writing the
**Expected behavior**
A clear and concise description of what you expected to happen.
**Environment Description**
* Hudi version : 0.12.3
* Spark version :3.3
* Hive version :
* Hadoop version :
* Storage (HDFS/S3/GCS..) : s3
* Running on Docker? (yes/no) :no
Num Executor= 15
Num Core Per Executor =6
Drive Memory = 6GB
Drive Core= 6
Memory per executor=8gb
**Additional context**
Add any other context about the problem here.
**Stacktrace**
```Add the stacktrace of the error.```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] REPARTITION In Bloom Index Causing Slow Down [hudi]
Posted by "Amar1404 (via GitHub)" <gi...@apache.org>.
Amar1404 commented on issue #11888:
URL: https://github.com/apache/hudi/issues/11888#issuecomment-2330674145
@ad1happy2go -
hoodie.datasource.write.recordkey.field=hudi_idempotence_key
hoodie.datasource.write.partitionpath.field=partion1:simple,partition2:simple,date_ts:SIMPLE
hoodie.datasource.write.keygenerator.class=org.apache.hudi.keygen.CustomKeyGenerator
hoodie.parquet.small.file.limit=134217728
hoodie.parquet.max.file.size=268435456
hoodie.cleaner.commits.retained=3
hoodie.deltastreamer.source.s3incr.spark.datasource.options={\"basePath\":\"s3://bucket/basepath\"}
hoodie.deltastreamer.source.hoodieincr.path=s3://bucket/incr_table_path
hoodie.deltastreamer.source.hoodieincr.read_latest_on_missing_ckpt=true
hoodie.datasource.write.hive_style_partitioning=true
hoodie.auto.adjust.lock.configs: true
hoodie.bootstrap.parallelism: 200
hoodie.bulk_insert.shuffle.parallelism: 100
hoodie.bulkinsert.shuffle.parallelism: 200
hoodie.clean.max.commits: 4
hoodie.cleaner.commits.retained: 3
hoodie.cleaner.policy.failed.writes: EAGER
hoodie.datasource.write.hive_style_partitioning: true
hoodie.embed.timeline.server: true
hoodie.enable.data.skipping: true
hoodie.filesystem.view.type: EMBEDDED_KV_STORE
hoodie.index.type: BLOOM
hoodie.insert.shuffle.parallelism: 100
hoodie.metadata.enable: true
hoodie.parquet.compression.codec: zstd
hoodie.parquet.max.file.size: 512000000
hoodie.parquet.small.file.limit: 268435456
hoodie.schema.on.read.enable: true
hoodie.upsert.shuffle.parallelism: 100
hoodie.write.concurrency.mode: single_writer
hoodie.write.lock.provider: org.apache.hudi.client.transaction.lock.ZookeeperBasedLockProvider
hoodie.write.lock.zookeeper.base_path: /hudi
hoodie.write.lock.zookeeper.port: 2181
hoodie.write.lock.zookeeper.url:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] REPARTITION In Bloom Index Causing Slow Down [hudi]
Posted by "Amar1404 (via GitHub)" <gi...@apache.org>.
Amar1404 commented on issue #11888:
URL: https://github.com/apache/hudi/issues/11888#issuecomment-2333188278
@ad1happy2go - Not possible that is lost, will try to regenerate.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] REPARTITION In Bloom Index Causing Slow Down [hudi]
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #11888:
URL: https://github.com/apache/hudi/issues/11888#issuecomment-2331616131
@Amar1404 Also please share the spark event logs or spark UI screenshot.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] REPARTITION In Bloom Index Causing Slow Down [hudi]
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #11888:
URL: https://github.com/apache/hudi/issues/11888#issuecomment-2330642195
@Amar1404 Can you please share the writer configuration.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] REPARTITION In Bloom Index Causing Slow Down [hudi]
Posted by "rangareddy (via GitHub)" <gi...@apache.org>.
rangareddy commented on issue #11888:
URL: https://github.com/apache/hudi/issues/11888#issuecomment-2646952759
Hi @Amar1404
Is there any update on this issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org