You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "Amar1404 (via GitHub)" <gi...@apache.org> on 2024/09/05 05:33:26 UTC

[I] REPARTITION In Bloom Index Causing Slow Down [hudi]

Amar1404 opened a new issue, #11888:
URL: https://github.com/apache/hudi/issues/11888

   
   **Describe the problem you faced**
   
   I am using bloom index where will finding the file it is repartition and sorting seen in the spark dag view causing the problem of executor failure even I have increase memory per executor to 8GB for only 10GB of data.
   
   **To Reproduce**
   
   Steps to reproduce the behavior:
   
   1. Create a partition path based on year month day and hour
   2.Enable Bloom Index and using HudiDeltstreamer to incrementally load data from the above path
   3. The size of data is 10GB in parquet partitioned between three hours.
   4. Try writing the 
   
   **Expected behavior**
   
   A clear and concise description of what you expected to happen.
   
   **Environment Description**
   
   * Hudi version : 0.12.3
   
   * Spark version :3.3
   
   * Hive version :
   
   * Hadoop version :
   
   * Storage (HDFS/S3/GCS..) : s3
   
   * Running on Docker? (yes/no) :no
   
   Num Executor= 15
   Num Core Per Executor =6
   Drive Memory = 6GB
   Drive Core= 6
   Memory per executor=8gb
   **Additional context**
   
   Add any other context about the problem here.
   
   **Stacktrace**
   
   ```Add the stacktrace of the error.```
   
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] REPARTITION In Bloom Index Causing Slow Down [hudi]

Posted by "Amar1404 (via GitHub)" <gi...@apache.org>.
Amar1404 commented on issue #11888:
URL: https://github.com/apache/hudi/issues/11888#issuecomment-2330674145

   @ad1happy2go -
   hoodie.datasource.write.recordkey.field=hudi_idempotence_key
   hoodie.datasource.write.partitionpath.field=partion1:simple,partition2:simple,date_ts:SIMPLE
   hoodie.datasource.write.keygenerator.class=org.apache.hudi.keygen.CustomKeyGenerator
   
   hoodie.parquet.small.file.limit=134217728
   hoodie.parquet.max.file.size=268435456
   hoodie.cleaner.commits.retained=3
   
   hoodie.deltastreamer.source.s3incr.spark.datasource.options={\"basePath\":\"s3://bucket/basepath\"} 
   
   hoodie.deltastreamer.source.hoodieincr.path=s3://bucket/incr_table_path
   
   
   hoodie.deltastreamer.source.hoodieincr.read_latest_on_missing_ckpt=true
   hoodie.datasource.write.hive_style_partitioning=true 
   hoodie.auto.adjust.lock.configs: true
   hoodie.bootstrap.parallelism: 200
   hoodie.bulk_insert.shuffle.parallelism: 100
   hoodie.bulkinsert.shuffle.parallelism: 200
   hoodie.clean.max.commits: 4
   hoodie.cleaner.commits.retained: 3
   hoodie.cleaner.policy.failed.writes: EAGER
   hoodie.datasource.write.hive_style_partitioning: true
   hoodie.embed.timeline.server: true
   hoodie.enable.data.skipping: true
   hoodie.filesystem.view.type: EMBEDDED_KV_STORE
   hoodie.index.type: BLOOM
   hoodie.insert.shuffle.parallelism: 100
   hoodie.metadata.enable: true
   hoodie.parquet.compression.codec: zstd
   hoodie.parquet.max.file.size: 512000000
   hoodie.parquet.small.file.limit: 268435456
   hoodie.schema.on.read.enable: true
   hoodie.upsert.shuffle.parallelism: 100
   hoodie.write.concurrency.mode: single_writer
   hoodie.write.lock.provider: org.apache.hudi.client.transaction.lock.ZookeeperBasedLockProvider
   hoodie.write.lock.zookeeper.base_path: /hudi
   hoodie.write.lock.zookeeper.port: 2181
   hoodie.write.lock.zookeeper.url: 
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] REPARTITION In Bloom Index Causing Slow Down [hudi]

Posted by "Amar1404 (via GitHub)" <gi...@apache.org>.
Amar1404 commented on issue #11888:
URL: https://github.com/apache/hudi/issues/11888#issuecomment-2333188278

   @ad1happy2go - Not possible that is lost, will try to regenerate.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] REPARTITION In Bloom Index Causing Slow Down [hudi]

Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #11888:
URL: https://github.com/apache/hudi/issues/11888#issuecomment-2331616131

   @Amar1404 Also please share the spark event logs or spark UI screenshot.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] REPARTITION In Bloom Index Causing Slow Down [hudi]

Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #11888:
URL: https://github.com/apache/hudi/issues/11888#issuecomment-2330642195

   @Amar1404 Can you please share the  writer configuration.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] REPARTITION In Bloom Index Causing Slow Down [hudi]

Posted by "rangareddy (via GitHub)" <gi...@apache.org>.
rangareddy commented on issue #11888:
URL: https://github.com/apache/hudi/issues/11888#issuecomment-2646952759

   Hi @Amar1404 
   
   Is there any update on this issue?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org