You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "soumilshah1995 (via GitHub)" <gi...@apache.org> on 2023/11/23 00:50:28 UTC
[I] [SUPPORT] Async Clustering: Seeking Help on Specific Partitioning and Regex Pattern [hudi]
soumilshah1995 opened a new issue, #10165:
URL: https://github.com/apache/hudi/issues/10165
Subject : Async Clustering: Seeking Help on Specific Partitioning and Regex Pattern
I'm currently exploring async clustering in Apache Hudi, and this is also intended for a community video. I've successfully executed async clustering, but I have a question regarding how to cluster specific partitions or partitions based on a regex pattern.
After reviewing the documentation at https://hudi.apache.org/docs/next/clustering/, I came across the following setting
hoodie.clustering.plan.strategy.partition.selected N/A (Required) Comma-separated list of
I attempted to use this, but when I run the "show clustering" command, I observe the following result:
```
+-----------------+----------------+---------+-------------------+
| timestamp | input_group_size| state | involved_partitions|
+-----------------+----------------+---------+-------------------+
|20231122190057844| 1 | COMPLETED | * |
+-----------------+----------------+---------+-------------------+
```
As you can see, the involved partition shows '*', and I'm perplexed as to why it applied to all partitions rather than specific ones.
Here's my Spark submit command:
spark-submit \
--class org.apache.hudi.utilities.HoodieClusteringJob \
--packages org.apache.hudi:hudi-spark3.4-bundle_2.12:0.14.0 \
--properties-file spark-config.properties \
--master 'local[*]' \
--executor-memory 1g \
jar/hudi-utilities-slim-bundle_2.12-0.14.0.jar \
--mode scheduleAndExecute \
--base-path file:///Users/soumilnitinshah/Downloads/hudidb/silver/ \
--table-name orders \
--hoodie-conf hoodie.clustering.async.enabled=true \
--hoodie-conf hoodie.clustering.async.max.commits=2 \
--hoodie-conf hoodie.clustering.plan.strategy.target.file.max.bytes=1073741824 \
--hoodie-conf hoodie.clustering.plan.strategy.small.file.limit=629145600 \
--hoodie-conf hoodie.clustering.execution.strategy.class=org.apache.hudi.client.clustering.run.strategy.SparkSortAndSizeExecutionStrategy \
--hoodie-conf hoodie.clustering.plan.strategy.sort.columns=order_date \
--hoodie-conf hoodie.clustering.plan.strategy.partition.selected=2023-10-23 \
--hoodie-conf hoodie.write.concurrency.mode=optimistic_concurrency_control \
--hoodie-conf hoodie.write.lock.provider=org.apache.hudi.client.transaction.lock.InProcessLockProvider
Additionally, I'm curious about how to cluster partitions based on a regex pattern using the setting:
hoodie.clustering.plan.strategy.partition.regex.pattern
Is it correct to set the value to "2023-10-[0-9]" to match all partitions based on this pattern? I appreciate any insights or guidance on this matter.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Async Clustering: Seeking Help on Specific Partitioning and Regex Pattern [hudi]
Posted by "soumilshah1995 (via GitHub)" <gi...@apache.org>.
soumilshah1995 commented on issue #10165:
URL: https://github.com/apache/hudi/issues/10165#issuecomment-1827927924
<img width="285" alt="Screenshot 2023-11-27 at 9 07 05 AM" src="https://github.com/apache/hudi/assets/39345855/1b019a83-1924-441f-bf07-a1816aea2b93">
Ran show clustering
```
+---------+----------------+-----+-------------------+
|timestamp|input_group_size|state|involved_partitions|
+---------+----------------+-----+-------------------+
+---------+----------------+-----+-------------------+
```
perfect
now lets execute async clustering
```
# ====================================================================
# INgest data
# ====================================================================
spark-submit \
--class org.apache.hudi.utilities.streamer.HoodieStreamer \
--packages org.apache.hudi:hudi-spark3.4-bundle_2.12:0.14.0 \
--properties-file spark-config.properties \
--master 'local[*]' \
--executor-memory 1g \
jar/hudi-utilities-slim-bundle_2.12-0.14.0.jar \
--continuous \
--source-limit 4000000 \
--min-sync-interval-seconds 20 \
--table-type COPY_ON_WRITE \
--op UPSERT \
--source-ordering-field ts \
--source-class org.apache.hudi.utilities.sources.CsvDFSSource \
--target-base-path file:///Users/soumilshah/IdeaProjects/SparkProject/apache-hudi-delta-streamer-labs/E5/hudidb/orders \
--target-table orders \
--transformer-class org.apache.hudi.utilities.transform.SqlQueryBasedTransformer \
--props hudi_tbl.props
spark-submit \
--class org.apache.hudi.utilities.HoodieClusteringJob \
--packages org.apache.hudi:hudi-spark3.4-bundle_2.12:0.14.0 \
--properties-file spark-config.properties \
--master 'local[*]' \
--executor-memory 1g \
jar/hudi-utilities-slim-bundle_2.12-0.14.0.jar \
--mode scheduleAndExecute \
--base-path file:///Users/soumilshah/IdeaProjects/SparkProject/apache-hudi-delta-streamer-labs/E5/hudidb/orders \
--table-name orders \
--hoodie-conf hoodie.clustering.async.enabled=true \
--hoodie-conf hoodie.clustering.async.max.commits=2 \
--hoodie-conf hoodie.clustering.plan.strategy.target.file.max.bytes=1073741824 \
--hoodie-conf hoodie.clustering.plan.strategy.small.file.limit=629145600 \
--hoodie-conf hoodie.clustering.execution.strategy.class=org.apache.hudi.client.clustering.run.strategy.SparkSortAndSizeExecutionStrategy \
--hoodie-conf hoodie.clustering.plan.strategy.sort.columns=order_date \
--hoodie-conf hoodie.clustering.plan.strategy.partition.regex.pattern="^2023-10-[0-9][0-9]$" \
--hoodie-conf hoodie.write.concurrency.mode=optimistic_concurrency_control \
--hoodie-conf hoodie.write.lock.provider=org.apache.hudi.client.transaction.lock.InProcessLockProvider
```
# Python file
```
try:
import os
import sys
import uuid
import pyspark
from pyspark.sql import SparkSession
from pyspark import SparkConf, SparkContext
from faker import Faker
print("Imports loaded ")
except Exception as e:
print("error",e)
HUDI_VERSION = '0.14.0'
SPARK_VERSION = '3.4'
SUBMIT_ARGS = f"--packages org.apache.hudi:hudi-spark{SPARK_VERSION}-bundle_2.12:{HUDI_VERSION} pyspark-shell"
os.environ["PYSPARK_SUBMIT_ARGS"] = SUBMIT_ARGS
os.environ['PYSPARK_PYTHON'] = sys.executable
spark = SparkSession.builder \
.config('spark.serializer', 'org.apache.spark.serializer.KryoSerializer') \
.config('spark.sql.extensions', 'org.apache.spark.sql.hudi.HoodieSparkSessionExtension') \
.config('className', 'org.apache.hudi') \
.config('spark.sql.hive.convertMetastoreParquet', 'false') \
.getOrCreate()
path = "file:///Users/soumilshah/IdeaProjects/SparkProject/apache-hudi-delta-streamer-labs/E5/hudidb/orders"
query = f"call show_clustering(path => '{path}');"
df = spark.sql(query)
print(df.show())
```
looks like clustering is not working am I missing something here ?
# Logs
```
23/11/27 09:19:47 INFO Javalin: Starting Javalin ...
23/11/27 09:19:47 INFO Javalin: You are running Javalin 4.6.7 (released October 24, 2022. Your Javalin version is 399 days old. Consider checking for a newer version.).
23/11/27 09:19:47 INFO Server: jetty-9.4.48.v20220622; built: 2022-06-21T20:42:25.880Z; git: 6b67c5719d1f4371b33655ff2d047d24e171e49a; jvm 11.0.21+0
23/11/27 09:19:47 INFO Server: Started @2057ms
23/11/27 09:19:47 INFO Javalin: Listening on http://localhost:54378/
23/11/27 09:19:47 INFO Javalin: Javalin started in 63ms \o/
23/11/27 09:19:47 INFO TimelineService: Starting Timeline server on port :54378
23/11/27 09:19:47 INFO EmbeddedTimelineService: Started embedded timeline server at soumils-mbp:54378
23/11/27 09:19:47 INFO BaseHoodieClient: Timeline Server already running. Not restarting the service
23/11/27 09:19:47 INFO TransactionManager: Transaction starting for Option{val=[==>20231127091947161__replacecommit__REQUESTED]} with latest completed transaction instant Optional.empty
23/11/27 09:19:47 INFO LockManager: LockProvider org.apache.hudi.client.transaction.lock.InProcessLockProvider
23/11/27 09:19:47 INFO InProcessLockProvider: Base Path file:///Users/soumilshah/IdeaProjects/SparkProject/apache-hudi-delta-streamer-labs/E5/hudidb/orders, Lock Instance java.util.concurrent.locks.ReentrantReadWriteLock@25e6c22a[Write locks = 0, Read locks = 0], Thread main, In-process lock state ACQUIRING
23/11/27 09:19:47 INFO InProcessLockProvider: Base Path file:///Users/soumilshah/IdeaProjects/SparkProject/apache-hudi-delta-streamer-labs/E5/hudidb/orders, Lock Instance java.util.concurrent.locks.ReentrantReadWriteLock@25e6c22a[Write locks = 1, Read locks = 0], Thread main, In-process lock state ACQUIRED
23/11/27 09:19:47 INFO TransactionManager: Transaction started for Option{val=[==>20231127091947161__replacecommit__REQUESTED]} with latest completed transaction instant Optional.empty
23/11/27 09:19:47 INFO BaseHoodieWriteClient: Scheduling table service CLUSTER
23/11/27 09:19:47 INFO HoodieTableMetaClient: Loading HoodieTableMetaClient from file:///Users/soumilshah/IdeaProjects/SparkProject/apache-hudi-delta-streamer-labs/E5/hudidb/orders
23/11/27 09:19:47 INFO HoodieTableConfig: Loading table properties from file:/Users/soumilshah/IdeaProjects/SparkProject/apache-hudi-delta-streamer-labs/E5/hudidb/orders/.hoodie/hoodie.properties
23/11/27 09:19:47 INFO HoodieTableMetaClient: Finished Loading Table of type COPY_ON_WRITE(version=1, baseFileFormat=PARQUET) from file:///Users/soumilshah/IdeaProjects/SparkProject/apache-hudi-delta-streamer-labs/E5/hudidb/orders
23/11/27 09:19:47 INFO HoodieTableMetaClient: Loading Active commit timeline for file:///Users/soumilshah/IdeaProjects/SparkProject/apache-hudi-delta-streamer-labs/E5/hudidb/orders
23/11/27 09:19:47 INFO HoodieActiveTimeline: Loaded instants upto : Option{val=[20231127091738886__clean__COMPLETED__20231127091739815]}
23/11/27 09:19:47 INFO HoodieTableMetaClient: Loading HoodieTableMetaClient from file:///Users/soumilshah/IdeaProjects/SparkProject/apache-hudi-delta-streamer-labs/E5/hudidb/orders
23/11/27 09:19:47 INFO HoodieTableConfig: Loading table properties from file:/Users/soumilshah/IdeaProjects/SparkProject/apache-hudi-delta-streamer-labs/E5/hudidb/orders/.hoodie/hoodie.properties
23/11/27 09:19:47 INFO HoodieTableMetaClient: Finished Loading Table of type COPY_ON_WRITE(version=1, baseFileFormat=PARQUET) from file:///Users/soumilshah/IdeaProjects/SparkProject/apache-hudi-delta-streamer-labs/E5/hudidb/orders
23/11/27 09:19:47 INFO HoodieTableMetaClient: Loading HoodieTableMetaClient from file:/Users/soumilshah/IdeaProjects/SparkProject/apache-hudi-delta-streamer-labs/E5/hudidb/orders/.hoodie/metadata
23/11/27 09:19:47 INFO HoodieTableConfig: Loading table properties from file:/Users/soumilshah/IdeaProjects/SparkProject/apache-hudi-delta-streamer-labs/E5/hudidb/orders/.hoodie/metadata/.hoodie/hoodie.properties
23/11/27 09:19:47 INFO HoodieTableMetaClient: Finished Loading Table of type MERGE_ON_READ(version=1, baseFileFormat=HFILE) from file:/Users/soumilshah/IdeaProjects/SparkProject/apache-hudi-delta-streamer-labs/E5/hudidb/orders/.hoodie/metadata
23/11/27 09:19:47 INFO HoodieActiveTimeline: Loaded instants upto : Option{val=[20231127091738886__deltacommit__COMPLETED__20231127091739769]}
23/11/27 09:19:47 INFO AbstractTableFileSystemView: Took 0 ms to read 0 instants, 0 replaced file groups
23/11/27 09:19:47 INFO ClusteringUtils: Found 0 files in pending clustering operations
23/11/27 09:19:47 INFO FileSystemViewManager: Creating View Manager with storage type :REMOTE_FIRST
23/11/27 09:19:47 INFO FileSystemViewManager: Creating remote first table view
23/11/27 09:19:47 INFO BaseHoodieWriteClient: Scheduling clustering at instant time :20231127091947161
23/11/27 09:19:47 INFO ClusteringPlanActionExecutor: Checking if clustering needs to be run on file:///Users/soumilshah/IdeaProjects/SparkProject/apache-hudi-delta-streamer-labs/E5/hudidb/orders
23/11/27 09:19:47 INFO ClusteringPlanActionExecutor: Generating clustering plan for table file:///Users/soumilshah/IdeaProjects/SparkProject/apache-hudi-delta-streamer-labs/E5/hudidb/orders
23/11/27 09:19:47 INFO PartitionAwareClusteringPlanStrategy: Scheduling clustering for file:/Users/soumilshah/IdeaProjects/SparkProject/apache-hudi-delta-streamer-labs/E5/hudidb/orders
23/11/27 09:19:47 INFO PartitionAwareClusteringPlanStrategy: Scheduling clustering partitionSelected: null
23/11/27 09:19:47 INFO HoodieTableMetaClient: Loading HoodieTableMetaClient from file:/Users/soumilshah/IdeaProjects/SparkProject/apache-hudi-delta-streamer-labs/E5/hudidb/orders
23/11/27 09:19:47 INFO HoodieTableConfig: Loading table properties from file:/Users/soumilshah/IdeaProjects/SparkProject/apache-hudi-delta-streamer-labs/E5/hudidb/orders/.hoodie/hoodie.properties
23/11/27 09:19:47 INFO HoodieTableMetaClient: Finished Loading Table of type COPY_ON_WRITE(version=1, baseFileFormat=PARQUET) from file:/Users/soumilshah/IdeaProjects/SparkProject/apache-hudi-delta-streamer-labs/E5/hudidb/orders
23/11/27 09:19:47 INFO HoodieTableMetaClient: Loading HoodieTableMetaClient from file:/Users/soumilshah/IdeaProjects/SparkProject/apache-hudi-delta-streamer-labs/E5/hudidb/orders/.hoodie/metadata
23/11/27 09:19:47 INFO HoodieTableConfig: Loading table properties from file:/Users/soumilshah/IdeaProjects/SparkProject/apache-hudi-delta-streamer-labs/E5/hudidb/orders/.hoodie/metadata/.hoodie/hoodie.properties
23/11/27 09:19:47 INFO HoodieTableMetaClient: Finished Loading Table of type MERGE_ON_READ(version=1, baseFileFormat=HFILE) from file:/Users/soumilshah/IdeaProjects/SparkProject/apache-hudi-delta-streamer-labs/E5/hudidb/orders/.hoodie/metadata
23/11/27 09:19:47 INFO HoodieActiveTimeline: Loaded instants upto : Option{val=[20231127091738886__deltacommit__COMPLETED__20231127091739769]}
23/11/27 09:19:47 INFO AbstractTableFileSystemView: Took 0 ms to read 0 instants, 0 replaced file groups
23/11/27 09:19:47 INFO ClusteringUtils: Found 0 files in pending clustering operations
23/11/27 09:19:47 INFO HoodieTableMetadataUtil: Loading latest merged file slices for metadata table partition files
23/11/27 09:19:47 INFO AbstractTableFileSystemView: Took 0 ms to read 0 instants, 0 replaced file groups
23/11/27 09:19:47 INFO ClusteringUtils: Found 0 files in pending clustering operations
23/11/27 09:19:47 INFO AbstractTableFileSystemView: Building file system view for partition (files)
23/11/27 09:19:47 INFO HoodieBackedTableMetadata: Opened metadata base file from file:/Users/soumilshah/IdeaProjects/SparkProject/apache-hudi-delta-streamer-labs/E5/hudidb/orders/.hoodie/metadata/files/files-0000-0_0-118-202_20231127091421558001.hfile at instant 20231127091421558001 in 3 ms
23/11/27 09:19:47 INFO HoodieActiveTimeline: Loaded instants upto : Option{val=[20231127091738886__clean__COMPLETED__20231127091739815]}
23/11/27 09:19:47 INFO HoodieTableMetaClient: Loading HoodieTableMetaClient from file:/Users/soumilshah/IdeaProjects/SparkProject/apache-hudi-delta-streamer-labs/E5/hudidb/orders/.hoodie/metadata
23/11/27 09:19:47 INFO HoodieTableConfig: Loading table properties from file:/Users/soumilshah/IdeaProjects/SparkProject/apache-hudi-delta-streamer-labs/E5/hudidb/orders/.hoodie/metadata/.hoodie/hoodie.properties
23/11/27 09:19:47 INFO HoodieTableMetaClient: Finished Loading Table of type MERGE_ON_READ(version=1, baseFileFormat=HFILE) from file:/Users/soumilshah/IdeaProjects/SparkProject/apache-hudi-delta-streamer-labs/E5/hudidb/orders/.hoodie/metadata
# WARNING: Unable to get Instrumentation. Dynamic Attach failed. You may add this JAR as -javaagent manually, or supply -Djdk.attach.allowAttachSelf
# WARNING: Unable to attach Serviceability Agent. Unable to attach even with module exceptions: [org.apache.hudi.org.openjdk.jol.vm.sa.SASupportException: Sense failed., org.apache.hudi.org.openjdk.jol.vm.sa.SASupportException: Sense failed., org.apache.hudi.org.openjdk.jol.vm.sa.SASupportException: Sense failed.]
23/11/27 09:19:47 INFO HoodieActiveTimeline: Loaded instants upto : Option{val=[20231127091738886__deltacommit__COMPLETED__20231127091739769]}
23/11/27 09:19:47 INFO AbstractHoodieLogRecordReader: Scanning log file HoodieLogFile{pathStr='file:/Users/soumilshah/IdeaProjects/SparkProject/apache-hudi-delta-streamer-labs/E5/hudidb/orders/.hoodie/metadata/files/.files-0000-0_20231127091421558001.log.1_0-150-249', fileLen=-1}
23/11/27 09:19:47 INFO AbstractHoodieLogRecordReader: Reading a data block from file file:/Users/soumilshah/IdeaProjects/SparkProject/apache-hudi-delta-streamer-labs/E5/hudidb/orders/.hoodie/metadata/files/.files-0000-0_20231127091421558001.log.1_0-150-249 at instant 20231127091441562
23/11/27 09:19:47 INFO HoodieLogFormatReader: Moving to the next reader for logfile HoodieLogFile{pathStr='file:/Users/soumilshah/IdeaProjects/SparkProject/apache-hudi-delta-streamer-labs/E5/hudidb/orders/.hoodie/metadata/files/.files-0000-0_20231127091421558001.log.2_0-187-305', fileLen=-1}
23/11/27 09:19:47 INFO AbstractHoodieLogRecordReader: Scanning log file HoodieLogFile{pathStr='file:/Users/soumilshah/IdeaProjects/SparkProject/apache-hudi-delta-streamer-labs/E5/hudidb/orders/.hoodie/metadata/files/.files-0000-0_20231127091421558001.log.2_0-187-305', fileLen=-1}
23/11/27 09:19:47 INFO AbstractHoodieLogRecordReader: Reading a data block from file file:/Users/soumilshah/IdeaProjects/SparkProject/apache-hudi-delta-streamer-labs/E5/hudidb/orders/.hoodie/metadata/files/.files-0000-0_20231127091421558001.log.2_0-187-305 at instant 20231127091501567
23/11/27 09:19:47 INFO HoodieLogFormatReader: Moving to the next reader for logfile HoodieLogFile{pathStr='file:/Users/soumilshah/IdeaProjects/SparkProject/apache-hudi-delta-streamer-labs/E5/hudidb/orders/.hoodie/metadata/files/.files-0000-0_20231127091421558001.log.3_0-33-75', fileLen=-1}
23/11/27 09:19:47 INFO AbstractHoodieLogRecordReader: Scanning log file HoodieLogFile{pathStr='file:/Users/soumilshah/IdeaProjects/SparkProject/apache-hudi-delta-streamer-labs/E5/hudidb/orders/.hoodie/metadata/files/.files-0000-0_20231127091421558001.log.3_0-33-75', fileLen=-1}
23/11/27 09:19:47 INFO AbstractHoodieLogRecordReader: Reading a data block from file file:/Users/soumilshah/IdeaProjects/SparkProject/apache-hudi-delta-streamer-labs/E5/hudidb/orders/.hoodie/metadata/files/.files-0000-0_20231127091421558001.log.3_0-33-75 at instant 20231127091656890
23/11/27 09:19:47 INFO HoodieLogFormatReader: Moving to the next reader for logfile HoodieLogFile{pathStr='file:/Users/soumilshah/IdeaProjects/SparkProject/apache-hudi-delta-streamer-labs/E5/hudidb/orders/.hoodie/metadata/files/.files-0000-0_20231127091421558001.log.4_0-44-90', fileLen=-1}
23/11/27 09:19:47 INFO AbstractHoodieLogRecordReader: Scanning log file HoodieLogFile{pathStr='file:/Users/soumilshah/IdeaProjects/SparkProject/apache-hudi-delta-streamer-labs/E5/hudidb/orders/.hoodie/metadata/files/.files-0000-0_20231127091421558001.log.4_0-44-90', fileLen=-1}
23/11/27 09:19:47 INFO AbstractHoodieLogRecordReader: Reading a data block from file file:/Users/soumilshah/IdeaProjects/SparkProject/apache-hudi-delta-streamer-labs/E5/hudidb/orders/.hoodie/metadata/files/.files-0000-0_20231127091421558001.log.4_0-44-90 at instant 20231127091703325
23/11/27 09:19:47 INFO HoodieLogFormatReader: Moving to the next reader for logfile HoodieLogFile{pathStr='file:/Users/soumilshah/IdeaProjects/SparkProject/apache-hudi-delta-streamer-labs/E5/hudidb/orders/.hoodie/metadata/files/.files-0000-0_20231127091421558001.log.5_0-80-192', fileLen=-1}
23/11/27 09:19:47 INFO AbstractHoodieLogRecordReader: Scanning log file HoodieLogFile{pathStr='file:/Users/soumilshah/IdeaProjects/SparkProject/apache-hudi-delta-streamer-labs/E5/hudidb/orders/.hoodie/metadata/files/.files-0000-0_20231127091421558001.log.5_0-80-192', fileLen=-1}
23/11/27 09:19:47 INFO AbstractHoodieLogRecordReader: Reading a data block from file file:/Users/soumilshah/IdeaProjects/SparkProject/apache-hudi-delta-streamer-labs/E5/hudidb/orders/.hoodie/metadata/files/.files-0000-0_20231127091421558001.log.5_0-80-192 at instant 20231127091716888
23/11/27 09:19:47 INFO HoodieLogFormatReader: Moving to the next reader for logfile HoodieLogFile{pathStr='file:/Users/soumilshah/IdeaProjects/SparkProject/apache-hudi-delta-streamer-labs/E5/hudidb/orders/.hoodie/metadata/files/.files-0000-0_20231127091421558001.log.6_0-91-207', fileLen=-1}
23/11/27 09:19:47 INFO AbstractHoodieLogRecordReader: Scanning log file HoodieLogFile{pathStr='file:/Users/soumilshah/IdeaProjects/SparkProject/apache-hudi-delta-streamer-labs/E5/hudidb/orders/.hoodie/metadata/files/.files-0000-0_20231127091421558001.log.6_0-91-207', fileLen=-1}
23/11/27 09:19:47 INFO AbstractHoodieLogRecordReader: Reading a data block from file file:/Users/soumilshah/IdeaProjects/SparkProject/apache-hudi-delta-streamer-labs/E5/hudidb/orders/.hoodie/metadata/files/.files-0000-0_20231127091421558001.log.6_0-91-207 at instant 20231127091718925
23/11/27 09:19:47 INFO HoodieLogFormatReader: Moving to the next reader for logfile HoodieLogFile{pathStr='file:/Users/soumilshah/IdeaProjects/SparkProject/apache-hudi-delta-streamer-labs/E5/hudidb/orders/.hoodie/metadata/files/.files-0000-0_20231127091421558001.log.7_0-127-309', fileLen=-1}
23/11/27 09:19:47 INFO AbstractHoodieLogRecordReader: Scanning log file HoodieLogFile{pathStr='file:/Users/soumilshah/IdeaProjects/SparkProject/apache-hudi-delta-streamer-labs/E5/hudidb/orders/.hoodie/metadata/files/.files-0000-0_20231127091421558001.log.7_0-127-309', fileLen=-1}
23/11/27 09:19:47 INFO AbstractHoodieLogRecordReader: Reading a data block from file file:/Users/soumilshah/IdeaProjects/SparkProject/apache-hudi-delta-streamer-labs/E5/hudidb/orders/.hoodie/metadata/files/.files-0000-0_20231127091421558001.log.7_0-127-309 at instant 20231127091736898
23/11/27 09:19:47 INFO HoodieLogFormatReader: Moving to the next reader for logfile HoodieLogFile{pathStr='file:/Users/soumilshah/IdeaProjects/SparkProject/apache-hudi-delta-streamer-labs/E5/hudidb/orders/.hoodie/metadata/files/.files-0000-0_20231127091421558001.log.8_0-138-324', fileLen=-1}
23/11/27 09:19:47 INFO AbstractHoodieLogRecordReader: Scanning log file HoodieLogFile{pathStr='file:/Users/soumilshah/IdeaProjects/SparkProject/apache-hudi-delta-streamer-labs/E5/hudidb/orders/.hoodie/metadata/files/.files-0000-0_20231127091421558001.log.8_0-138-324', fileLen=-1}
23/11/27 09:19:47 INFO AbstractHoodieLogRecordReader: Reading a data block from file file:/Users/soumilshah/IdeaProjects/SparkProject/apache-hudi-delta-streamer-labs/E5/hudidb/orders/.hoodie/metadata/files/.files-0000-0_20231127091421558001.log.8_0-138-324 at instant 20231127091738886
23/11/27 09:19:47 INFO AbstractHoodieLogRecordReader: Merging the final data blocks
23/11/27 09:19:47 INFO AbstractHoodieLogRecordReader: Number of remaining logblocks to merge 8
23/11/27 09:19:47 WARN MetricsConfig: Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties
23/11/27 09:19:47 INFO MetricsSystemImpl: Scheduled Metric snapshot period at 10 second(s).
23/11/27 09:19:47 INFO MetricsSystemImpl: HBase metrics system started
23/11/27 09:19:47 INFO MetricRegistries: Loaded MetricRegistries class org.apache.hudi.org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl
23/11/27 09:19:47 INFO CodecPool: Got brand-new decompressor [.gz]
23/11/27 09:19:47 INFO CodecPool: Got brand-new decompressor [.gz]
23/11/27 09:19:47 INFO CodecPool: Got brand-new decompressor [.gz]
23/11/27 09:19:47 INFO CodecPool: Got brand-new decompressor [.gz]
23/11/27 09:19:47 INFO AbstractHoodieLogRecordReader: Number of remaining logblocks to merge 7
23/11/27 09:19:47 INFO CodecPool: Got brand-new decompressor [.gz]
23/11/27 09:19:47 INFO CodecPool: Got brand-new decompressor [.gz]
23/11/27 09:19:47 INFO CodecPool: Got brand-new decompressor [.gz]
23/11/27 09:19:47 INFO CodecPool: Got brand-new decompressor [.gz]
23/11/27 09:19:47 INFO AbstractHoodieLogRecordReader: Number of remaining logblocks to merge 6
23/11/27 09:19:47 INFO CodecPool: Got brand-new decompressor [.gz]
23/11/27 09:19:47 INFO CodecPool: Got brand-new decompressor [.gz]
23/11/27 09:19:47 INFO CodecPool: Got brand-new decompressor [.gz]
23/11/27 09:19:47 INFO CodecPool: Got brand-new decompressor [.gz]
23/11/27 09:19:47 INFO AbstractHoodieLogRecordReader: Number of remaining logblocks to merge 5
23/11/27 09:19:47 INFO CodecPool: Got brand-new decompressor [.gz]
23/11/27 09:19:47 INFO CodecPool: Got brand-new decompressor [.gz]
23/11/27 09:19:47 INFO CodecPool: Got brand-new decompressor [.gz]
23/11/27 09:19:47 INFO CodecPool: Got brand-new decompressor [.gz]
23/11/27 09:19:47 INFO AbstractHoodieLogRecordReader: Number of remaining logblocks to merge 4
23/11/27 09:19:47 INFO CodecPool: Got brand-new decompressor [.gz]
23/11/27 09:19:47 INFO CodecPool: Got brand-new decompressor [.gz]
23/11/27 09:19:47 INFO CodecPool: Got brand-new decompressor [.gz]
23/11/27 09:19:47 INFO CodecPool: Got brand-new decompressor [.gz]
23/11/27 09:19:47 INFO AbstractHoodieLogRecordReader: Number of remaining logblocks to merge 3
23/11/27 09:19:47 INFO CodecPool: Got brand-new decompressor [.gz]
23/11/27 09:19:47 INFO CodecPool: Got brand-new decompressor [.gz]
23/11/27 09:19:47 INFO CodecPool: Got brand-new decompressor [.gz]
23/11/27 09:19:47 INFO CodecPool: Got brand-new decompressor [.gz]
23/11/27 09:19:47 INFO AbstractHoodieLogRecordReader: Number of remaining logblocks to merge 2
23/11/27 09:19:47 INFO CodecPool: Got brand-new decompressor [.gz]
23/11/27 09:19:47 INFO CodecPool: Got brand-new decompressor [.gz]
23/11/27 09:19:47 INFO CodecPool: Got brand-new decompressor [.gz]
23/11/27 09:19:47 INFO CodecPool: Got brand-new decompressor [.gz]
23/11/27 09:19:47 INFO AbstractHoodieLogRecordReader: Number of remaining logblocks to merge 1
23/11/27 09:19:47 INFO CodecPool: Got brand-new decompressor [.gz]
23/11/27 09:19:47 INFO CodecPool: Got brand-new decompressor [.gz]
23/11/27 09:19:47 INFO CodecPool: Got brand-new decompressor [.gz]
23/11/27 09:19:47 INFO CodecPool: Got brand-new decompressor [.gz]
23/11/27 09:19:47 INFO HoodieMergedLogRecordScanner: Number of log files scanned => 8
23/11/27 09:19:47 INFO HoodieMergedLogRecordScanner: MaxMemoryInBytes allowed for compaction => 1073741824
23/11/27 09:19:47 INFO HoodieMergedLogRecordScanner: Number of entries in MemoryBasedMap in ExternalSpillableMap => 3
23/11/27 09:19:47 INFO HoodieMergedLogRecordScanner: Total size in bytes of MemoryBasedMap in ExternalSpillableMap => 1584
23/11/27 09:19:47 INFO HoodieMergedLogRecordScanner: Number of entries in DiskBasedMap in ExternalSpillableMap => 0
23/11/27 09:19:47 INFO HoodieMergedLogRecordScanner: Size of file spilled to disk => 0
23/11/27 09:19:47 INFO HoodieBackedTableMetadata: Opened 8 metadata log files (dataset instant=20231127091738886, metadata instant=20231127091738886) in 748 ms
23/11/27 09:19:47 INFO CodecPool: Got brand-new decompressor [.gz]
23/11/27 09:19:47 INFO CodecPool: Got brand-new decompressor [.gz]
23/11/27 09:19:47 INFO CodecPool: Got brand-new decompressor [.gz]
23/11/27 09:19:47 INFO CodecPool: Got brand-new decompressor [.gz]
23/11/27 09:19:47 INFO CodecPool: Got brand-new decompressor [.gz]
23/11/27 09:19:47 INFO CodecPool: Got brand-new decompressor [.gz]
23/11/27 09:19:47 INFO CodecPool: Got brand-new decompressor [.gz]
23/11/27 09:19:47 INFO BaseTableMetadata: Listed partitions from metadata: #partitions=2
23/11/27 09:19:47 INFO PartitionAwareClusteringPlanStrategy: Scheduling clustering partitionPaths: []
23/11/27 09:19:47 INFO TransactionManager: Transaction ending with transaction owner Option{val=[==>20231127091947161__replacecommit__REQUESTED]}
23/11/27 09:19:47 INFO InProcessLockProvider: Base Path file:///Users/soumilshah/IdeaProjects/SparkProject/apache-hudi-delta-streamer-labs/E5/hudidb/orders, Lock Instance java.util.concurrent.locks.ReentrantReadWriteLock@25e6c22a[Write locks = 1, Read locks = 0], Thread main, In-process lock state RELEASING
23/11/27 09:19:47 INFO InProcessLockProvider: Base Path file:///Users/soumilshah/IdeaProjects/SparkProject/apache-hudi-delta-streamer-labs/E5/hudidb/orders, Lock Instance java.util.concurrent.locks.ReentrantReadWriteLock@25e6c22a[Write locks = 0, Read locks = 0], Thread main, In-process lock state RELEASED
23/11/27 09:19:47 INFO InProcessLockProvider: Base Path file:///Users/soumilshah/IdeaProjects/SparkProject/apache-hudi-delta-streamer-labs/E5/hudidb/orders, Lock Instance java.util.concurrent.locks.ReentrantReadWriteLock@25e6c22a[Write locks = 0, Read locks = 0], Thread main, In-process lock state ALREADY_RELEASED
23/11/27 09:19:47 INFO LockManager: Released connection created for acquiring lock
23/11/27 09:19:47 INFO TransactionManager: Transaction ended with transaction owner Option{val=[==>20231127091947161__replacecommit__REQUESTED]}
23/11/27 09:19:47 INFO HoodieClusteringJob: Couldn't generate cluster plan
23/11/27 09:19:47 INFO BaseHoodieClient: Stopping Timeline service !!
23/11/27 09:19:47 INFO EmbeddedTimelineService: Closing Timeline server
23/11/27 09:19:47 INFO TimelineService: Closing Timeline Service
23/11/27 09:19:47 INFO Javalin: Stopping Javalin ...
23/11/27 09:19:47 INFO Javalin: Javalin has stopped
23/11/27 09:19:47 INFO TimelineService: Closed Timeline Service
23/11/27 09:19:47 INFO EmbeddedTimelineService: Closed Timeline server
23/11/27 09:19:47 INFO TransactionManager: Transaction manager closed
23/11/27 09:19:47 INFO TransactionManager: Transaction manager closed
23/11/27 09:19:47 ERROR HoodieClusteringJob: Clustering with basePath: file:///Users/soumilshah/IdeaProjects/SparkProject/apache-hudi-delta-streamer-labs/E5/hudidb/orders, tableName: orders, runningMode: scheduleAndExecute failed
23/11/27 09:19:47 INFO SparkContext: SparkContext is stopping with exitCode 0.
23/11/27 09:19:48 INFO SparkUI: Stopped Spark web UI at http://soumils-mbp:8090
23/11/27 09:19:48 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
23/11/27 09:19:48 INFO MemoryStore: MemoryStore cleared
23/11/27 09:19:48 INFO BlockManager: BlockManager stopped
23/11/27 09:19:48 INFO BlockManagerMaster: BlockManagerMaster stopped
23/11/27 09:19:48 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
23/11/27 09:19:48 INFO SparkContext: Successfully stopped SparkContext
23/11/27 09:19:52 INFO ShutdownHookManager: Shutdown hook called
23/11/27 09:19:52 INFO ShutdownHookManager: Deleting directory /private/var/folders/qq/s_1bjv516pn_mck29cwdwxnm0000gp/T/spark-d28f6914-927f-446a-81f6-d3a99bcefb32
23/11/27 09:19:52 INFO ShutdownHookManager: Deleting directory /private/var/folders/qq/s_1bjv516pn_mck29cwdwxnm0000gp/T/spark-e1be381e-501f-486a-a5ef-3ad6febbda30
soumilshah@Soumils-MBP E5 %
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Async Clustering: Seeking Help on Specific Partitioning and Regex Pattern [hudi]
Posted by "soumilshah1995 (via GitHub)" <gi...@apache.org>.
soumilshah1995 commented on issue #10165:
URL: https://github.com/apache/hudi/issues/10165#issuecomment-1828665873
I changed my partition to look like this
<img width="260" alt="Screenshot 2023-11-27 at 4 35 15 PM" src="https://github.com/apache/hudi/assets/39345855/b645db43-f074-4880-bb26-1952752e7683">
#### Hudi configs
```
hoodie.datasource.write.recordkey.field=order_id
hoodie.datasource.write.partitionpath.field=order_date
hoodie.streamer.source.dfs.root=file:///Users/soumilshah/IdeaProjects/SparkProject/apache-hudi-delta-streamer-labs/E5/sampledata/orders
hoodie.datasource.write.precombine.field=ts
hoodie.clustering.inline=false
hoodie.clustering.async.enabled=true
hoodie.write.concurrency.mode=optimistic_concurrency_control
hoodie.write.lock.provider=org.apache.hudi.client.transaction.lock.InProcessLockProvider
hoodie.deltastreamer.csv.header=true
hoodie.deltastreamer.csv.sep=\t
```
# Delta Streamer Jobs
```
spark-submit \
--class org.apache.hudi.utilities.streamer.HoodieStreamer \
--packages org.apache.hudi:hudi-spark3.4-bundle_2.12:0.14.0 \
--properties-file spark-config.properties \
--master 'local[*]' \
--executor-memory 1g \
jar/hudi-utilities-slim-bundle_2.12-0.14.0.jar \
--continuous \
--source-limit 4000000 \
--min-sync-interval-seconds 20 \
--table-type COPY_ON_WRITE \
--op UPSERT \
--source-ordering-field ts \
--source-class org.apache.hudi.utilities.sources.CsvDFSSource \
--target-base-path file:///Users/soumilshah/IdeaProjects/SparkProject/apache-hudi-delta-streamer-labs/E5/hudidb/orders \
--target-table orders \
--props hudi_tbl.props
```
# Async Clustering jobs
```
spark-submit \
--class org.apache.hudi.utilities.HoodieClusteringJob \
--packages org.apache.hudi:hudi-spark3.4-bundle_2.12:0.14.0 \
--properties-file spark-config.properties \
--master 'local[*]' \
--executor-memory 1g \
jar/hudi-utilities-slim-bundle_2.12-0.14.0.jar \
--mode scheduleAndExecute \
--base-path file:///Users/soumilshah/IdeaProjects/SparkProject/apache-hudi-delta-streamer-labs/E5/hudidb/orders \
--table-name orders \
--hoodie-conf hoodie.clustering.async.enabled=true \
--hoodie-conf hoodie.clustering.async.max.commits=2 \
--hoodie-conf hoodie.clustering.plan.strategy.target.file.max.bytes=1073741824 \
--hoodie-conf hoodie.clustering.plan.strategy.small.file.limit=629145600 \
--hoodie-conf hoodie.clustering.execution.strategy.class=org.apache.hudi.client.clustering.run.strategy.SparkSortAndSizeExecutionStrategy \
--hoodie-conf hoodie.clustering.plan.strategy.sort.columns=order_date \
--hoodie-conf hoodie.clustering.plan.strategy.partition.regex.pattern="^2023-10-[0-9][0-9]$" \
--hoodie-conf hoodie.write.concurrency.mode=optimistic_concurrency_control \
--hoodie-conf hoodie.write.lock.provider=org.apache.hudi.client.transaction.lock.InProcessLockProvider
```
# Python Job to see clustering
```
try:
import os
import sys
import uuid
import pyspark
from pyspark.sql import SparkSession
from pyspark import SparkConf, SparkContext
from faker import Faker
print("Imports loaded ")
except Exception as e:
print("error",e)
HUDI_VERSION = '0.14.0'
SPARK_VERSION = '3.4'
SUBMIT_ARGS = f"--packages org.apache.hudi:hudi-spark{SPARK_VERSION}-bundle_2.12:{HUDI_VERSION} pyspark-shell"
os.environ["PYSPARK_SUBMIT_ARGS"] = SUBMIT_ARGS
os.environ['PYSPARK_PYTHON'] = sys.executable
spark = SparkSession.builder \
.config('spark.serializer', 'org.apache.spark.serializer.KryoSerializer') \
.config('spark.sql.extensions', 'org.apache.spark.sql.hudi.HoodieSparkSessionExtension') \
.config('className', 'org.apache.hudi') \
.config('spark.sql.hive.convertMetastoreParquet', 'false') \
.getOrCreate()
path = "file:///Users/soumilshah/IdeaProjects/SparkProject/apache-hudi-delta-streamer-labs/E5/hudidb/orders"
spark.read.format("org.apache.hudi").load(path).createOrReplaceTempView("hudi_snapshot")
print(spark.sql("SELECT _hoodie_partition_path from hudi_snapshot").show(10))
query = f"call show_clustering(path => '{path}');"
df = spark.sql(query)
print(df.show())
```
# o.p
```
+----------------------+
|_hoodie_partition_path|
+----------------------+
| 2023-10-30|
| 2023-10-30|
| 2023-10-30|
| 2023-10-30|
| 2023-10-30|
| 2023-10-30|
| 2023-10-30|
| 2023-10-30|
| 2023-10-30|
| 2023-10-30|
+----------------------+
only showing top 10 rows
None
+-----------------+----------------+---------+-------------------+
| timestamp|input_group_size| state|involved_partitions|
+-----------------+----------------+---------+-------------------+
|20231127164220485| 4|COMPLETED| *|
|20231127164029948| 30|COMPLETED| *|
+-----------------+----------------+---------+------------------
```
why is involved_partitions showing. * ??
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Async Clustering: Seeking Help on Specific Partitioning and Regex Pattern [hudi]
Posted by "soumilshah1995 (via GitHub)" <gi...@apache.org>.
soumilshah1995 commented on issue #10165:
URL: https://github.com/apache/hudi/issues/10165#issuecomment-1828671661
Here is my replacement commits
<img width="583" alt="Screenshot 2023-11-27 at 4 47 03 PM" src="https://github.com/apache/hudi/assets/39345855/1d5d32ef-0e78-4b19-96e6-3e9dc450231f">
```
{
"partitionToWriteStats" : {
"2023-10-30" : [ {
"fileId" : "766df3e6-b73c-4f22-a74c-fd3d1af6c0f8-0",
"path" : "2023-10-30/766df3e6-b73c-4f22-a74c-fd3d1af6c0f8-0_2-5-10_20231127164220485.parquet",
"cdcStats" : null,
"prevCommit" : "null",
"numWrites" : 1187,
"numDeletes" : 0,
"numUpdateWrites" : 0,
"numInserts" : 1187,
"totalWriteBytes" : 538568,
"totalWriteErrors" : 0,
"tempPath" : null,
"partitionPath" : "2023-10-30",
"totalLogRecords" : 0,
"totalLogFilesCompacted" : 0,
"totalLogSizeCompacted" : 0,
"totalUpdatedRecordsCompacted" : 0,
"totalLogBlocks" : 0,
"totalCorruptLogBlock" : 0,
"totalRollbackBlocks" : 0,
"fileSizeInBytes" : 538568,
"minEventTime" : null,
"maxEventTime" : null,
"runtimeStats" : {
"totalScanTime" : 0,
"totalUpsertTime" : 0,
"totalCreateTime" : 322
}
} ],
"2023-10-31" : [ {
"fileId" : "c8b2f471-a874-493b-a62a-f1f9353fd428-0",
"path" : "2023-10-31/c8b2f471-a874-493b-a62a-f1f9353fd428-0_3-5-11_20231127164220485.parquet",
"cdcStats" : null,
"prevCommit" : "null",
"numWrites" : 1123,
"numDeletes" : 0,
"numUpdateWrites" : 0,
"numInserts" : 1123,
"totalWriteBytes" : 533196,
"totalWriteErrors" : 0,
"tempPath" : null,
"partitionPath" : "2023-10-31",
"totalLogRecords" : 0,
"totalLogFilesCompacted" : 0,
"totalLogSizeCompacted" : 0,
"totalUpdatedRecordsCompacted" : 0,
"totalLogBlocks" : 0,
"totalCorruptLogBlock" : 0,
"totalRollbackBlocks" : 0,
"fileSizeInBytes" : 533196,
"minEventTime" : null,
"maxEventTime" : null,
"runtimeStats" : {
"totalScanTime" : 0,
"totalUpsertTime" : 0,
"totalCreateTime" : 322
}
} ],
"2023-10-28" : [ {
"fileId" : "cf01176b-0272-49ee-8e84-f2e32c81a7ea-0",
"path" : "2023-10-28/cf01176b-0272-49ee-8e84-f2e32c81a7ea-0_0-5-8_20231127164220485.parquet",
"cdcStats" : null,
"prevCommit" : "null",
"numWrites" : 1200,
"numDeletes" : 0,
"numUpdateWrites" : 0,
"numInserts" : 1200,
"totalWriteBytes" : 538090,
"totalWriteErrors" : 0,
"tempPath" : null,
"partitionPath" : "2023-10-28",
"totalLogRecords" : 0,
"totalLogFilesCompacted" : 0,
"totalLogSizeCompacted" : 0,
"totalUpdatedRecordsCompacted" : 0,
"totalLogBlocks" : 0,
"totalCorruptLogBlock" : 0,
"totalRollbackBlocks" : 0,
"fileSizeInBytes" : 538090,
"minEventTime" : null,
"maxEventTime" : null,
"runtimeStats" : {
"totalScanTime" : 0,
"totalUpsertTime" : 0,
"totalCreateTime" : 322
}
} ],
"2023-10-29" : [ {
"fileId" : "288fe88d-f879-44fe-b47c-0e7e75bc3849-0",
"path" : "2023-10-29/288fe88d-f879-44fe-b47c-0e7e75bc3849-0_1-5-9_20231127164220485.parquet",
"cdcStats" : null,
"prevCommit" : "null",
"numWrites" : 1168,
"numDeletes" : 0,
"numUpdateWrites" : 0,
"numInserts" : 1168,
"totalWriteBytes" : 535539,
"totalWriteErrors" : 0,
"tempPath" : null,
"partitionPath" : "2023-10-29",
"totalLogRecords" : 0,
"totalLogFilesCompacted" : 0,
"totalLogSizeCompacted" : 0,
"totalUpdatedRecordsCompacted" : 0,
"totalLogBlocks" : 0,
"totalCorruptLogBlock" : 0,
"totalRollbackBlocks" : 0,
"fileSizeInBytes" : 535539,
"minEventTime" : null,
"maxEventTime" : null,
"runtimeStats" : {
"totalScanTime" : 0,
"totalUpsertTime" : 0,
"totalCreateTime" : 322
}
} ]
},
"compacted" : false,
"extraMetadata" : {
"schema" : "{\"type\":\"record\",\"name\":\"hoodie_source\",\"namespace\":\"hoodie.source\",\"fields\":[{\"name\":\"order_id\",\"type\":[\"null\",\"string\"],\"default\":null},{\"name\":\"name\",\"type\":[\"null\",\"string\"],\"default\":null},{\"name\":\"order_value\",\"type\":[\"null\",\"int\"],\"default\":null},{\"name\":\"priority\",\"type\":[\"null\",\"string\"],\"default\":null},{\"name\":\"order_date\",\"type\":[\"null\",{\"type\":\"int\",\"logicalType\":\"date\"}],\"default\":null},{\"name\":\"customer_id\",\"type\":[\"null\",\"string\"],\"default\":null},{\"name\":\"ts\",\"type\":[\"null\",\"double\"],\"default\":null}]}"
},
"operationType" : "CLUSTER",
"partitionToReplaceFileIds" : {
"2023-10-30" : [ "59772a5b-d4c3-43a2-a87e-13f8daf9ed83-0" ],
"2023-10-31" : [ "3a6f9bd0-a880-40d7-af56-521bb104f1d2-0" ],
"2023-10-28" : [ "0dfb7248-9430-40bd-a4a6-bf5d785e2735-0" ],
"2023-10-29" : [ "1b4ff837-5921-406b-ae1f-7d13885355cf-0" ]
}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Async Clustering: Seeking Help on Specific Partitioning and Regex Pattern [hudi]
Posted by "soumilshah1995 (via GitHub)" <gi...@apache.org>.
soumilshah1995 commented on issue #10165:
URL: https://github.com/apache/hudi/issues/10165#issuecomment-1828824587
@noahtaite you are right
I did set the flag and now I can see all partition that were clustered
```
involved_partitions
20231127164220485 4 COMPLETED 2023-10-28,2023-10-29,2023-10-30,2023-10-31
20231127164029948 30 COMPLETED 2023-10-28,2023-10-29,2023-10-30,2023-10-31,2023-11-01,2023-11-02,2023-11-03,2023-11-04,2023-11-05,2023-11-06,2023-11-07,2023-11-08,2023-11-09,2023-11-10,2023-11-11,2023-11-12,2023-11-13,2023-11-14,2023-11-15,2023-11-16,2023-11-17,2023-11-18,2023-11-19,2023-11-20,2023-11-21,2023-11-22,2023-11-23,2023-11-24,2023-11-25,2023-11-26
```
# Questions
When configuring properties for the Delta Streamer Async Clustering, how should I set the following property?
```
--hoodie-conf hoodie.clustering.plan.strategy.partition.regex.pattern="^2023-10-[0-9][0-9]$"
OR
--hoodie-conf hoodie.clustering.plan.strategy.partition.regex.pattern=^2023-10-[0-9][0-9]$
```
Is it necessary to enclose the property value in quotes? I'm concerned that without quotes, the Spark submit job might fail due to special characters.
Based on output looks like clustering was applied to all partition and regex pattern didn't work
# below are my config am I missing anything or proving param in wrong way ?
```
spark-submit \
--class org.apache.hudi.utilities.HoodieClusteringJob \
--packages org.apache.hudi:hudi-spark3.4-bundle_2.12:0.14.0 \
--properties-file spark-config.properties \
--master 'local[*]' \
--executor-memory 1g \
jar/hudi-utilities-slim-bundle_2.12-0.14.0.jar \
--mode scheduleAndExecute \
--base-path file:///Users/soumilshah/IdeaProjects/SparkProject/apache-hudi-delta-streamer-labs/E5/hudidb/orders \
--table-name orders \
--hoodie-conf hoodie.clustering.async.enabled=true \
--hoodie-conf hoodie.clustering.async.max.commits=2 \
--hoodie-conf hoodie.clustering.plan.strategy.target.file.max.bytes=1073741824 \
--hoodie-conf hoodie.clustering.plan.strategy.small.file.limit=629145600 \
--hoodie-conf hoodie.clustering.execution.strategy.class=org.apache.hudi.client.clustering.run.strategy.SparkSortAndSizeExecutionStrategy \
--hoodie-conf hoodie.clustering.plan.strategy.sort.columns=order_date \
--hoodie-conf hoodie.clustering.plan.strategy.partition.regex.pattern="^2023-10-[0-9][0-9]$" \
--hoodie-conf hoodie.write.concurrency.mode=optimistic_concurrency_control \
--hoodie-conf hoodie.write.lock.provider=org.apache.hudi.client.transaction.lock.InProcessLockProvider
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Async Clustering: Seeking Help on Specific Partitioning and Regex Pattern [hudi]
Posted by "soumilshah1995 (via GitHub)" <gi...@apache.org>.
soumilshah1995 commented on issue #10165:
URL: https://github.com/apache/hudi/issues/10165#issuecomment-1828669324
if i am missing any config please let me know happy to learn from experts here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Async Clustering: Seeking Help on Specific Partitioning and Regex Pattern [hudi]
Posted by "subash-metica (via GitHub)" <gi...@apache.org>.
subash-metica commented on issue #10165:
URL: https://github.com/apache/hudi/issues/10165#issuecomment-1828598815
> The partition paths in your example, does it contain only dates in the format of yyyy-MM-dd ?
@noahtaite is right! Looks like your partition path is not yyyy-MM-dd format
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Async Clustering: Seeking Help on Specific Partitioning and Regex Pattern [hudi]
Posted by "soumilshah1995 (via GitHub)" <gi...@apache.org>.
soumilshah1995 commented on issue #10165:
URL: https://github.com/apache/hudi/issues/10165#issuecomment-1828853204
Well I was testing this out looked fine for few run why is it clustering 2023-11 ?
```
None
+-----------------+----------------+---------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|timestamp |input_group_size|state |involved_partitions |
+-----------------+----------------+---------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|20231127191545839|4 |COMPLETED|2023-10-28,2023-10-28,2023-10-29,2023-10-29,2023-10-30,2023-10-30,2023-10-31,2023-10-31 |
|20231127191517836|4 |COMPLETED|2023-10-28,2023-10-28,2023-10-29,2023-10-29,2023-10-30,2023-10-30,2023-10-31,2023-10-31 |
|20231127191357053|4 |COMPLETED|2023-10-28,2023-10-29,2023-10-30,2023-10-31 |
|20231127191307107|4 |COMPLETED|2023-10-28,2023-10-29,2023-10-30,2023-10-31 |
|20231127191233209|4 |COMPLETED|2023-10-28,2023-10-29,2023-10-30,2023-10-31 |
|20231127191124365|30 |COMPLETED|2023-10-28,2023-10-29,2023-10-30,2023-10-31,2023-11-01,2023-11-02,2023-11-03,2023-11-04,2023-11-05,2023-11-06,2023-11-07,2023-11-08,2023-11-09,2023-11-10,2023-11-11,2023-11-12,2023-11-13,2023-11-14,2023-11-15,2023-11-16,2023-11-17,2023-11-18,2023-11-19,2023-11-20,2023-11-21,2023-11-22,2023-11-23,2023-11-24,2023-11-25,2023-11-26|
|20231127191003664|30 |COMPLETED|2023-10-28,2023-10-29,2023-10-30,2023-10-31,2023-11-01,2023-11-02,2023-11-03,2023-11-04,2023-11-05,2023-11-06,2023-11-07,2023-11-08,2023-11-09,2023-11-10,2023-11-11,2023-11-12,2023-11-13,2023-11-14,2023-11-15,2023-11-16,2023-11-17,2023-11-18,2023-11-19,2023-11-20,2023-11-21,2023-11-22,2023-11-23,2023-11-24,2023-11-25,2023-11-26|
|20231127190717007|4 |COMPLETED|2023-10-28,2023-10-29,2023-10-30,2023-10-31 |
+-----------------+----------------+---------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
```
Need to do more testing here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Async Clustering: Seeking Help on Specific Partitioning and Regex Pattern [hudi]
Posted by "soumilshah1995 (via GitHub)" <gi...@apache.org>.
soumilshah1995 closed issue #10165: [SUPPORT] Async Clustering: Seeking Help on Specific Partitioning and Regex Pattern
URL: https://github.com/apache/hudi/issues/10165
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Async Clustering: Seeking Help on Specific Partitioning and Regex Pattern [hudi]
Posted by "soumilshah1995 (via GitHub)" <gi...@apache.org>.
soumilshah1995 commented on issue #10165:
URL: https://github.com/apache/hudi/issues/10165#issuecomment-1828865064
# Test Passed
Ran Delta Streamer in CONT Mode
```
spark-submit \
--class org.apache.hudi.utilities.streamer.HoodieStreamer \
--packages org.apache.hudi:hudi-spark3.4-bundle_2.12:0.14.0 \
--properties-file spark-config.properties \
--master 'local[*]' \
--executor-memory 1g \
jar/hudi-utilities-slim-bundle_2.12-0.14.0.jar \
--continuous \
--source-limit 4000000 \
--min-sync-interval-seconds 20 \
--table-type COPY_ON_WRITE \
--op UPSERT \
--source-ordering-field ts \
--source-class org.apache.hudi.utilities.sources.CsvDFSSource \
--target-base-path file:///Users/soumilshah/IdeaProjects/SparkProject/apache-hudi-delta-streamer-labs/E5/hudidb/orders \
--target-table orders \
--props hudi_tbl.props
```
# CONF
```
hoodie.datasource.write.recordkey.field=order_id
hoodie.datasource.write.partitionpath.field=order_date
hoodie.streamer.source.dfs.root=file:///Users/soumilshah/IdeaProjects/SparkProject/apache-hudi-delta-streamer-labs/E5/sampledata/orders
hoodie.datasource.write.precombine.field=ts
hoodie.clustering.inline=false
hoodie.clustering.async.enabled=true
hoodie.write.concurrency.mode=optimistic_concurrency_control
hoodie.write.lock.provider=org.apache.hudi.client.transaction.lock.InProcessLockProvider
hoodie.deltastreamer.csv.header=true
hoodie.deltastreamer.csv.sep=\t
```
# Lets See Commits and partitions Path
```
+----------------------+-------------------+
|_hoodie_partition_path|_hoodie_commit_time|
+----------------------+-------------------+
| 2023-10-30| 20231127192056631|
| 2023-11-17| 20231127192036634|
| 2023-11-13| 20231127192036634|
| 2023-10-30| 20231127192036634|
| 2023-11-13| 20231127192056631|
| 2023-11-17| 20231127192056631|
| 2023-11-01| 20231127192036634|
| 2023-11-10| 20231127192036634|
| 2023-11-10| 20231127192056631|
| 2023-11-05| 20231127192056631|
+----------------------+-------------------+
only showing top 10 rows
+---------+----------------+-----+-------------------+
|timestamp|input_group_size|state|involved_partitions|
+---------+----------------+-----+-------------------+
+---------+----------------+-----+-------------------+
```
# lets Run Async clustering to cluster only 2023-10 Pattern
<img width="1649" alt="Screenshot 2023-11-27 at 7 24 58 PM" src="https://github.com/apache/hudi/assets/39345855/c87b87e1-3511-4ddf-b040-aa70654270d3">
HoodieClusteringJob: Clustering with basePath: file:///Users/soumilshah/IdeaProjects/SparkProject/apache-hudi-delta-streamer-labs/E5/hudidb/orders, tableName: orders, runningMode: scheduleAndExecute success
# Great
```
+----------------------+-------------------+
|_hoodie_partition_path|_hoodie_commit_time|
+----------------------+-------------------+
| 2023-10-30| 20231127192056631|
| 2023-11-17| 20231127192036634|
| 2023-11-13| 20231127192036634|
| 2023-10-30| 20231127192036634|
| 2023-11-13| 20231127192056631|
| 2023-11-17| 20231127192056631|
| 2023-11-01| 20231127192036634|
| 2023-11-10| 20231127192036634|
| 2023-11-10| 20231127192056631|
| 2023-11-05| 20231127192056631|
+----------------------+-------------------+
only showing top 10 rows
+-----------------+----------------+---------+-------------------------------------------+
|timestamp |input_group_size|state |involved_partitions |
+-----------------+----------------+---------+-------------------------------------------+
|20231127192505457|4 |COMPLETED|2023-10-28,2023-10-29,2023-10-30,2023-10-31|
+-----------------+----------------+---------+-------------------------------------------+
```
# Running DeltaStreamer Again and then running Async Clustering
INFO HoodieClusteringJob: Clustering with basePath: file:///Users/soumilshah/IdeaProjects/SparkProject/apache-hudi-delta-streamer-labs/E5/hudidb/orders, tableName: orders, runningMode: scheduleAndExecute success
# Great
```
+----------------------+-------------------+
|_hoodie_partition_path|_hoodie_commit_time|
+----------------------+-------------------+
| 2023-10-30| 20231127192056631|
| 2023-11-17| 20231127192036634|
| 2023-11-17| 20231127192650158|
| 2023-11-17| 20231127192630161|
| 2023-11-13| 20231127192036634|
| 2023-10-30| 20231127192036634|
| 2023-10-30| 20231127192630161|
| 2023-11-13| 20231127192056631|
| 2023-11-13| 20231127192610153|
| 2023-10-30| 20231127192610153|
+----------------------+-------------------+
only showing top 10 rows
+-----------------+----------------+---------+-------------------------------------------+
|timestamp |input_group_size|state |involved_partitions |
+-----------------+----------------+---------+-------------------------------------------+
|20231127192719681|4 |COMPLETED|2023-10-28,2023-10-29,2023-10-30,2023-10-31|
|20231127192505457|4 |COMPLETED|2023-10-28,2023-10-29,2023-10-30,2023-10-31|
+-----------------+----------------+---------+-------------------------------------------+
```
# Running Deal Streamer Again Third Time and then clustering
HoodieClusteringJob: Clustering with basePath: file:///Users/soumilshah/IdeaProjects/SparkProject/apache-hudi-delta-streamer-labs/E5/hudidb/orders, tableName: orders, runningMode: scheduleAndExecute success
```
+----------------------+-------------------+
|_hoodie_partition_path|_hoodie_commit_time|
+----------------------+-------------------+
| 2023-11-17| 20231127192036634|
| 2023-11-17| 20231127192850185|
| 2023-11-17| 20231127192650158|
| 2023-11-17| 20231127192630161|
| 2023-11-13| 20231127192036634|
| 2023-11-13| 20231127192056631|
| 2023-11-13| 20231127192610153|
| 2023-11-13| 20231127192850185|
| 2023-11-05| 20231127192850185|
| 2023-11-13| 20231127192910187|
+----------------------+-------------------+
only showing top 10 rows
+-----------------+----------------+---------+-------------------------------------------+
|timestamp |input_group_size|state |involved_partitions |
+-----------------+----------------+---------+-------------------------------------------+
|20231127192929734|4 |COMPLETED|2023-10-28,2023-10-29,2023-10-30,2023-10-31|
|20231127192719681|4 |COMPLETED|2023-10-28,2023-10-29,2023-10-30,2023-10-31|
|20231127192505457|4 |COMPLETED|2023-10-28,2023-10-29,2023-10-30,2023-10-31|
+-----------------+----------------+---------+-------------------------------------------+
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Async Clustering: Seeking Help on Specific Partitioning and Regex Pattern [hudi]
Posted by "subash-metica (via GitHub)" <gi...@apache.org>.
subash-metica commented on issue #10165:
URL: https://github.com/apache/hudi/issues/10165#issuecomment-1826948776
1. The partition paths in your example, does it contain only dates in the format of yyyy-MM-dd ?
2. For regex, you have to follow java pattern matcher expression. The code internally uses Pattern.matches() Ref: [https://github.com/apache/hudi/blob/499423c85cda6e9ccf5cec405732517b0ea39355/hudi-[…]tion/cluster/strategy/PartitionAwareClusteringPlanStrategy.java](https://github.com/apache/hudi/blob/499423c85cda6e9ccf5cec405732517b0ea39355/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/cluster/strategy/PartitionAwareClusteringPlanStrategy.java#L183)
Example:
`Regex = ^2023-10-[0-9][0-9]$ [to match all 2023 Oct dates. (warning: this pattern include invalid dates)]`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Async Clustering: Seeking Help on Specific Partitioning and Regex Pattern [hudi]
Posted by "noahtaite (via GitHub)" <gi...@apache.org>.
noahtaite commented on issue #10165:
URL: https://github.com/apache/hudi/issues/10165#issuecomment-1828736370
@soumilshah1995 I did a bit of digging, https://hudi.apache.org/docs/procedures/ doesn't explicitly say it, but there appears to be a `show_involved_partition` boolean flag we need to set to true to see anything other than "*" in that result. I've never used the Hudi procedures before but can you try setting that and let us know the result?
https://github.com/apache/hudi/blob/4c3a1db146bc5f81545f5d17020175ea9797fcd9/hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/procedures/ShowClusteringProcedure.scala#L57-L80
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Async Clustering: Seeking Help on Specific Partitioning and Regex Pattern [hudi]
Posted by "soumilshah1995 (via GitHub)" <gi...@apache.org>.
soumilshah1995 commented on issue #10165:
URL: https://github.com/apache/hudi/issues/10165#issuecomment-1828844187
Issue has been resolved
I had to pass this in quotes
```
--hoodie-conf 'hoodie.clustering.plan.strategy.partition.regex.pattern=^2023-10-[0-9][0-9]$' \
```
<img width="1071" alt="Screenshot 2023-11-27 at 7 08 22 PM" src="https://github.com/apache/hudi/assets/39345855/efe6c78e-33c8-4f66-8515-2b6b0ebc07b8">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Async Clustering: Seeking Help on Specific Partitioning and Regex Pattern [hudi]
Posted by "noahtaite (via GitHub)" <gi...@apache.org>.
noahtaite commented on issue #10165:
URL: https://github.com/apache/hudi/issues/10165#issuecomment-1828595136
> <img alt="Screenshot 2023-11-27 at 9 07 05 AM" width="285" src="https://private-user-images.githubusercontent.com/39345855/285898963-1b019a83-1924-441f-bf07-a1816aea2b93.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTEiLCJleHAiOjE3MDExMTgwMzMsIm5iZiI6MTcwMTExNzczMywicGF0aCI6Ii8zOTM0NTg1NS8yODU4OTg5NjMtMWIwMTlhODMtMTkyNC00NDFmLWJmMDctYTE4MTZhZWEyYjkzLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFJV05KWUFYNENTVkVINTNBJTJGMjAyMzExMjclMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjMxMTI3VDIwNDIxM1omWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTczYWVhM2E2OWNlMzMwYzg1Mzk2YWEyODZkYTIyODkxYmEzMmMwOWZmOGM4ZmE4ZWIzZjBkNzFhYTczMDBkZmEmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.KNfr2st0vjAD46TUdp8tCRE4gEH0fA9s0Jf5rHoYpAU">
Hey @soumilshah1995 can we see the output of df.select("_hoodie_partition_path")? To me it appears as though your data uses the partition fields "year,month" given that your data is structured like:
```
orders/2023/10
orders/2023/11
```
So I think partition_path may be like "2023/10" + "2023/11". Which could explain why it is not getting selected as you expect. From my experience working with Hudi services that rely on partition path, they usually expect it in the format that the data is physically written (which can be figured out by looking at the `_hoodie_partition_path` field).
Note I am also a Hudi user, so just trying to help given the information presented.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Async Clustering: Seeking Help on Specific Partitioning and Regex Pattern [hudi]
Posted by "soumilshah1995 (via GitHub)" <gi...@apache.org>.
soumilshah1995 commented on issue #10165:
URL: https://github.com/apache/hudi/issues/10165#issuecomment-1828848290
I am looking fwd to make YouTube videos on this topics
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Async Clustering: Seeking Help on Specific Partitioning and Regex Pattern [hudi]
Posted by "soumilshah1995 (via GitHub)" <gi...@apache.org>.
soumilshah1995 closed issue #10165: [SUPPORT] Async Clustering: Seeking Help on Specific Partitioning and Regex Pattern
URL: https://github.com/apache/hudi/issues/10165
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Async Clustering: Seeking Help on Specific Partitioning and Regex Pattern [hudi]
Posted by "soumilshah1995 (via GitHub)" <gi...@apache.org>.
soumilshah1995 commented on issue #10165:
URL: https://github.com/apache/hudi/issues/10165#issuecomment-1826951261
Thanks I will try out soon and provide an update on this tickets
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org