You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "soumilshah1995 (via GitHub)" <gi...@apache.org> on 2024/02/09 13:33:54 UTC
[I] [FeatureRequest] Inquiry Regarding Hudi Exporter with SQL Transformer for Data Filtering [hudi]
soumilshah1995 opened a new issue, #10644:
URL: https://github.com/apache/hudi/issues/10644
Hello,
I'm reaching out to inquire about the Hudi exporter service. I've had some experience working with it, but I'm particularly interested in whether we support the integration of SQL transformer with it.
The concept is to utilize the Hudi export utility for exporting Hudi data. However, there could be instances where customers require exporting filtered data. For instance, they might need all data related to a specific stock like AAPL.
Do we have plans to incorporate a filtering mechanism into the Hudi exporter?
Here's an example of the Spark-submit command:
```
park-submit \
--class org.apache.hudi.utilities.HoodieSnapshotExporter \
--packages 'org.apache.hudi:hudi-spark3.4-bundle_2.12:0.14.0' \
--master 'local[*]' \
--executor-memory 1g \
/Users/soumilshah/IdeaProjects/SparkProject/DeltaStreamer/jar/hudi-utilities-slim-bundle_2.12-0.14.0.jar \
--source-base-path 'file:///Users/soumilshah/IdeaProjects/SparkProject/DeltaStreamer/hudi/bronze_orders' \
--target-output-path 'file:///Users/soumilshah/IdeaProjects/SparkProject/DeltaStreamer/hudi/json/' \
--output-format 'json'
```
i tried this flags
```
--transformer-class org.apache.hudi.utilities.transform.SqlQueryBasedTransformer \
--hoodie-conf hoodie.deltastreamer.transformer.sql='SELECT *, extract(year from order_date) as year, extract(month from order_date) as month FROM <SRC> a' \
```
Looks like its not supported with HoodieSnapshotExporter
REF
https://hudi.apache.org/docs/snapshot_exporter (edited)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [FeatureRequest] Inquiry Regarding Hudi Exporter with SQL Transformer for Data Filtering [hudi]
Posted by "soumilshah1995 (via GitHub)" <gi...@apache.org>.
soumilshah1995 commented on issue #10644:
URL: https://github.com/apache/hudi/issues/10644#issuecomment-2251065010
Hi there I saw that this ticket was completed and I was trying out this functionality
Docs
```
xport to json or parquet dataset with transformation/filtering
The Exporter supports custom transformation/filtering on records before writing to json or parquet dataset. This is done by supplying implementation of org.apache.hudi.utilities.transform.Transformer via --transformer-class option.
spark-submit \
--jars "packaging/hudi-spark-bundle/target/hudi-spark-bundle_2.11-0.15.0.jar" \
--deploy-mode "client" \
--class "org.apache.hudi.utilities.HoodieSnapshotExporter" \
packaging/hudi-utilities-bundle/target/hudi-utilities-bundle_2.11-0.15.0.jar \
--source-base-path "/tmp/" \
--target-output-path "/tmp/exported/json/" \
--transformer-class "org.apache.hudi.utilities.transform.SqlQueryBasedTransformer" \
--transformer-sql "SELECT substr(rider,1,10) as rider, trip_type as tripType FROM <SRC> WHERE trip_type = 'BLACK' LIMIT 10" \
--output-format "json" # or "parquet"
```
https://hudi.apache.org/docs/next/snapshot_exporter/
Following is failing
TEST 1: NO tranformer PASS
```
spark-submit \
--class org.apache.hudi.utilities.HoodieSnapshotExporter \
--packages org.apache.hudi:hudi-spark3.4-bundle_2.12:0.15.0 \
--master 'local[*]' \
--executor-memory 1g \
/Users/soumilshah/IdeaProjects/SparkProject/apache-hudi-delta-streamer-labs/E1/jar/hudi-utilities-slim-bundle_2.12-0.15.0.jar \
--source-base-path '/Users/soumilshah/IdeaProjects/SparkProject/apache-hudi-delta-streamer-labs/E1/silver/' \
--target-output-path '/Users/soumilshah/IdeaProjects/SparkProject/apache-hudi-delta-streamer-labs/E1/dump/json/' \
--output-format 'parquet'
```
# Test 2 : With transformer
```
spark-submit \
--class org.apache.hudi.utilities.HoodieSnapshotExporter \
--packages org.apache.hudi:hudi-spark3.4-bundle_2.12:0.15.0 \
--master 'local[*]' \
--executor-memory 1g \
/Users/soumilshah/IdeaProjects/SparkProject/apache-hudi-delta-streamer-labs/E1/jar/hudi-utilities-slim-bundle_2.12-0.15.0.jar \
--source-base-path '/Users/soumilshah/IdeaProjects/SparkProject/apache-hudi-delta-streamer-labs/E1/silver/' \
--target-output-path '/Users/soumilshah/IdeaProjects/SparkProject/apache-hudi-delta-streamer-labs/E1/dump/json/' \
--transformer-class org.apache.hudi.utilities.transform.SqlQueryBasedTransformer \
--transformer-sql "SELECT * FROM <SRC> WHERE destinationstate='NY'" \
--output-format 'parquet'
```
# logs
```
vy Default Cache set to: /Users/soumilshah/.ivy2/cache
The jars for the packages stored in: /Users/soumilshah/.ivy2/jars
org.apache.hudi#hudi-spark3.4-bundle_2.12 added as a dependency
:: resolving dependencies :: org.apache.spark#spark-submit-parent-f0fce1d3-e446-495c-a37a-e2dd7e335611;1.0
confs: [default]
found org.apache.hudi#hudi-spark3.4-bundle_2.12;0.15.0 in central
:: resolution report :: resolve 56ms :: artifacts dl 1ms
:: modules in use:
org.apache.hudi#hudi-spark3.4-bundle_2.12;0.15.0 from central in [default]
---------------------------------------------------------------------
| | modules || artifacts |
| conf | number| search|dwnlded|evicted|| number|dwnlded|
---------------------------------------------------------------------
| default | 1 | 0 | 0 | 0 || 1 | 0 |
---------------------------------------------------------------------
:: retrieving :: org.apache.spark#spark-submit-parent-f0fce1d3-e446-495c-a37a-e2dd7e335611
confs: [default]
0 artifacts copied, 1 already retrieved (0kB/2ms)
24/07/25 13:43:59 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Exception in thread "main" org.apache.hudi.com.beust.jcommander.ParameterException: Was passed main parameter '--transformer-class' but no main parameter was defined in your arg class
at org.apache.hudi.com.beust.jcommander.JCommander.initMainParameterValue(JCommander.java:954)
at org.apache.hudi.com.beust.jcommander.JCommander.parseValues(JCommander.java:755)
at org.apache.hudi.com.beust.jcommander.JCommander.parse(JCommander.java:356)
at org.apache.hudi.com.beust.jcommander.JCommander.parse(JCommander.java:335)
at org.apache.hudi.com.beust.jcommander.JCommander.<init>(JCommander.java:251)
at org.apache.hudi.utilities.HoodieSnapshotExporter.main(HoodieSnapshotExporter.java:292)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:1020)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:192)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:215)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:91)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1111)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1120)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
24/07/25 13:43:59 INFO ShutdownHookManager: Shutdown hook called
24/07/25 13:43:59 INFO ShutdownHookManager: Deleting directory /private/var/folders/qq/s_1bjv516pn_mck29cwdwxnm0000gp/T/spark-4431a82f-15b8-4ac6-948a-db853cbf9fe3
(base) soumilshah@ip-192-168-1-31 E1 %
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [FeatureRequest] Inquiry Regarding Hudi Exporter with SQL Transformer for Data Filtering [hudi]
Posted by "soumilshah1995 (via GitHub)" <gi...@apache.org>.
soumilshah1995 commented on issue #10644:
URL: https://github.com/apache/hudi/issues/10644#issuecomment-1939762270
Thank you sir
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [FeatureRequest] Inquiry Regarding Hudi Exporter with SQL Transformer for Data Filtering [hudi]
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #10644:
URL: https://github.com/apache/hudi/issues/10644#issuecomment-1938822414
Hey Soumil,
Thanks for the suggestion. Can be a good feature add to hudi exporter utility.
Created tracking JIRA for the same - https://issues.apache.org/jira/browse/HUDI-7403
Thanks,
Aditya
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [FeatureRequest] Inquiry Regarding Hudi Exporter with SQL Transformer for Data Filtering [hudi]
Posted by "soumilshah1995 (via GitHub)" <gi...@apache.org>.
soumilshah1995 commented on issue #10644:
URL: https://github.com/apache/hudi/issues/10644#issuecomment-2251899581
Yes you are right I will compile master branch and keep you posted
Thanking You,
Soumil Nitin Shah
On Thu, Jul 25, 2024 at 11:29 PM Vova Kolmakov ***@***.***>
wrote:
> @soumilshah1995 <https://github.com/soumilshah1995> hi! This feature
> exists only in master branch, but it looks like you are trying to test it
> on release 0.15, which does not contain this commit. Am i right?
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/apache/hudi/issues/10644#issuecomment-2251893241>, or
> unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AJMF5P6H2EX6Q5REORLHCQ3ZOG7BPAVCNFSM6AAAAABDBPL24WVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDENJRHA4TGMRUGE>
> .
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [FeatureRequest] Inquiry Regarding Hudi Exporter with SQL Transformer for Data Filtering [hudi]
Posted by "wombatu-kun (via GitHub)" <gi...@apache.org>.
wombatu-kun commented on issue #10644:
URL: https://github.com/apache/hudi/issues/10644#issuecomment-2251767611
> @wombatu-kun can you take care of this?
Yes of course, i'm already trying to figure out
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [FeatureRequest] Inquiry Regarding Hudi Exporter with SQL Transformer for Data Filtering [hudi]
Posted by "soumilshah1995 (via GitHub)" <gi...@apache.org>.
soumilshah1995 commented on issue #10644:
URL: https://github.com/apache/hudi/issues/10644#issuecomment-2253027370
tested with Master Branch
Test passed
```
spark-submit \
--class org.apache.hudi.utilities.HoodieSnapshotExporter \
--packages org.apache.hudi:hudi-spark3.4-bundle_2.12:1.0.0-beta2 \
--master 'local[*]' \
--executor-memory 1g \
/Users/soumilshah/IdeaProjects/SparkProject/apache-hudi-delta-streamer-labs/E1/jar/hudi-utilities-slim-bundle_2.12-1.0.0-beta2.jar \
--source-base-path '/Users/soumilshah/IdeaProjects/SparkProject/apache-hudi-delta-streamer-labs/E1/silver/' \
--target-output-path '/Users/soumilshah/IdeaProjects/SparkProject/apache-hudi-delta-streamer-labs/E1/dump/parquet/' \
--transformer-class "org.apache.hudi.utilities.transform.SqlQueryBasedTransformer" \
--transformer-sql "SELECT * FROM <SRC> WHERE destinationstate='NY'" \
--output-format 'parquet'
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [FeatureRequest] Inquiry Regarding Hudi Exporter with SQL Transformer for Data Filtering [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #10644:
URL: https://github.com/apache/hudi/issues/10644#issuecomment-2251766311
@wombatu-kun can you take care of this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [FeatureRequest] Inquiry Regarding Hudi Exporter with SQL Transformer for Data Filtering [hudi]
Posted by "wombatu-kun (via GitHub)" <gi...@apache.org>.
wombatu-kun commented on issue #10644:
URL: https://github.com/apache/hudi/issues/10644#issuecomment-2251893241
@soumilshah1995 hi! This feature exists only in master branch, but it looks like you are trying to test it on release 0.15, which does not contain this commit. Am i right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [FeatureRequest] Inquiry Regarding Hudi Exporter with SQL Transformer for Data Filtering [hudi]
Posted by "soumilshah1995 (via GitHub)" <gi...@apache.org>.
soumilshah1995 closed issue #10644: [FeatureRequest] Inquiry Regarding Hudi Exporter with SQL Transformer for Data Filtering
URL: https://github.com/apache/hudi/issues/10644
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org