You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2021/11/20 01:36:37 UTC
[GitHub] [hudi] kywe665 opened a new pull request #4049: [HUDI-2726] - Docs for HoodieSnapshotExporter
kywe665 opened a new pull request #4049:
URL: https://github.com/apache/hudi/pull/4049
## What is the purpose of the pull request
HoodieSnapshotExporter did not have any documentation.
## Brief change log
- Added docs for HoodieSnapshotExporter
## Verify this pull request
This pull request is a docs update only
## Committer checklist
- [X] Has a corresponding JIRA in PR title & commit
- [X] Commit message is descriptive of the change
- [X] CI is green
- [X] Necessary doc changes done or have another open PR
- [X] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] kywe665 commented on pull request #4049: [HUDI-2726] - Docs for HoodieSnapshotExporter
Posted by GitBox <gi...@apache.org>.
kywe665 commented on pull request #4049:
URL: https://github.com/apache/hudi/pull/4049#issuecomment-977051311
Thanks I updated those minor typos
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] bhasudha commented on a change in pull request #4049: [HUDI-2726] - Docs for HoodieSnapshotExporter
Posted by GitBox <gi...@apache.org>.
bhasudha commented on a change in pull request #4049:
URL: https://github.com/apache/hudi/pull/4049#discussion_r754864117
##########
File path: website/docs/snapshot_exporter.md
##########
@@ -0,0 +1,115 @@
+---
+title: Snapshot Exporter
+keywords: [hudi, snapshotexporter, export]
+toc: true
+---
+
+## Introduction
+HoodieSnapshotExporter allows you to copy data from one location to another for backups or other purposes.
+You can write data as Hudi, Json, or Parquet file formats. In addition to copying data, you can also repartition data
Review comment:
`orc` format as well
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] bhasudha merged pull request #4049: [HUDI-2726] - Docs for HoodieSnapshotExporter
Posted by GitBox <gi...@apache.org>.
bhasudha merged pull request #4049:
URL: https://github.com/apache/hudi/pull/4049
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] bhasudha commented on a change in pull request #4049: [HUDI-2726] - Docs for HoodieSnapshotExporter
Posted by GitBox <gi...@apache.org>.
bhasudha commented on a change in pull request #4049:
URL: https://github.com/apache/hudi/pull/4049#discussion_r754866921
##########
File path: website/docs/snapshot_exporter.md
##########
@@ -0,0 +1,115 @@
+---
+title: Snapshot Exporter
+keywords: [hudi, snapshotexporter, export]
+toc: true
+---
+
+## Introduction
+HoodieSnapshotExporter allows you to copy data from one location to another for backups or other purposes.
+You can write data as Hudi, Json, or Parquet file formats. In addition to copying data, you can also repartition data
+with a provided field or implement custom repartitioning by extending a class shown in detail below.
+
+## Arguments
+HoodieSnapshotExporter accepts a reference to a source path and a destination path. The utility will issue a
+query, perform any repartitioning if required and will write the data as Hudi, parquet, or json format.
+
+|Argument|Description|Required|Note|
+|------------|--------|-----------|--|
+|--source-base-path|Base path for the source Hudi dataset to be snapshotted|required||
+|--target-output-path|Output path for storing a particular snapshot|required||
+|--output-format|Output format for the exported dataset; accept these values: json,parquet,hudi|required||
+|--output-partition-field|A field to be used by Spark repartitioning|optional|Ignored when "Hudi" or when --output-partitioner is specified.The output dataset's default partition field will inherent from the source Hudi dataset.|
+|--output-partitioner|A class to facilitate custom repartitioning|optional|Ignored when using output-format "Hudi"|
+
+## Examples
+
+### Copy a Hudi dataset
+
+Exporter scans the source dataset and then makes a copy of it to the target output path.
+```bash
+spark-submit \
+ --jars "packaging/hudi-spark-bundle/target/hudi-spark-bundle_2.11-0.6.0-SNAPSHOT.jar" \
+ --deploy-mode "client" \
+ --class "org.apache.hudi.utilities.HoodieSnapshotExporter" \
+ packaging/hudi-utilities-bundle/target/hudi-utilities-bundle_2.11-0.6.0-SNAPSHOT.jar \
+ --source-base-path "/tmp/" \
+ --target-output-path "/tmp/exported/hudi/" \
+ --output-format "hudi"
+```
+
+### Export to json or parquet dataset
+The Exporter can also convert the source dataset into other formats. Currently only "json" and "parquet" are supported.
+
+```bash
+spark-submit \
+ --jars "packaging/hudi-spark-bundle/target/hudi-spark-bundle_2.11-0.6.0-SNAPSHOT.jar" \
+ --deploy-mode "client" \
+ --class "org.apache.hudi.utilities.HoodieSnapshotExporter" \
+ packaging/hudi-utilities-bundle/target/hudi-utilities-bundle_2.11-0.6.0-SNAPSHOT.jar \
+ --source-base-path "/tmp/" \
+ --target-output-path "/tmp/exported/json/" \
+ --output-format "json" # or "parquet"
+```
+
+### Re-partitioning
+When exporting to a different format, the Exporter takes the `--output-partition-field` parameter to do some custom re-partitioning.
+Note: All `_hoodie_*` metadata fields will be stripped during export, so make sure to use an existing non-metadata field as the output partitions.
+
+By default, if no partitioning parameters are given, the output dataset will have no partition.
+
+Example:
+```bash
+spark-submit \
+ --jars "packaging/hudi-spark-bundle/target/hudi-spark-bundle_2.11-0.6.0-SNAPSHOT.jar" \
+ --deploy-mode "client" \
+ --class "org.apache.hudi.utilities.HoodieSnapshotExporter" \
+ packaging/hudi-utilities-bundle/target/hudi-utilities-bundle_2.11-0.6.0-SNAPSHOT.jar \
+ --source-base-path "/tmp/" \
+ --target-output-path "/tmp/exported/json/" \
+ --output-format "json" \
+ --output-partition-field "symbol" # assume the source dataset contains a field `symbol`
+```
+
+The output directory will look like this
+
+```bash
+`_SUCCESS symbol=AMRS symbol=AYX symbol=CDMO symbol=CRC symbol=DRNA ...`
+```
+
+### Custom Re=partitioning
Review comment:
Typo : Re-Partitioning
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] kywe665 commented on a change in pull request #4049: [HUDI-2726] - Docs for HoodieSnapshotExporter
Posted by GitBox <gi...@apache.org>.
kywe665 commented on a change in pull request #4049:
URL: https://github.com/apache/hudi/pull/4049#discussion_r755438799
##########
File path: website/docs/snapshot_exporter.md
##########
@@ -0,0 +1,115 @@
+---
+title: Snapshot Exporter
+keywords: [hudi, snapshotexporter, export]
+toc: true
+---
+
+## Introduction
+HoodieSnapshotExporter allows you to copy data from one location to another for backups or other purposes.
+You can write data as Hudi, Json, or Parquet file formats. In addition to copying data, you can also repartition data
Review comment:
fixed
##########
File path: website/docs/snapshot_exporter.md
##########
@@ -0,0 +1,115 @@
+---
+title: Snapshot Exporter
+keywords: [hudi, snapshotexporter, export]
+toc: true
+---
+
+## Introduction
+HoodieSnapshotExporter allows you to copy data from one location to another for backups or other purposes.
+You can write data as Hudi, Json, or Parquet file formats. In addition to copying data, you can also repartition data
+with a provided field or implement custom repartitioning by extending a class shown in detail below.
+
+## Arguments
+HoodieSnapshotExporter accepts a reference to a source path and a destination path. The utility will issue a
+query, perform any repartitioning if required and will write the data as Hudi, parquet, or json format.
+
+|Argument|Description|Required|Note|
+|------------|--------|-----------|--|
+|--source-base-path|Base path for the source Hudi dataset to be snapshotted|required||
+|--target-output-path|Output path for storing a particular snapshot|required||
+|--output-format|Output format for the exported dataset; accept these values: json,parquet,hudi|required||
+|--output-partition-field|A field to be used by Spark repartitioning|optional|Ignored when "Hudi" or when --output-partitioner is specified.The output dataset's default partition field will inherent from the source Hudi dataset.|
+|--output-partitioner|A class to facilitate custom repartitioning|optional|Ignored when using output-format "Hudi"|
+
+## Examples
+
+### Copy a Hudi dataset
+
+Exporter scans the source dataset and then makes a copy of it to the target output path.
+```bash
+spark-submit \
+ --jars "packaging/hudi-spark-bundle/target/hudi-spark-bundle_2.11-0.6.0-SNAPSHOT.jar" \
+ --deploy-mode "client" \
+ --class "org.apache.hudi.utilities.HoodieSnapshotExporter" \
+ packaging/hudi-utilities-bundle/target/hudi-utilities-bundle_2.11-0.6.0-SNAPSHOT.jar \
+ --source-base-path "/tmp/" \
+ --target-output-path "/tmp/exported/hudi/" \
+ --output-format "hudi"
+```
+
+### Export to json or parquet dataset
+The Exporter can also convert the source dataset into other formats. Currently only "json" and "parquet" are supported.
+
+```bash
+spark-submit \
+ --jars "packaging/hudi-spark-bundle/target/hudi-spark-bundle_2.11-0.6.0-SNAPSHOT.jar" \
+ --deploy-mode "client" \
+ --class "org.apache.hudi.utilities.HoodieSnapshotExporter" \
+ packaging/hudi-utilities-bundle/target/hudi-utilities-bundle_2.11-0.6.0-SNAPSHOT.jar \
+ --source-base-path "/tmp/" \
+ --target-output-path "/tmp/exported/json/" \
+ --output-format "json" # or "parquet"
+```
+
+### Re-partitioning
+When exporting to a different format, the Exporter takes the `--output-partition-field` parameter to do some custom re-partitioning.
+Note: All `_hoodie_*` metadata fields will be stripped during export, so make sure to use an existing non-metadata field as the output partitions.
+
+By default, if no partitioning parameters are given, the output dataset will have no partition.
+
+Example:
+```bash
+spark-submit \
+ --jars "packaging/hudi-spark-bundle/target/hudi-spark-bundle_2.11-0.6.0-SNAPSHOT.jar" \
+ --deploy-mode "client" \
+ --class "org.apache.hudi.utilities.HoodieSnapshotExporter" \
+ packaging/hudi-utilities-bundle/target/hudi-utilities-bundle_2.11-0.6.0-SNAPSHOT.jar \
+ --source-base-path "/tmp/" \
+ --target-output-path "/tmp/exported/json/" \
+ --output-format "json" \
+ --output-partition-field "symbol" # assume the source dataset contains a field `symbol`
+```
+
+The output directory will look like this
+
+```bash
+`_SUCCESS symbol=AMRS symbol=AYX symbol=CDMO symbol=CRC symbol=DRNA ...`
+```
+
+### Custom Re=partitioning
Review comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org