You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "the-other-tim-brown (via GitHub)" <gi...@apache.org> on 2023/05/04 06:14:21 UTC
[GitHub] [hudi] the-other-tim-brown opened a new pull request, #8629: [HUDI=6168] Add ability to parse partition value into row for S3 and GCS sources
the-other-tim-brown opened a new pull request, #8629:
URL: https://github.com/apache/hudi/pull/8629
### Change Logs
Adds the ability to parse the partition path in a source if specified and add the value into the row.
Add unit tests for the class
### Impact
Adds new feature to add value that is present in the path
### Risk level (write none, low medium or high below)
none
### Documentation Update
_Describe any necessary documentation update if there is any new feature, config, or user-facing change_
- _The config description must be updated if new configs are added or the default value of the configs are changed_
- _Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
ticket number here and follow the [instruction](https://hudi.apache.org/contribute/developer-setup#website) to make
changes to the website._
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8629: [HUDI-6168] Add ability to parse partition value into row for S3 and GCS sources
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8629:
URL: https://github.com/apache/hudi/pull/8629#issuecomment-1535093275
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "653ccdca8099c2729b6afb415277916ff5e08e16",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16818",
"triggerID" : "653ccdca8099c2729b6afb415277916ff5e08e16",
"triggerType" : "PUSH"
}, {
"hash" : "ba0e8193c6ed04723c709b57b83c57280449f681",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16835",
"triggerID" : "ba0e8193c6ed04723c709b57b83c57280449f681",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 653ccdca8099c2729b6afb415277916ff5e08e16 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16818)
* ba0e8193c6ed04723c709b57b83c57280449f681 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16835)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #8629: [HUDI-6168] Add ability to parse partition value into row for S3 and GCS sources
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on PR #8629:
URL: https://github.com/apache/hudi/pull/8629#issuecomment-1535723560
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8629: [HUDI-6168] Add ability to parse partition value into row for S3 and GCS sources
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8629:
URL: https://github.com/apache/hudi/pull/8629#issuecomment-1536081650
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "653ccdca8099c2729b6afb415277916ff5e08e16",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16818",
"triggerID" : "653ccdca8099c2729b6afb415277916ff5e08e16",
"triggerType" : "PUSH"
}, {
"hash" : "ba0e8193c6ed04723c709b57b83c57280449f681",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16835",
"triggerID" : "ba0e8193c6ed04723c709b57b83c57280449f681",
"triggerType" : "PUSH"
}, {
"hash" : "ba0e8193c6ed04723c709b57b83c57280449f681",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16855",
"triggerID" : "1535723560",
"triggerType" : "MANUAL"
}, {
"hash" : "ae779fccf9f8e511c0cc42d0ef37ea80db16627f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16857",
"triggerID" : "ae779fccf9f8e511c0cc42d0ef37ea80db16627f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ae779fccf9f8e511c0cc42d0ef37ea80db16627f Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16857)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8629: [HUDI-6168] Add ability to parse partition value into row for S3 and GCS sources
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8629:
URL: https://github.com/apache/hudi/pull/8629#issuecomment-1535337893
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "653ccdca8099c2729b6afb415277916ff5e08e16",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16818",
"triggerID" : "653ccdca8099c2729b6afb415277916ff5e08e16",
"triggerType" : "PUSH"
}, {
"hash" : "ba0e8193c6ed04723c709b57b83c57280449f681",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16835",
"triggerID" : "ba0e8193c6ed04723c709b57b83c57280449f681",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ba0e8193c6ed04723c709b57b83c57280449f681 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16835)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8629: [HUDI-6168] Add ability to parse partition value into row for S3 and GCS sources
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8629:
URL: https://github.com/apache/hudi/pull/8629#issuecomment-1535047359
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "653ccdca8099c2729b6afb415277916ff5e08e16",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16818",
"triggerID" : "653ccdca8099c2729b6afb415277916ff5e08e16",
"triggerType" : "PUSH"
}, {
"hash" : "ba0e8193c6ed04723c709b57b83c57280449f681",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ba0e8193c6ed04723c709b57b83c57280449f681",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 653ccdca8099c2729b6afb415277916ff5e08e16 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16818)
* ba0e8193c6ed04723c709b57b83c57280449f681 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan merged pull request #8629: [HUDI-6168] Add ability to parse partition value into row for S3 and GCS sources
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan merged PR #8629:
URL: https://github.com/apache/hudi/pull/8629
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope commented on a diff in pull request #8629: [HUDI-6168] Add ability to parse partition value into row for S3 and GCS sources
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #8629:
URL: https://github.com/apache/hudi/pull/8629#discussion_r1184599557
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/CloudObjectsSelectorCommon.java:
##########
@@ -176,6 +179,18 @@ public static Option<Dataset<Row>> loadAsDataset(SparkSession spark, List<CloudO
totalSize *= 1.1;
long parquetMaxFileSize = props.getLong(PARQUET_MAX_FILE_SIZE.key(), Long.parseLong(PARQUET_MAX_FILE_SIZE.defaultValue()));
int numPartitions = (int) Math.max(totalSize / parquetMaxFileSize, 1);
- return Option.of(reader.load(paths.toArray(new String[cloudObjectMetadata.size()])).coalesce(numPartitions));
+ Dataset<Row> dataset = reader.load(paths.toArray(new String[cloudObjectMetadata.size()])).coalesce(numPartitions);
+
+ // add partition column from source path if configured
+ if (props.containsKey(PATH_BASED_PARTITION_FIELDS)) {
+ String[] partitionKeysToAdd = props.getString(PATH_BASED_PARTITION_FIELDS).split(",");
+ // Add partition column for all path-based partition keys
+ for (String partitionKey : partitionKeysToAdd) {
+ String partitionPathPattern = String.format("%s=", partitionKey);
+ LOG.info(String.format("Adding column %s to dataset", partitionKey));
+ dataset = dataset.withColumn(partitionKey, split(split(input_file_name(), partitionPathPattern).getItem(1), "/").getItem(0));
Review Comment:
What happens if the partitionPathPattern is not found in the file name?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8629: [HUDI-6168] Add ability to parse partition value into row for S3 and GCS sources
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8629:
URL: https://github.com/apache/hudi/pull/8629#issuecomment-1535738482
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "653ccdca8099c2729b6afb415277916ff5e08e16",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16818",
"triggerID" : "653ccdca8099c2729b6afb415277916ff5e08e16",
"triggerType" : "PUSH"
}, {
"hash" : "ba0e8193c6ed04723c709b57b83c57280449f681",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16835",
"triggerID" : "ba0e8193c6ed04723c709b57b83c57280449f681",
"triggerType" : "PUSH"
}, {
"hash" : "ba0e8193c6ed04723c709b57b83c57280449f681",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16855",
"triggerID" : "1535723560",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* ba0e8193c6ed04723c709b57b83c57280449f681 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16835) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16855)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8629: [HUDI-6168] Add ability to parse partition value into row for S3 and GCS sources
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8629:
URL: https://github.com/apache/hudi/pull/8629#issuecomment-1535752542
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "653ccdca8099c2729b6afb415277916ff5e08e16",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16818",
"triggerID" : "653ccdca8099c2729b6afb415277916ff5e08e16",
"triggerType" : "PUSH"
}, {
"hash" : "ba0e8193c6ed04723c709b57b83c57280449f681",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16835",
"triggerID" : "ba0e8193c6ed04723c709b57b83c57280449f681",
"triggerType" : "PUSH"
}, {
"hash" : "ba0e8193c6ed04723c709b57b83c57280449f681",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16855",
"triggerID" : "1535723560",
"triggerType" : "MANUAL"
}, {
"hash" : "ae779fccf9f8e511c0cc42d0ef37ea80db16627f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ae779fccf9f8e511c0cc42d0ef37ea80db16627f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ba0e8193c6ed04723c709b57b83c57280449f681 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16835) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16855)
* ae779fccf9f8e511c0cc42d0ef37ea80db16627f UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8629: [HUDI-6168] Add ability to parse partition value into row for S3 and GCS sources
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8629:
URL: https://github.com/apache/hudi/pull/8629#issuecomment-1534200979
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "653ccdca8099c2729b6afb415277916ff5e08e16",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16818",
"triggerID" : "653ccdca8099c2729b6afb415277916ff5e08e16",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 653ccdca8099c2729b6afb415277916ff5e08e16 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16818)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8629: [HUDI-6168] Add ability to parse partition value into row for S3 and GCS sources
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8629:
URL: https://github.com/apache/hudi/pull/8629#issuecomment-1534446944
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "653ccdca8099c2729b6afb415277916ff5e08e16",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16818",
"triggerID" : "653ccdca8099c2729b6afb415277916ff5e08e16",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 653ccdca8099c2729b6afb415277916ff5e08e16 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16818)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8629: [HUDI-6168] Add ability to parse partition value into row for S3 and GCS sources
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8629:
URL: https://github.com/apache/hudi/pull/8629#issuecomment-1534191590
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "653ccdca8099c2729b6afb415277916ff5e08e16",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "653ccdca8099c2729b6afb415277916ff5e08e16",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 653ccdca8099c2729b6afb415277916ff5e08e16 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #8629: [HUDI-6168] Add ability to parse partition value into row for S3 and GCS sources
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on PR #8629:
URL: https://github.com/apache/hudi/pull/8629#issuecomment-1536822131
going ahead w/ landing. the test failure is a known flaky one. https://github.com/apache/hudi/pull/8646
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] the-other-tim-brown commented on a diff in pull request #8629: [HUDI-6168] Add ability to parse partition value into row for S3 and GCS sources
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #8629:
URL: https://github.com/apache/hudi/pull/8629#discussion_r1185234065
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/CloudObjectsSelectorCommon.java:
##########
@@ -176,6 +179,18 @@ public static Option<Dataset<Row>> loadAsDataset(SparkSession spark, List<CloudO
totalSize *= 1.1;
long parquetMaxFileSize = props.getLong(PARQUET_MAX_FILE_SIZE.key(), Long.parseLong(PARQUET_MAX_FILE_SIZE.defaultValue()));
int numPartitions = (int) Math.max(totalSize / parquetMaxFileSize, 1);
- return Option.of(reader.load(paths.toArray(new String[cloudObjectMetadata.size()])).coalesce(numPartitions));
+ Dataset<Row> dataset = reader.load(paths.toArray(new String[cloudObjectMetadata.size()])).coalesce(numPartitions);
+
+ // add partition column from source path if configured
+ if (props.containsKey(PATH_BASED_PARTITION_FIELDS)) {
+ String[] partitionKeysToAdd = props.getString(PATH_BASED_PARTITION_FIELDS).split(",");
+ // Add partition column for all path-based partition keys
+ for (String partitionKey : partitionKeysToAdd) {
+ String partitionPathPattern = String.format("%s=", partitionKey);
+ LOG.info(String.format("Adding column %s to dataset", partitionKey));
+ dataset = dataset.withColumn(partitionKey, split(split(input_file_name(), partitionPathPattern).getItem(1), "/").getItem(0));
Review Comment:
It will return null as the value for that column. Added a unit test and updated the comment in the code to reflect this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8629: [HUDI-6168] Add ability to parse partition value into row for S3 and GCS sources
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8629:
URL: https://github.com/apache/hudi/pull/8629#issuecomment-1535764299
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "653ccdca8099c2729b6afb415277916ff5e08e16",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16818",
"triggerID" : "653ccdca8099c2729b6afb415277916ff5e08e16",
"triggerType" : "PUSH"
}, {
"hash" : "ba0e8193c6ed04723c709b57b83c57280449f681",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16835",
"triggerID" : "ba0e8193c6ed04723c709b57b83c57280449f681",
"triggerType" : "PUSH"
}, {
"hash" : "ba0e8193c6ed04723c709b57b83c57280449f681",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16855",
"triggerID" : "1535723560",
"triggerType" : "MANUAL"
}, {
"hash" : "ae779fccf9f8e511c0cc42d0ef37ea80db16627f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16857",
"triggerID" : "ae779fccf9f8e511c0cc42d0ef37ea80db16627f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ba0e8193c6ed04723c709b57b83c57280449f681 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16835) Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16855)
* ae779fccf9f8e511c0cc42d0ef37ea80db16627f Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16857)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on a diff in pull request #8629: [HUDI-6168] Add ability to parse partition value into row for S3 and GCS sources
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #8629:
URL: https://github.com/apache/hudi/pull/8629#discussion_r1192889237
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/CloudStoreIngestionConfig.java:
##########
@@ -103,4 +103,8 @@ public class CloudStoreIngestionConfig {
@Deprecated
public static final String DATAFILE_FORMAT = CloudSourceConfig.DATAFILE_FORMAT.key();
+ /**
+ * A comma delimited list of path-based partition fields in the source file structure
+ */
+ public static final String PATH_BASED_PARTITION_FIELDS = "hoodie.deltastreamer.source.cloud.data.partition.fields.from.path";
Review Comment:
We should add new configs in `CloudSourceConfig`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org