You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "parisni (via GitHub)" <gi...@apache.org> on 2023/06/10 14:45:07 UTC
[GitHub] [hudi] parisni opened a new pull request, #8929: [HUDI-6350] Allow athena to use the metadata table
parisni opened a new pull request, #8929:
URL: https://github.com/apache/hudi/pull/8929
### Change Logs
_Describe context and summary for this change. Highlight if any code was copied._
### Impact
_Describe any public API or user-facing feature change or any performance impact._
### Risk level (write none, low medium or high below)
_If medium or high, explain what verification was done to mitigate the risks._
### Documentation Update
_Describe any necessary documentation update if there is any new feature, config, or user-facing change_
- _The config description must be updated if new configs are added or the default value of the configs are changed_
- _Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
ticket number here and follow the [instruction](https://hudi.apache.org/contribute/developer-setup#website) to make
changes to the website._
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 closed pull request #8929: [HUDI-6350] Allow athena to use the metadata table
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 closed pull request #8929: [HUDI-6350] Allow athena to use the metadata table
URL: https://github.com/apache/hudi/pull/8929
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] parisni commented on a diff in pull request #8929: [HUDI-6350] Allow athena to use the metadata table
Posted by "parisni (via GitHub)" <gi...@apache.org>.
parisni commented on code in PR #8929:
URL: https://github.com/apache/hudi/pull/8929#discussion_r1302088139
##########
hudi-aws/src/main/java/org/apache/hudi/aws/sync/AWSGlueCatalogSyncClient.java:
##########
@@ -91,16 +92,23 @@ public class AWSGlueCatalogSyncClient extends HoodieSyncClient {
private static final Logger LOG = LoggerFactory.getLogger(AWSGlueCatalogSyncClient.class);
private static final int MAX_PARTITIONS_PER_REQUEST = 100;
private static final long BATCH_REQUEST_SLEEP_MILLIS = 1000L;
+ /**
+ * athena v2/v3 table property
+ * see https://docs.aws.amazon.com/athena/latest/ug/querying-hudi.html
+ */
+ private static final String ENABLE_MDT_LISTING = "hudi.metadata-listing-enabled";
private final AWSGlue awsGlue;
private final String databaseName;
private final Boolean skipTableArchive;
+ private final String enableMetadataTable;
public AWSGlueCatalogSyncClient(HiveSyncConfig config) {
super(config);
this.awsGlue = AWSGlueClientBuilder.standard().build();
this.databaseName = config.getStringOrDefault(META_SYNC_DATABASE_NAME);
this.skipTableArchive = config.getBooleanOrDefault(GlueCatalogSyncClientConfig.GLUE_SKIP_TABLE_ARCHIVE);
+ this.enableMetadataTable = Boolean.toString(config.getBoolean(GLUE_METADATA_FILE_LISTING)).toUpperCase();
Review Comment:
`hoodie.meta.sync.metadata_file_listing` is useful during hive sync.
This PR is useful during Athena reads.
Then this is not related
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #8929: [HUDI-6350] Allow athena to use the metadata table
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #8929:
URL: https://github.com/apache/hudi/pull/8929#discussion_r1228876173
##########
hudi-aws/src/main/java/org/apache/hudi/config/GlueCatalogSyncClientConfig.java:
##########
@@ -39,4 +39,11 @@ public class GlueCatalogSyncClientConfig extends HoodieConfig {
.markAdvanced()
.sinceVersion("0.14.0")
.withDocumentation("Glue catalog sync based client will skip archiving the table version if this config is set to true");
+
+ public static final ConfigProperty<Boolean> GLUE_ENABLE_METADATA_TABLE = ConfigProperty
+ .key(GLUE_CLIENT_PROPERTY_PREFIX + "enable_metadata_table")
Review Comment:
general looks good, only one question: where does these option naming convention come from?
##########
hudi-aws/src/main/java/org/apache/hudi/config/GlueCatalogSyncClientConfig.java:
##########
@@ -39,4 +39,11 @@ public class GlueCatalogSyncClientConfig extends HoodieConfig {
.markAdvanced()
.sinceVersion("0.14.0")
.withDocumentation("Glue catalog sync based client will skip archiving the table version if this config is set to true");
+
+ public static final ConfigProperty<Boolean> GLUE_ENABLE_METADATA_TABLE = ConfigProperty
+ .key(GLUE_CLIENT_PROPERTY_PREFIX + "enable_metadata_table")
Review Comment:
generally looks good, only one question: where does these option naming convention come from?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] parisni commented on a diff in pull request #8929: [HUDI-6350] Allow athena to use the metadata table
Posted by "parisni (via GitHub)" <gi...@apache.org>.
parisni commented on code in PR #8929:
URL: https://github.com/apache/hudi/pull/8929#discussion_r1230805180
##########
hudi-aws/src/main/java/org/apache/hudi/aws/sync/AWSGlueCatalogSyncClient.java:
##########
@@ -89,16 +90,19 @@ public class AWSGlueCatalogSyncClient extends HoodieSyncClient {
private static final Logger LOG = LoggerFactory.getLogger(AWSGlueCatalogSyncClient.class);
private static final int MAX_PARTITIONS_PER_REQUEST = 100;
private static final long BATCH_REQUEST_SLEEP_MILLIS = 1000L;
+ private static final String ENABLE_MDT_LISTING = "hudi.metadata-listing-enabled";
Review Comment:
All details in jira
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on a diff in pull request #8929: [HUDI-6350] Allow athena to use the metadata table
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #8929:
URL: https://github.com/apache/hudi/pull/8929#discussion_r1231270029
##########
hudi-aws/src/main/java/org/apache/hudi/aws/sync/AWSGlueCatalogSyncClient.java:
##########
@@ -89,16 +90,19 @@ public class AWSGlueCatalogSyncClient extends HoodieSyncClient {
private static final Logger LOG = LoggerFactory.getLogger(AWSGlueCatalogSyncClient.class);
private static final int MAX_PARTITIONS_PER_REQUEST = 100;
private static final long BATCH_REQUEST_SLEEP_MILLIS = 1000L;
+ private static final String ENABLE_MDT_LISTING = "hudi.metadata-listing-enabled";
Review Comment:
@parisni Let's add the javadocs here in the code as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8929: [HUDI-6350] Allow athena to use the metadata table
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8929:
URL: https://github.com/apache/hudi/pull/8929#issuecomment-1593412933
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "95ea08e65dc471263b6340e14ef71db0576d1ea2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17728",
"triggerID" : "95ea08e65dc471263b6340e14ef71db0576d1ea2",
"triggerType" : "PUSH"
}, {
"hash" : "7d809746f9d7c09c769d00e8c044d70a2bc4a782",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17774",
"triggerID" : "7d809746f9d7c09c769d00e8c044d70a2bc4a782",
"triggerType" : "PUSH"
}, {
"hash" : "828e8dbf2f46591f92c38d7cf8e2f9f5905af561",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17887",
"triggerID" : "828e8dbf2f46591f92c38d7cf8e2f9f5905af561",
"triggerType" : "PUSH"
}, {
"hash" : "2a0b024a8c3a088ce44be9328ace1bad8648eea2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2a0b024a8c3a088ce44be9328ace1bad8648eea2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7d809746f9d7c09c769d00e8c044d70a2bc4a782 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17774)
* 828e8dbf2f46591f92c38d7cf8e2f9f5905af561 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17887)
* 2a0b024a8c3a088ce44be9328ace1bad8648eea2 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #8929: [HUDI-6350] Allow athena to use the metadata table
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #8929:
URL: https://github.com/apache/hudi/pull/8929#discussion_r1229812837
##########
hudi-aws/src/main/java/org/apache/hudi/aws/sync/AWSGlueCatalogSyncClient.java:
##########
@@ -285,6 +290,7 @@ public void createTable(String tableName,
if (!config.getBoolean(HIVE_CREATE_MANAGED_TABLE)) {
params.put("EXTERNAL", "TRUE");
}
+ params.put(ENABLE_MDT_LISTING, this.enableMetadataTable);
params.putAll(tableProperties);
Review Comment:
I'm assuming you have tested it locally right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8929: [HUDI-6350] Allow athena to use the metadata table
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8929:
URL: https://github.com/apache/hudi/pull/8929#issuecomment-1587605410
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "95ea08e65dc471263b6340e14ef71db0576d1ea2",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17728",
"triggerID" : "95ea08e65dc471263b6340e14ef71db0576d1ea2",
"triggerType" : "PUSH"
}, {
"hash" : "7d809746f9d7c09c769d00e8c044d70a2bc4a782",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17774",
"triggerID" : "7d809746f9d7c09c769d00e8c044d70a2bc4a782",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 95ea08e65dc471263b6340e14ef71db0576d1ea2 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17728)
* 7d809746f9d7c09c769d00e8c044d70a2bc4a782 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17774)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8929: [HUDI-6350] Allow athena to use the metadata table
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8929:
URL: https://github.com/apache/hudi/pull/8929#issuecomment-1585708861
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "95ea08e65dc471263b6340e14ef71db0576d1ea2",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17728",
"triggerID" : "95ea08e65dc471263b6340e14ef71db0576d1ea2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 95ea08e65dc471263b6340e14ef71db0576d1ea2 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17728)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] parisni commented on a diff in pull request #8929: [HUDI-6350] Allow athena to use the metadata table
Posted by "parisni (via GitHub)" <gi...@apache.org>.
parisni commented on code in PR #8929:
URL: https://github.com/apache/hudi/pull/8929#discussion_r1229694714
##########
hudi-aws/src/main/java/org/apache/hudi/aws/sync/AWSGlueCatalogSyncClient.java:
##########
@@ -285,6 +290,7 @@ public void createTable(String tableName,
if (!config.getBoolean(HIVE_CREATE_MANAGED_TABLE)) {
params.put("EXTERNAL", "TRUE");
}
+ params.put(ENABLE_MDT_LISTING, this.enableMetadataTable);
params.putAll(tableProperties);
Review Comment:
We dont have any test on the aws glue sync because there is no way out there to mock the glue backend locally
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8929: [HUDI-6350] Allow athena to use the metadata table
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8929:
URL: https://github.com/apache/hudi/pull/8929#issuecomment-1593203848
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "95ea08e65dc471263b6340e14ef71db0576d1ea2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17728",
"triggerID" : "95ea08e65dc471263b6340e14ef71db0576d1ea2",
"triggerType" : "PUSH"
}, {
"hash" : "7d809746f9d7c09c769d00e8c044d70a2bc4a782",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17774",
"triggerID" : "7d809746f9d7c09c769d00e8c044d70a2bc4a782",
"triggerType" : "PUSH"
}, {
"hash" : "828e8dbf2f46591f92c38d7cf8e2f9f5905af561",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "828e8dbf2f46591f92c38d7cf8e2f9f5905af561",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7d809746f9d7c09c769d00e8c044d70a2bc4a782 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17774)
* 828e8dbf2f46591f92c38d7cf8e2f9f5905af561 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8929: [HUDI-6350] Allow athena to use the metadata table
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8929:
URL: https://github.com/apache/hudi/pull/8929#issuecomment-1587955091
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "95ea08e65dc471263b6340e14ef71db0576d1ea2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17728",
"triggerID" : "95ea08e65dc471263b6340e14ef71db0576d1ea2",
"triggerType" : "PUSH"
}, {
"hash" : "7d809746f9d7c09c769d00e8c044d70a2bc4a782",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17774",
"triggerID" : "7d809746f9d7c09c769d00e8c044d70a2bc4a782",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7d809746f9d7c09c769d00e8c044d70a2bc4a782 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17774)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8929: [HUDI-6350] Allow athena to use the metadata table
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8929:
URL: https://github.com/apache/hudi/pull/8929#issuecomment-1587592087
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "95ea08e65dc471263b6340e14ef71db0576d1ea2",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17728",
"triggerID" : "95ea08e65dc471263b6340e14ef71db0576d1ea2",
"triggerType" : "PUSH"
}, {
"hash" : "7d809746f9d7c09c769d00e8c044d70a2bc4a782",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7d809746f9d7c09c769d00e8c044d70a2bc4a782",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 95ea08e65dc471263b6340e14ef71db0576d1ea2 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17728)
* 7d809746f9d7c09c769d00e8c044d70a2bc4a782 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8929: [HUDI-6350] Allow athena to use the metadata table
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8929:
URL: https://github.com/apache/hudi/pull/8929#issuecomment-1593222065
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "95ea08e65dc471263b6340e14ef71db0576d1ea2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17728",
"triggerID" : "95ea08e65dc471263b6340e14ef71db0576d1ea2",
"triggerType" : "PUSH"
}, {
"hash" : "7d809746f9d7c09c769d00e8c044d70a2bc4a782",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17774",
"triggerID" : "7d809746f9d7c09c769d00e8c044d70a2bc4a782",
"triggerType" : "PUSH"
}, {
"hash" : "828e8dbf2f46591f92c38d7cf8e2f9f5905af561",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17887",
"triggerID" : "828e8dbf2f46591f92c38d7cf8e2f9f5905af561",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7d809746f9d7c09c769d00e8c044d70a2bc4a782 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17774)
* 828e8dbf2f46591f92c38d7cf8e2f9f5905af561 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17887)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #8929: [HUDI-6350] Allow athena to use the metadata table
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #8929:
URL: https://github.com/apache/hudi/pull/8929#discussion_r1229359821
##########
hudi-aws/src/main/java/org/apache/hudi/aws/sync/AWSGlueCatalogSyncClient.java:
##########
@@ -89,16 +90,19 @@ public class AWSGlueCatalogSyncClient extends HoodieSyncClient {
private static final Logger LOG = LoggerFactory.getLogger(AWSGlueCatalogSyncClient.class);
private static final int MAX_PARTITIONS_PER_REQUEST = 100;
private static final long BATCH_REQUEST_SLEEP_MILLIS = 1000L;
+ private static final String ENABLE_MDT_LISTING = "hudi.metadata-listing-enabled";
private final AWSGlue awsGlue;
Review Comment:
Where does this option come from, I didn't see any definitions in Hudi.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] parisni commented on pull request #8929: [HUDI-6350] Allow athena to use the metadata table
Posted by "parisni (via GitHub)" <gi...@apache.org>.
parisni commented on PR #8929:
URL: https://github.com/apache/hudi/pull/8929#issuecomment-1585695255
maybe @nsivabalan ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] parisni commented on pull request #8929: [HUDI-6350] Allow athena to use the metadata table
Posted by "parisni (via GitHub)" <gi...@apache.org>.
parisni commented on PR #8929:
URL: https://github.com/apache/hudi/pull/8929#issuecomment-1593129622
Debased + conf renaming
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] CTTY commented on pull request #8929: [HUDI-6350] Allow athena to use the metadata table
Posted by "CTTY (via GitHub)" <gi...@apache.org>.
CTTY commented on PR #8929:
URL: https://github.com/apache/hudi/pull/8929#issuecomment-1700184768
I've manually tested this on EMR 6.12 by setting configs below when writing data:
```
.option(DataSourceWriteOptions.META_SYNC_CLIENT_TOOL_CLASS_NAME.key, "org.apache.hudi.aws.sync.AwsGlueCatalogSyncTool")
.option(GlueCatalogSyncClientConfig.GLUE_METADATA_FILE_LISTING.key, "TRUE")
```
It works very well. Nice work!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8929: [HUDI-6350] Allow athena to use the metadata table
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8929:
URL: https://github.com/apache/hudi/pull/8929#issuecomment-1585761815
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "95ea08e65dc471263b6340e14ef71db0576d1ea2",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17728",
"triggerID" : "95ea08e65dc471263b6340e14ef71db0576d1ea2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 95ea08e65dc471263b6340e14ef71db0576d1ea2 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17728)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8929: [HUDI-6350] Allow athena to use the metadata table
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8929:
URL: https://github.com/apache/hudi/pull/8929#issuecomment-1593490196
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "95ea08e65dc471263b6340e14ef71db0576d1ea2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17728",
"triggerID" : "95ea08e65dc471263b6340e14ef71db0576d1ea2",
"triggerType" : "PUSH"
}, {
"hash" : "7d809746f9d7c09c769d00e8c044d70a2bc4a782",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17774",
"triggerID" : "7d809746f9d7c09c769d00e8c044d70a2bc4a782",
"triggerType" : "PUSH"
}, {
"hash" : "828e8dbf2f46591f92c38d7cf8e2f9f5905af561",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17887",
"triggerID" : "828e8dbf2f46591f92c38d7cf8e2f9f5905af561",
"triggerType" : "PUSH"
}, {
"hash" : "2a0b024a8c3a088ce44be9328ace1bad8648eea2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2a0b024a8c3a088ce44be9328ace1bad8648eea2",
"triggerType" : "PUSH"
}, {
"hash" : "6db6c357d7d1c27fc171da68f59216e2c1df772e",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17893",
"triggerID" : "6db6c357d7d1c27fc171da68f59216e2c1df772e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 828e8dbf2f46591f92c38d7cf8e2f9f5905af561 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17887)
* 2a0b024a8c3a088ce44be9328ace1bad8648eea2 UNKNOWN
* 6db6c357d7d1c27fc171da68f59216e2c1df772e Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17893)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8929: [HUDI-6350] Allow athena to use the metadata table
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8929:
URL: https://github.com/apache/hudi/pull/8929#issuecomment-1593432980
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "95ea08e65dc471263b6340e14ef71db0576d1ea2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17728",
"triggerID" : "95ea08e65dc471263b6340e14ef71db0576d1ea2",
"triggerType" : "PUSH"
}, {
"hash" : "7d809746f9d7c09c769d00e8c044d70a2bc4a782",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17774",
"triggerID" : "7d809746f9d7c09c769d00e8c044d70a2bc4a782",
"triggerType" : "PUSH"
}, {
"hash" : "828e8dbf2f46591f92c38d7cf8e2f9f5905af561",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17887",
"triggerID" : "828e8dbf2f46591f92c38d7cf8e2f9f5905af561",
"triggerType" : "PUSH"
}, {
"hash" : "2a0b024a8c3a088ce44be9328ace1bad8648eea2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2a0b024a8c3a088ce44be9328ace1bad8648eea2",
"triggerType" : "PUSH"
}, {
"hash" : "6db6c357d7d1c27fc171da68f59216e2c1df772e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6db6c357d7d1c27fc171da68f59216e2c1df772e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 828e8dbf2f46591f92c38d7cf8e2f9f5905af561 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17887)
* 2a0b024a8c3a088ce44be9328ace1bad8648eea2 UNKNOWN
* 6db6c357d7d1c27fc171da68f59216e2c1df772e UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on a diff in pull request #8929: [HUDI-6350] Allow athena to use the metadata table
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #8929:
URL: https://github.com/apache/hudi/pull/8929#discussion_r1230294848
##########
hudi-aws/src/main/java/org/apache/hudi/aws/sync/AWSGlueCatalogSyncClient.java:
##########
@@ -89,16 +90,19 @@ public class AWSGlueCatalogSyncClient extends HoodieSyncClient {
private static final Logger LOG = LoggerFactory.getLogger(AWSGlueCatalogSyncClient.class);
private static final int MAX_PARTITIONS_PER_REQUEST = 100;
private static final long BATCH_REQUEST_SLEEP_MILLIS = 1000L;
+ private static final String ENABLE_MDT_LISTING = "hudi.metadata-listing-enabled";
Review Comment:
Is this based on Presto or Trino connector configuration? How does it enable Athena to use the metadata table for file listing? Could you add docs and references here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] parisni commented on a diff in pull request #8929: [HUDI-6350] Allow athena to use the metadata table
Posted by "parisni (via GitHub)" <gi...@apache.org>.
parisni commented on code in PR #8929:
URL: https://github.com/apache/hudi/pull/8929#discussion_r1231297813
##########
hudi-aws/src/main/java/org/apache/hudi/aws/sync/AWSGlueCatalogSyncClient.java:
##########
@@ -89,16 +90,19 @@ public class AWSGlueCatalogSyncClient extends HoodieSyncClient {
private static final Logger LOG = LoggerFactory.getLogger(AWSGlueCatalogSyncClient.class);
private static final int MAX_PARTITIONS_PER_REQUEST = 100;
private static final long BATCH_REQUEST_SLEEP_MILLIS = 1000L;
+ private static final String ENABLE_MDT_LISTING = "hudi.metadata-listing-enabled";
Review Comment:
@yihua added details
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on pull request #8929: [HUDI-6350] Allow athena to use the metadata table
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on PR #8929:
URL: https://github.com/apache/hudi/pull/8929#issuecomment-1592848445
@parisni Can you rebase with the latest master to fix the compile error?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan commented on a diff in pull request #8929: [HUDI-6350] Allow athena to use the metadata table
Posted by "xushiyan (via GitHub)" <gi...@apache.org>.
xushiyan commented on code in PR #8929:
URL: https://github.com/apache/hudi/pull/8929#discussion_r1302429672
##########
hudi-aws/src/main/java/org/apache/hudi/aws/sync/AWSGlueCatalogSyncClient.java:
##########
@@ -91,16 +92,23 @@ public class AWSGlueCatalogSyncClient extends HoodieSyncClient {
private static final Logger LOG = LoggerFactory.getLogger(AWSGlueCatalogSyncClient.class);
private static final int MAX_PARTITIONS_PER_REQUEST = 100;
private static final long BATCH_REQUEST_SLEEP_MILLIS = 1000L;
+ /**
+ * athena v2/v3 table property
+ * see https://docs.aws.amazon.com/athena/latest/ug/querying-hudi.html
+ */
+ private static final String ENABLE_MDT_LISTING = "hudi.metadata-listing-enabled";
private final AWSGlue awsGlue;
private final String databaseName;
private final Boolean skipTableArchive;
+ private final String enableMetadataTable;
public AWSGlueCatalogSyncClient(HiveSyncConfig config) {
super(config);
this.awsGlue = AWSGlueClientBuilder.standard().build();
this.databaseName = config.getStringOrDefault(META_SYNC_DATABASE_NAME);
this.skipTableArchive = config.getBooleanOrDefault(GlueCatalogSyncClientConfig.GLUE_SKIP_TABLE_ARCHIVE);
+ this.enableMetadataTable = Boolean.toString(config.getBoolean(GLUE_METADATA_FILE_LISTING)).toUpperCase();
Review Comment:
right `hoodie.meta.sync.metadata_file_listing` is a separate config. my main idea is to infer it from the master flag `hoodie.metadata.enable`, just like for all readers. in `BaseHoodieTableFileIndex`, we check the master flag with a dedicated default value `DEFAULT_METADATA_ENABLE_FOR_READERS`, which all readers should follow. There are too many configs already we should avoid unnecessary extra configs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8929: [HUDI-6350] Allow athena to use the metadata table
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8929:
URL: https://github.com/apache/hudi/pull/8929#issuecomment-1593702638
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "95ea08e65dc471263b6340e14ef71db0576d1ea2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17728",
"triggerID" : "95ea08e65dc471263b6340e14ef71db0576d1ea2",
"triggerType" : "PUSH"
}, {
"hash" : "7d809746f9d7c09c769d00e8c044d70a2bc4a782",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17774",
"triggerID" : "7d809746f9d7c09c769d00e8c044d70a2bc4a782",
"triggerType" : "PUSH"
}, {
"hash" : "828e8dbf2f46591f92c38d7cf8e2f9f5905af561",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17887",
"triggerID" : "828e8dbf2f46591f92c38d7cf8e2f9f5905af561",
"triggerType" : "PUSH"
}, {
"hash" : "2a0b024a8c3a088ce44be9328ace1bad8648eea2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2a0b024a8c3a088ce44be9328ace1bad8648eea2",
"triggerType" : "PUSH"
}, {
"hash" : "6db6c357d7d1c27fc171da68f59216e2c1df772e",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17893",
"triggerID" : "6db6c357d7d1c27fc171da68f59216e2c1df772e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2a0b024a8c3a088ce44be9328ace1bad8648eea2 UNKNOWN
* 6db6c357d7d1c27fc171da68f59216e2c1df772e Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17893)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 merged pull request #8929: [HUDI-6350] Allow athena to use the metadata table
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 merged PR #8929:
URL: https://github.com/apache/hudi/pull/8929
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8929: [HUDI-6350] Allow athena to use the metadata table
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8929:
URL: https://github.com/apache/hudi/pull/8929#issuecomment-1585705583
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "95ea08e65dc471263b6340e14ef71db0576d1ea2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "95ea08e65dc471263b6340e14ef71db0576d1ea2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 95ea08e65dc471263b6340e14ef71db0576d1ea2 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] parisni commented on pull request #8929: [HUDI-6350] Allow athena to use the metadata table
Posted by "parisni (via GitHub)" <gi...@apache.org>.
parisni commented on PR #8929:
URL: https://github.com/apache/hudi/pull/8929#issuecomment-1590530852
Please take a look at thé jira ticket for details
On June 14, 2023 1:41:44 AM UTC, Danny Chan ***@***.***> wrote:
>@danny0405 commented on this pull request.
>
>
>
>> @@ -39,4 +39,11 @@ public class GlueCatalogSyncClientConfig extends HoodieConfig {
> .markAdvanced()
> .sinceVersion("0.14.0")
> .withDocumentation("Glue catalog sync based client will skip archiving the table version if this config is set to true");
>+
>+ public static final ConfigProperty<Boolean> GLUE_ENABLE_METADATA_TABLE = ConfigProperty
>+ .key(GLUE_CLIENT_PROPERTY_PREFIX + "enable_metadata_table")
>
>generally looks good, only one question: where does these option naming convention come from?
>
>--
>Reply to this email directly or view it on GitHub:
>https://github.com/apache/hudi/pull/8929#pullrequestreview-1478307094
>You are receiving this because you authored the thread.
>
>Message ID: ***@***.***>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #8929: [HUDI-6350] Allow athena to use the metadata table
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #8929:
URL: https://github.com/apache/hudi/pull/8929#discussion_r1229360141
##########
hudi-aws/src/main/java/org/apache/hudi/aws/sync/AWSGlueCatalogSyncClient.java:
##########
@@ -285,6 +290,7 @@ public void createTable(String tableName,
if (!config.getBoolean(HIVE_CREATE_MANAGED_TABLE)) {
params.put("EXTERNAL", "TRUE");
}
+ params.put(ENABLE_MDT_LISTING, this.enableMetadataTable);
params.putAll(tableProperties);
Review Comment:
Can we write a test case for it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] parisni commented on a diff in pull request #8929: [HUDI-6350] Allow athena to use the metadata table
Posted by "parisni (via GitHub)" <gi...@apache.org>.
parisni commented on code in PR #8929:
URL: https://github.com/apache/hudi/pull/8929#discussion_r1229958800
##########
hudi-aws/src/main/java/org/apache/hudi/aws/sync/AWSGlueCatalogSyncClient.java:
##########
@@ -285,6 +290,7 @@ public void createTable(String tableName,
if (!config.getBoolean(HIVE_CREATE_MANAGED_TABLE)) {
params.put("EXTERNAL", "TRUE");
}
+ params.put(ENABLE_MDT_LISTING, this.enableMetadataTable);
params.putAll(tableProperties);
Review Comment:
Yes we did test locally until success
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan commented on a diff in pull request #8929: [HUDI-6350] Allow athena to use the metadata table
Posted by "xushiyan (via GitHub)" <gi...@apache.org>.
xushiyan commented on code in PR #8929:
URL: https://github.com/apache/hudi/pull/8929#discussion_r1299845069
##########
hudi-aws/src/main/java/org/apache/hudi/aws/sync/AWSGlueCatalogSyncClient.java:
##########
@@ -91,16 +92,23 @@ public class AWSGlueCatalogSyncClient extends HoodieSyncClient {
private static final Logger LOG = LoggerFactory.getLogger(AWSGlueCatalogSyncClient.class);
private static final int MAX_PARTITIONS_PER_REQUEST = 100;
private static final long BATCH_REQUEST_SLEEP_MILLIS = 1000L;
+ /**
+ * athena v2/v3 table property
+ * see https://docs.aws.amazon.com/athena/latest/ug/querying-hudi.html
+ */
+ private static final String ENABLE_MDT_LISTING = "hudi.metadata-listing-enabled";
private final AWSGlue awsGlue;
private final String databaseName;
private final Boolean skipTableArchive;
+ private final String enableMetadataTable;
public AWSGlueCatalogSyncClient(HiveSyncConfig config) {
super(config);
this.awsGlue = AWSGlueClientBuilder.standard().build();
this.databaseName = config.getStringOrDefault(META_SYNC_DATABASE_NAME);
this.skipTableArchive = config.getBooleanOrDefault(GlueCatalogSyncClientConfig.GLUE_SKIP_TABLE_ARCHIVE);
+ this.enableMetadataTable = Boolean.toString(config.getBoolean(GLUE_METADATA_FILE_LISTING)).toUpperCase();
Review Comment:
we can avoid the extra Glue-specific config by reading value of `hoodie.meta.sync.metadata_file_listing`, which derives the enable flag from the master config `hoodie.metadata.enable`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org