You are viewing a plain text version of this content. The canonical link for it is here.
Posted to gitbox@hive.apache.org by "ayushtkn (via GitHub)" <gi...@apache.org> on 2023/09/07 17:35:41 UTC
[GitHub] [hive] ayushtkn opened a new pull request, #4673: HIVE-27322: Iceberg: metadata location overrides can cause data breach.
ayushtkn opened a new pull request, #4673:
URL: https://github.com/apache/hive/pull/4673
### What changes were proposed in this pull request?
Pass metadata location to authorizer if that is provided as part of create table
### Why are the changes needed?
Better Authorizaton checks
### Does this PR introduce _any_ user-facing change?
Yes, metadata location will go through auth checks while creating table if they are specified.
### Is the change a dependency upgrade?
No
### How was this patch tested?
UT
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] ayushtkn commented on a diff in pull request #4673: HIVE-27322: Iceberg: metadata location overrides can cause data breach.
Posted by "ayushtkn (via GitHub)" <gi...@apache.org>.
ayushtkn commented on code in PR #4673:
URL: https://github.com/apache/hive/pull/4673#discussion_r1321630264
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java:
##########
@@ -926,18 +926,25 @@ public URI getURIForAuth(org.apache.hadoop.hive.metastore.api.Table hmsTable) th
StringBuilder authURI =
new StringBuilder(ICEBERG_URI_PREFIX).append(encodeString(dbName)).append("/").append(encodeString(tableName))
.append("?snapshot=");
- Optional<String> locationProperty = SessionStateUtil.getProperty(conf, hive_metastoreConstants.META_TABLE_LOCATION);
- if (locationProperty.isPresent()) {
- Preconditions.checkArgument(locationProperty.get() != null,
- "Table location is not set in SessionState. Authorization URI cannot be supplied.");
- // this property is set during the create operation before the hive table was created
- // we are returning a dummy iceberg metadata file
- authURI.append(encodeString(URI.create(locationProperty.get()).getPath()))
- .append(encodeString("/metadata/dummy.metadata.json"));
+ // If metadata location is provided we should use that location for auth, since during create if the
+ // metadata_location is explicitly provided we register a table using that path.
+ Optional<String> metadataLocation =
+ SessionStateUtil.getProperty(conf, BaseMetastoreTableOperations.METADATA_LOCATION_PROP);
+ if (metadataLocation.isPresent()) {
+ authURI.append(encodeString(metadataLocation.get()));
} else {
- Table table = IcebergTableUtil.getTable(conf, hmsTable);
- authURI.append(
- encodeString(URI.create(((BaseTable) table).operations().current().metadataFileLocation()).getPath()));
+ Optional<String> locationProperty =
+ SessionStateUtil.getProperty(conf, hive_metastoreConstants.META_TABLE_LOCATION);
+ if (locationProperty.isPresent()) {
+ // this property is set during the create operation before the hive table was created
+ // we are returning a dummy iceberg metadata file
+ authURI.append(encodeString(URI.create(locationProperty.get()).getPath()))
+ .append(encodeString("/metadata/dummy.metadata.json"));
Review Comment:
old code, shows up in refactor, when location is provided in create table, the metadata.json file will be created in that location /metadata/some.json, we don't know the exact path, so we pass this dummy.json to check if we would be able to create a file inside /metadata directory, the name of the file is irrelevant here, as what I could decode from history
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] aturoczy commented on a diff in pull request #4673: HIVE-27322: Iceberg: metadata location overrides can cause data breach.
Posted by "aturoczy (via GitHub)" <gi...@apache.org>.
aturoczy commented on code in PR #4673:
URL: https://github.com/apache/hive/pull/4673#discussion_r1318950652
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java:
##########
@@ -1537,6 +1544,15 @@ public boolean shouldOverwrite(org.apache.hadoop.hive.ql.metadata.Table mTable,
return COPY_ON_WRITE.equalsIgnoreCase(mode);
}
+ @Override
+ public void addResourcesForCreateTable(Map<String, String> tblProps, HiveConf hiveConf) {
+ String metadataLocation = tblProps.get(BaseMetastoreTableOperations.METADATA_LOCATION_PROP);
+ if (StringUtils.isNotEmpty(metadataLocation)) {
Review Comment:
Would it not be simpler if the metadataLocation is empty than throw exception?
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java:
##########
@@ -926,18 +926,25 @@ public URI getURIForAuth(org.apache.hadoop.hive.metastore.api.Table hmsTable) th
StringBuilder authURI =
new StringBuilder(ICEBERG_URI_PREFIX).append(encodeString(dbName)).append("/").append(encodeString(tableName))
.append("?snapshot=");
- Optional<String> locationProperty = SessionStateUtil.getProperty(conf, hive_metastoreConstants.META_TABLE_LOCATION);
- if (locationProperty.isPresent()) {
- Preconditions.checkArgument(locationProperty.get() != null,
- "Table location is not set in SessionState. Authorization URI cannot be supplied.");
- // this property is set during the create operation before the hive table was created
- // we are returning a dummy iceberg metadata file
- authURI.append(encodeString(URI.create(locationProperty.get()).getPath()))
- .append(encodeString("/metadata/dummy.metadata.json"));
+ // If metadata location is provided we should use that location for auth, since during create if the
+ // metadata_location is explicitly provided we register a table using that path.
+ Optional<String> metadataLocation =
+ SessionStateUtil.getProperty(conf, BaseMetastoreTableOperations.METADATA_LOCATION_PROP);
+ if (metadataLocation.isPresent()) {
+ authURI.append(encodeString(metadataLocation.get()));
} else {
- Table table = IcebergTableUtil.getTable(conf, hmsTable);
- authURI.append(
- encodeString(URI.create(((BaseTable) table).operations().current().metadataFileLocation()).getPath()));
+ Optional<String> locationProperty =
+ SessionStateUtil.getProperty(conf, hive_metastoreConstants.META_TABLE_LOCATION);
+ if (locationProperty.isPresent()) {
+ // this property is set during the create operation before the hive table was created
+ // we are returning a dummy iceberg metadata file
+ authURI.append(encodeString(URI.create(locationProperty.get()).getPath()))
+ .append(encodeString("/metadata/dummy.metadata.json"));
Review Comment:
Why this dummy needs?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] sonarcloud[bot] commented on pull request #4673: HIVE-27322: Iceberg: metadata location overrides can cause data breach.
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #4673:
URL: https://github.com/apache/hive/pull/4673#issuecomment-1710826989
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4673)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4673&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4673&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4673&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4673&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4673&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4673&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4673&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4673&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4673&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4673&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4673&resolved=false&types=CODE_SMELL) [9 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4673&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4673&metric=coverage&view=list) No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4673&metric=duplicated_lines_density&view=list) No Duplication information
 The version of Java (11.0.8) you have used to run this analysis is deprecated and we will stop accepting it soon. Please update to at least Java 17.
Read more [here](https://docs.sonarcloud.io/appendices/scanner-environment/)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] ayushtkn commented on a diff in pull request #4673: HIVE-27322: Iceberg: metadata location overrides can cause data breach.
Posted by "ayushtkn (via GitHub)" <gi...@apache.org>.
ayushtkn commented on code in PR #4673:
URL: https://github.com/apache/hive/pull/4673#discussion_r1321859005
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java:
##########
@@ -1537,6 +1544,15 @@ public boolean shouldOverwrite(org.apache.hadoop.hive.ql.metadata.Table mTable,
return COPY_ON_WRITE.equalsIgnoreCase(mode);
}
+ @Override
+ public void addResourcesForCreateTable(Map<String, String> tblProps, HiveConf hiveConf) {
+ String metadataLocation = tblProps.get(BaseMetastoreTableOperations.METADATA_LOCATION_PROP);
+ if (StringUtils.isNotEmpty(metadataLocation)) {
Review Comment:
may be I can change it to just ``null`` check, if it is empty iceberg should, if it doesn't resolve to anything
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] ayushtkn commented on a diff in pull request #4673: HIVE-27322: Iceberg: metadata location overrides can cause data breach.
Posted by "ayushtkn (via GitHub)" <gi...@apache.org>.
ayushtkn commented on code in PR #4673:
URL: https://github.com/apache/hive/pull/4673#discussion_r1321705921
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java:
##########
@@ -1537,6 +1544,15 @@ public boolean shouldOverwrite(org.apache.hadoop.hive.ql.metadata.Table mTable,
return COPY_ON_WRITE.equalsIgnoreCase(mode);
}
+ @Override
+ public void addResourcesForCreateTable(Map<String, String> tblProps, HiveConf hiveConf) {
+ String metadataLocation = tblProps.get(BaseMetastoreTableOperations.METADATA_LOCATION_PROP);
+ if (StringUtils.isNotEmpty(metadataLocation)) {
Review Comment:
it would be empty in general unless someone provides the the metadata location as part of the create command. like in normal create command it would be empty
create table ice (schema) stored by iceberg;
it would be empty here.
create table ice TBLPROPERTIES(metadata-location='') stored by iceberg;
we handle this case, & if it is provided we do the logic
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] ayushtkn merged pull request #4673: HIVE-27322: Iceberg: metadata location overrides can cause data breach.
Posted by "ayushtkn (via GitHub)" <gi...@apache.org>.
ayushtkn merged PR #4673:
URL: https://github.com/apache/hive/pull/4673
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] sonarcloud[bot] commented on pull request #4673: HIVE-27322: Iceberg: metadata location overrides can cause data breach.
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #4673:
URL: https://github.com/apache/hive/pull/4673#issuecomment-1719174750
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4673)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4673&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4673&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4673&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4673&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4673&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4673&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4673&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4673&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4673&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4673&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4673&resolved=false&types=CODE_SMELL) [9 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4673&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4673&metric=coverage&view=list) No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4673&metric=duplicated_lines_density&view=list) No Duplication information
 The version of Java (11.0.8) you have used to run this analysis is deprecated and we will stop accepting it soon. Please update to at least Java 17.
Read more [here](https://docs.sonarcloud.io/appendices/scanner-environment/)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org