You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2021/03/16 12:32:15 UTC
[GitHub] [iceberg] lcspinter opened a new pull request #2340: Hive: Remove unnecessary SerializationUtil.serializeToBase64 from HiveIcebergStorageHandler.overlayTableProperties
lcspinter opened a new pull request #2340:
URL: https://github.com/apache/iceberg/pull/2340
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] lcspinter commented on a change in pull request #2340: Hive: Remove unnecessary SerializationUtil.serializeToBase64 from HiveIcebergStorageHandler.overlayTableProperties
Posted by GitBox <gi...@apache.org>.
lcspinter commented on a change in pull request #2340:
URL: https://github.com/apache/iceberg/pull/2340#discussion_r597638909
##########
File path: core/src/main/java/org/apache/iceberg/StaticTableOperations.java
##########
@@ -29,24 +29,37 @@
* and cannot be used to create or delete files.
*/
public class StaticTableOperations implements TableOperations {
- private final TableMetadata staticMetadata;
+ private TableMetadata staticMetadata;
+ private final String metadataFileLocation;
private final FileIO io;
+ private final LocationProvider locationProvider;
/**
* Creates a StaticTableOperations tied to a specific static version of the TableMetadata
*/
public StaticTableOperations(String metadataFileLocation, FileIO io) {
+ this(metadataFileLocation, io, null);
+ }
+
+ public StaticTableOperations(String metadataFileLocation, FileIO io, LocationProvider locationProvider) {
this.io = io;
- this.staticMetadata = TableMetadataParser.read(io, metadataFileLocation);
+ this.metadataFileLocation = metadataFileLocation;
+ this.locationProvider = locationProvider;
}
@Override
public TableMetadata current() {
+ if (staticMetadata == null) {
+ staticMetadata = TableMetadataParser.read(io, metadataFileLocation);
+ }
return staticMetadata;
}
@Override
public TableMetadata refresh() {
+ if (staticMetadata == null) {
Review comment:
We have a couple of negative test cases that tries to modify StaticTables. Obviously, the operation will fail with an exception during commit(), but before it does it tries to create a snapshot that calls refresh(). If we return null, we will run into a nullpointer exception.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] lcspinter commented on pull request #2340: Hive: Remove unnecessary SerializationUtil.serializeToBase64 from HiveIcebergStorageHandler.overlayTableProperties
Posted by GitBox <gi...@apache.org>.
lcspinter commented on pull request #2340:
URL: https://github.com/apache/iceberg/pull/2340#issuecomment-800267841
@marton-bod @pvary Could you please review this PR? Thanks
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] lcspinter commented on a change in pull request #2340: Hive: Remove unnecessary SerializationUtil.serializeToBase64 from HiveIcebergStorageHandler.overlayTableProperties
Posted by GitBox <gi...@apache.org>.
lcspinter commented on a change in pull request #2340:
URL: https://github.com/apache/iceberg/pull/2340#discussion_r597618180
##########
File path: mr/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java
##########
@@ -224,24 +185,16 @@ static void overlayTableProperties(Configuration configuration, TableDesc tableD
.filter(entry -> !map.containsKey(entry.getKey())) // map overrides tableDesc properties
.forEach(entry -> map.put(entry.getKey(), entry.getValue()));
- map.put(InputFormatConfig.TABLE_IDENTIFIER, props.getProperty(Catalogs.NAME));
- map.put(InputFormatConfig.TABLE_LOCATION, table.location());
+ String tableIdentifier = props.getProperty(Catalogs.NAME);
Review comment:
Removed variables.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] pvary commented on a change in pull request #2340: Hive: Remove unnecessary SerializationUtil.serializeToBase64 from HiveIcebergStorageHandler.overlayTableProperties
Posted by GitBox <gi...@apache.org>.
pvary commented on a change in pull request #2340:
URL: https://github.com/apache/iceberg/pull/2340#discussion_r597510158
##########
File path: spark/src/main/java/org/apache/iceberg/spark/actions/BaseExpireSnapshotsSparkAction.java
##########
@@ -200,7 +200,7 @@ public BaseExpireSnapshotsSparkAction deleteWith(Consumer<String> newDeleteFunc)
}
private Dataset<Row> buildValidFileDF(TableMetadata metadata) {
- Table staticTable = newStaticTable(metadata, this.table.io());
+ Table staticTable = newStaticTable(metadata, this.table.io(), table.locationProvider());
Review comment:
Since we kept the original constructor we might not need this change now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] pvary merged pull request #2340: Hive: Remove unnecessary SerializationUtil.serializeToBase64 from HiveIcebergStorageHandler.overlayTableProperties
Posted by GitBox <gi...@apache.org>.
pvary merged pull request #2340:
URL: https://github.com/apache/iceberg/pull/2340
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] lcspinter commented on a change in pull request #2340: Hive: Remove unnecessary SerializationUtil.serializeToBase64 from HiveIcebergStorageHandler.overlayTableProperties
Posted by GitBox <gi...@apache.org>.
lcspinter commented on a change in pull request #2340:
URL: https://github.com/apache/iceberg/pull/2340#discussion_r596906499
##########
File path: mr/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java
##########
@@ -239,9 +218,6 @@ static void overlayTableProperties(Configuration configuration, TableDesc tableD
map.put(InputFormatConfig.SERIALIZED_TABLE, SerializationUtil.serializeToBase64(table));
}
- map.put(InputFormatConfig.FILE_IO, SerializationUtil.serializeToBase64(table.io()));
- map.put(InputFormatConfig.LOCATION_PROVIDER, SerializationUtil.serializeToBase64(table.locationProvider()));
- map.put(InputFormatConfig.ENCRYPTION_MANAGER, SerializationUtil.serializeToBase64(table.encryption()));
Review comment:
Removed most of the objects. What remained is the schema and location.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] marton-bod commented on a change in pull request #2340: Hive: Remove unnecessary SerializationUtil.serializeToBase64 from HiveIcebergStorageHandler.overlayTableProperties
Posted by GitBox <gi...@apache.org>.
marton-bod commented on a change in pull request #2340:
URL: https://github.com/apache/iceberg/pull/2340#discussion_r597512668
##########
File path: core/src/main/java/org/apache/iceberg/StaticTableOperations.java
##########
@@ -29,24 +29,37 @@
* and cannot be used to create or delete files.
*/
public class StaticTableOperations implements TableOperations {
- private final TableMetadata staticMetadata;
+ private TableMetadata staticMetadata;
+ private final String metadataFileLocation;
private final FileIO io;
+ private final LocationProvider locationProvider;
/**
* Creates a StaticTableOperations tied to a specific static version of the TableMetadata
*/
public StaticTableOperations(String metadataFileLocation, FileIO io) {
+ this(metadataFileLocation, io, null);
+ }
+
+ public StaticTableOperations(String metadataFileLocation, FileIO io, LocationProvider locationProvider) {
this.io = io;
- this.staticMetadata = TableMetadataParser.read(io, metadataFileLocation);
+ this.metadataFileLocation = metadataFileLocation;
+ this.locationProvider = locationProvider;
}
@Override
public TableMetadata current() {
+ if (staticMetadata == null) {
+ staticMetadata = TableMetadataParser.read(io, metadataFileLocation);
+ }
return staticMetadata;
}
@Override
public TableMetadata refresh() {
+ if (staticMetadata == null) {
Review comment:
Also, I was thinking that not refreshing the metadata on calling `refresh()` might come as a surprise for users. Do we want to either throw unsupported operation exc (probably not preferable), or at least add some short javadoc explaining why the refresh() works differently here than for the other table operations? We might even simply call `current()` here to make it clear it's the same behaviour. Let me know what you guys think
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] pvary commented on a change in pull request #2340: Hive: Remove unnecessary SerializationUtil.serializeToBase64 from HiveIcebergStorageHandler.overlayTableProperties
Posted by GitBox <gi...@apache.org>.
pvary commented on a change in pull request #2340:
URL: https://github.com/apache/iceberg/pull/2340#discussion_r597555828
##########
File path: core/src/main/java/org/apache/iceberg/StaticTableOperations.java
##########
@@ -29,24 +29,37 @@
* and cannot be used to create or delete files.
*/
public class StaticTableOperations implements TableOperations {
- private final TableMetadata staticMetadata;
+ private TableMetadata staticMetadata;
+ private final String metadataFileLocation;
private final FileIO io;
+ private final LocationProvider locationProvider;
/**
* Creates a StaticTableOperations tied to a specific static version of the TableMetadata
*/
public StaticTableOperations(String metadataFileLocation, FileIO io) {
+ this(metadataFileLocation, io, null);
+ }
+
+ public StaticTableOperations(String metadataFileLocation, FileIO io, LocationProvider locationProvider) {
this.io = io;
- this.staticMetadata = TableMetadataParser.read(io, metadataFileLocation);
+ this.metadataFileLocation = metadataFileLocation;
+ this.locationProvider = locationProvider;
}
@Override
public TableMetadata current() {
+ if (staticMetadata == null) {
+ staticMetadata = TableMetadataParser.read(io, metadataFileLocation);
+ }
return staticMetadata;
}
@Override
public TableMetadata refresh() {
+ if (staticMetadata == null) {
Review comment:
Comment is needed, but I think not doing anything is the best option.
Calling `current()` calls metadata parsing, which might be unnecessary if the metadata is not used later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] pvary commented on a change in pull request #2340: Hive: Remove unnecessary SerializationUtil.serializeToBase64 from HiveIcebergStorageHandler.overlayTableProperties
Posted by GitBox <gi...@apache.org>.
pvary commented on a change in pull request #2340:
URL: https://github.com/apache/iceberg/pull/2340#discussion_r597507325
##########
File path: core/src/main/java/org/apache/iceberg/StaticTableOperations.java
##########
@@ -29,24 +29,37 @@
* and cannot be used to create or delete files.
*/
public class StaticTableOperations implements TableOperations {
- private final TableMetadata staticMetadata;
+ private TableMetadata staticMetadata;
+ private final String metadataFileLocation;
private final FileIO io;
+ private final LocationProvider locationProvider;
/**
* Creates a StaticTableOperations tied to a specific static version of the TableMetadata
*/
public StaticTableOperations(String metadataFileLocation, FileIO io) {
+ this(metadataFileLocation, io, null);
+ }
+
+ public StaticTableOperations(String metadataFileLocation, FileIO io, LocationProvider locationProvider) {
this.io = io;
- this.staticMetadata = TableMetadataParser.read(io, metadataFileLocation);
+ this.metadataFileLocation = metadataFileLocation;
+ this.locationProvider = locationProvider;
}
@Override
public TableMetadata current() {
+ if (staticMetadata == null) {
+ staticMetadata = TableMetadataParser.read(io, metadataFileLocation);
+ }
return staticMetadata;
}
@Override
public TableMetadata refresh() {
+ if (staticMetadata == null) {
Review comment:
Do we need to refresh static tables?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] pvary commented on a change in pull request #2340: Hive: Remove unnecessary SerializationUtil.serializeToBase64 from HiveIcebergStorageHandler.overlayTableProperties
Posted by GitBox <gi...@apache.org>.
pvary commented on a change in pull request #2340:
URL: https://github.com/apache/iceberg/pull/2340#discussion_r597509186
##########
File path: mr/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java
##########
@@ -224,24 +185,16 @@ static void overlayTableProperties(Configuration configuration, TableDesc tableD
.filter(entry -> !map.containsKey(entry.getKey())) // map overrides tableDesc properties
.forEach(entry -> map.put(entry.getKey(), entry.getValue()));
- map.put(InputFormatConfig.TABLE_IDENTIFIER, props.getProperty(Catalogs.NAME));
- map.put(InputFormatConfig.TABLE_LOCATION, table.location());
+ String tableIdentifier = props.getProperty(Catalogs.NAME);
Review comment:
Do we need these local variables?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] pvary commented on a change in pull request #2340: Hive: Remove unnecessary SerializationUtil.serializeToBase64 from HiveIcebergStorageHandler.overlayTableProperties
Posted by GitBox <gi...@apache.org>.
pvary commented on a change in pull request #2340:
URL: https://github.com/apache/iceberg/pull/2340#discussion_r595198985
##########
File path: core/src/main/java/org/apache/iceberg/StaticTableOperations.java
##########
@@ -31,15 +31,23 @@
public class StaticTableOperations implements TableOperations {
private final TableMetadata staticMetadata;
private final FileIO io;
+ private final LocationProvider locationProvider;
/**
* Creates a StaticTableOperations tied to a specific static version of the TableMetadata
*/
public StaticTableOperations(String metadataFileLocation, FileIO io) {
+ this(metadataFileLocation, io, null);
+ }
+
+
Review comment:
nit: extra line
##########
File path: core/src/main/java/org/apache/iceberg/StaticTableOperations.java
##########
@@ -31,15 +31,23 @@
public class StaticTableOperations implements TableOperations {
private final TableMetadata staticMetadata;
private final FileIO io;
+ private final LocationProvider locationProvider;
/**
* Creates a StaticTableOperations tied to a specific static version of the TableMetadata
*/
public StaticTableOperations(String metadataFileLocation, FileIO io) {
+ this(metadataFileLocation, io, null);
+ }
+
+
+ public StaticTableOperations(String metadataFileLocation, FileIO io, LocationProvider locationProvider) {
this.io = io;
this.staticMetadata = TableMetadataParser.read(io, metadataFileLocation);
+ this.locationProvider = locationProvider;
}
+
Review comment:
nit: extra line
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] pvary commented on a change in pull request #2340: Hive: Remove unnecessary SerializationUtil.serializeToBase64 from HiveIcebergStorageHandler.overlayTableProperties
Posted by GitBox <gi...@apache.org>.
pvary commented on a change in pull request #2340:
URL: https://github.com/apache/iceberg/pull/2340#discussion_r597509944
##########
File path: spark/src/main/java/org/apache/iceberg/actions/BaseSparkAction.java
##########
@@ -141,9 +142,9 @@ protected JobGroupInfo newJobGroupInfo(String groupId, String desc) {
return otherMetadataFiles;
}
- protected Table newStaticTable(TableMetadata metadata, FileIO io) {
+ protected Table newStaticTable(TableMetadata metadata, FileIO io, LocationProvider locationProvider) {
Review comment:
Since we kept the original constructor we might not need this change now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] lcspinter commented on a change in pull request #2340: Hive: Remove unnecessary SerializationUtil.serializeToBase64 from HiveIcebergStorageHandler.overlayTableProperties
Posted by GitBox <gi...@apache.org>.
lcspinter commented on a change in pull request #2340:
URL: https://github.com/apache/iceberg/pull/2340#discussion_r597618547
##########
File path: core/src/main/java/org/apache/iceberg/StaticTableOperations.java
##########
@@ -29,24 +29,37 @@
* and cannot be used to create or delete files.
*/
public class StaticTableOperations implements TableOperations {
- private final TableMetadata staticMetadata;
+ private TableMetadata staticMetadata;
+ private final String metadataFileLocation;
private final FileIO io;
+ private final LocationProvider locationProvider;
/**
* Creates a StaticTableOperations tied to a specific static version of the TableMetadata
*/
public StaticTableOperations(String metadataFileLocation, FileIO io) {
+ this(metadataFileLocation, io, null);
+ }
+
+ public StaticTableOperations(String metadataFileLocation, FileIO io, LocationProvider locationProvider) {
this.io = io;
- this.staticMetadata = TableMetadataParser.read(io, metadataFileLocation);
+ this.metadataFileLocation = metadataFileLocation;
+ this.locationProvider = locationProvider;
}
@Override
public TableMetadata current() {
+ if (staticMetadata == null) {
+ staticMetadata = TableMetadataParser.read(io, metadataFileLocation);
+ }
return staticMetadata;
}
@Override
public TableMetadata refresh() {
+ if (staticMetadata == null) {
Review comment:
Added a comment to the refresh method.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] lcspinter commented on a change in pull request #2340: Hive: Remove unnecessary SerializationUtil.serializeToBase64 from HiveIcebergStorageHandler.overlayTableProperties
Posted by GitBox <gi...@apache.org>.
lcspinter commented on a change in pull request #2340:
URL: https://github.com/apache/iceberg/pull/2340#discussion_r596905934
##########
File path: mr/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java
##########
@@ -154,30 +151,12 @@ public DecomposedPredicate decomposePredicate(JobConf jobConf, Deserializer dese
}
/**
- * Returns the Table FileIO serialized to the configuration.
+ * Returns the Table serialized to the configuration.
* @param config The configuration used to get the data from
- * @return The Table FileIO object
+ * @return The Table
*/
- public static FileIO io(Configuration config) {
- return SerializationUtil.deserializeFromBase64(config.get(InputFormatConfig.FILE_IO));
- }
-
- /**
- * Returns the Table LocationProvider serialized to the configuration.
- * @param config The configuration used to get the data from
- * @return The Table LocationProvider object
- */
- public static LocationProvider location(Configuration config) {
- return SerializationUtil.deserializeFromBase64(config.get(InputFormatConfig.LOCATION_PROVIDER));
- }
-
- /**
- * Returns the Table EncryptionManager serialized to the configuration.
- * @param config The configuration used to get the data from
- * @return The Table EncryptionManager object
- */
- public static EncryptionManager encryption(Configuration config) {
- return SerializationUtil.deserializeFromBase64(config.get(InputFormatConfig.ENCRYPTION_MANAGER));
+ public static Table table(Configuration config) {
+ return SerializationUtil.deserializeFromBase64(config.get(InputFormatConfig.SERIALIZED_TABLE));
Review comment:
I like the idea. I created a cache object to store the table with the key pair of table identifier and table location.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] pvary commented on a change in pull request #2340: Hive: Remove unnecessary SerializationUtil.serializeToBase64 from HiveIcebergStorageHandler.overlayTableProperties
Posted by GitBox <gi...@apache.org>.
pvary commented on a change in pull request #2340:
URL: https://github.com/apache/iceberg/pull/2340#discussion_r595201123
##########
File path: mr/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java
##########
@@ -154,30 +151,12 @@ public DecomposedPredicate decomposePredicate(JobConf jobConf, Deserializer dese
}
/**
- * Returns the Table FileIO serialized to the configuration.
+ * Returns the Table serialized to the configuration.
* @param config The configuration used to get the data from
- * @return The Table FileIO object
+ * @return The Table
*/
- public static FileIO io(Configuration config) {
- return SerializationUtil.deserializeFromBase64(config.get(InputFormatConfig.FILE_IO));
- }
-
- /**
- * Returns the Table LocationProvider serialized to the configuration.
- * @param config The configuration used to get the data from
- * @return The Table LocationProvider object
- */
- public static LocationProvider location(Configuration config) {
- return SerializationUtil.deserializeFromBase64(config.get(InputFormatConfig.LOCATION_PROVIDER));
- }
-
- /**
- * Returns the Table EncryptionManager serialized to the configuration.
- * @param config The configuration used to get the data from
- * @return The Table EncryptionManager object
- */
- public static EncryptionManager encryption(Configuration config) {
- return SerializationUtil.deserializeFromBase64(config.get(InputFormatConfig.ENCRYPTION_MANAGER));
+ public static Table table(Configuration config) {
+ return SerializationUtil.deserializeFromBase64(config.get(InputFormatConfig.SERIALIZED_TABLE));
Review comment:
Do we want to "cache" the table. So we do not have to deserialize it multiple times?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] pvary commented on a change in pull request #2340: Hive: Remove unnecessary SerializationUtil.serializeToBase64 from HiveIcebergStorageHandler.overlayTableProperties
Posted by GitBox <gi...@apache.org>.
pvary commented on a change in pull request #2340:
URL: https://github.com/apache/iceberg/pull/2340#discussion_r595204199
##########
File path: mr/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java
##########
@@ -239,9 +218,6 @@ static void overlayTableProperties(Configuration configuration, TableDesc tableD
map.put(InputFormatConfig.SERIALIZED_TABLE, SerializationUtil.serializeToBase64(table));
}
- map.put(InputFormatConfig.FILE_IO, SerializationUtil.serializeToBase64(table.io()));
- map.put(InputFormatConfig.LOCATION_PROVIDER, SerializationUtil.serializeToBase64(table.locationProvider()));
- map.put(InputFormatConfig.ENCRYPTION_MANAGER, SerializationUtil.serializeToBase64(table.encryption()));
Review comment:
What about the other stuff put into the map derived from the table, like `schema`, `location`, `fileFormat` etc?
Could we get rid of them too?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org