You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "Gabor Kaszab (JIRA)" <ji...@apache.org> on 2018/05/24 14:43:00 UTC
[jira] [Comment Edited] (IMPALA-6119) Inconsistent file metadata
updates when multiple partitions point to the same path
[ https://issues.apache.org/jira/browse/IMPALA-6119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16489122#comment-16489122 ]
Gabor Kaszab edited comment on IMPALA-6119 at 5/24/18 2:42 PM:
---------------------------------------------------------------
I identified the issue to be located somewhere else. What I found is that in case an "insert into test partition(b=1) values (2);" is invoked then only the b=1 partition is reloaded. However the b=2 partition should also be reloaded to be aware of the new file created by the insert.
In updatePartitionsFromHms()
{code:java}
if (loadPartitionFileMetadata) {
if (partitionsToUpdate != null) {
// Only reload file metadata of partitions specified in 'partitionsToUpdate'
Preconditions.checkState(partitionsToUpdateFileMdByPath.isEmpty());
partitionsToUpdateFileMdByPath = getPartitionsByPath(partitionsToUpdate);
}
loadMetadataAndDiskIds(partitionsToUpdateFileMdByPath, true);
}
{code}
getPartitionsByPath() in this case receives the b=1 partition an returns it's path and the partition itself. As a result loadMetadataAndDiskIds() is called only for b=1 partition.
I made some experiments to modify getPartitionsByPath() to find all the partitions that point to the location where the ones received by parameter points to and apparently that fixes this issue. However, it's might not the most optimal solution.
+My general fix proposal is then the following:+
- When a particular partition is being reloaded then find all the other partitions that has the same 'location' and reload them as well. Finding these partitions in an optimal way is not that straightforward, though.
+2 proposals for finding the partitions with the same location:+
1) When a set of partitions are received in updatePartitionsFromHms() then go through this set and compare the current partition from the set with all the partitions in the table to check which one has the same 'location'. When we have big inserts that affect a number of partitions and in the same time the total number of partitions is high than this operation can be computational heavy and has to be done for each insert even if we don't have partitions pointing to the same location.
2) Similarly to HdfsTable:: partitionMap_ we can keep track of a (path -> set of partitions) mapping. This way when a set of partitions are received in updatePartitionsFromHms() then it's enough to go through this received list and then we can find the partitions pointing to the same location using this mapping in constant time. One downside of this is that everywhere partitionMap_ is changed this mapping has to be maintained as well. Still this seems the better approach for me.
was (Author: gaborkaszab):
I identified the issue to be located somewhere else. What I found is that in case an "insert into test partition(b=1) values (2);" is invoked then only the b=1 partition is reloaded. However the b=2 partition should also be reloaded to be aware of the new file created by the insert.
In updatePartitionsFromHms()
{code:java}
if (loadPartitionFileMetadata) {
if (partitionsToUpdate != null) {
// Only reload file metadata of partitions specified in 'partitionsToUpdate'
Preconditions.checkState(partitionsToUpdateFileMdByPath.isEmpty());
partitionsToUpdateFileMdByPath = getPartitionsByPath(partitionsToUpdate);
}
loadMetadataAndDiskIds(partitionsToUpdateFileMdByPath, true);
}
{code}
getPartitionsByPath() in this case receives the b=1 partition an returns it's path and the partition itself. As a result loadMetadataAndDiskIds() is called only for b=1 partition.
I made some experiments to modify getPartitionsByPath() to find all the partitions that point to the location where the ones received by parameter points to and apparently that fixes this issue. However, it's might not the most optimal solution.
+My general fix proposal is then the following:+
- When a particular partition is being reloaded then find all the other partitions that has the same 'location' and reload them as well. Finding these partitions in an optimal way is not that straightforward, though.
+2 proposals for finding the partitions with the same location:+
1) When a set of partitions are received in updatePartitionsFromHms() then go through this set and compare the current partition from the set with all the partitions in the table to check which one has the same 'location'. When we have big inserts that affect a number of partitions and in the same time the total number of partitions is high than this operation can be computational heavy and has to be done for each insert even if we don't have partitions pointing to the same location.
2) Similarly to HdfsTable:: partitionMap_ we can keep track of a (path -> set of partitions) mapping. This way when a set of partitions are received in updatePartitionsFromHms() then it's enough to go through this received list and then we can find the partitions pointing to the same location using this mapping in constant time. One downside of this is that everywhere partitionMap_ is changed this mapping has to be maintained as well. Still this seems the better approach for me.
> Inconsistent file metadata updates when multiple partitions point to the same path
> ----------------------------------------------------------------------------------
>
> Key: IMPALA-6119
> URL: https://issues.apache.org/jira/browse/IMPALA-6119
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog
> Affects Versions: Impala 2.8.0, Impala 2.9.0, Impala 2.10.0
> Reporter: bharath v
> Assignee: Gabor Kaszab
> Priority: Critical
> Labels: correctness, ramp-up
>
> Following steps can give inconsistent results.
> {noformat}
> // Create a partitioned table
> create table test(a int) partitioned by (b int);
> // Create two partitions b=1/b=2 mapped to the same HDFS location.
> insert into test partition(b=1) values (1);
> alter table test add partition (b=2) location 'hdfs://localhost:20500/test-warehouse/test/b=1/'
> [localhost:21000] > show partitions test;
> Query: show partitions test
> +-------+-------+--------+------+--------------+-------------------+--------+-------------------+------------------------------------------------+
> | b | #Rows | #Files | Size | Bytes Cached | Cache Replication | Format | Incremental stats | Location |
> +-------+-------+--------+------+--------------+-------------------+--------+-------------------+------------------------------------------------+
> | 1 | -1 | 1 | 2B | NOT CACHED | NOT CACHED | TEXT | false | hdfs://localhost:20500/test-warehouse/test/b=1 |
> | 2 | -1 | 1 | 2B | NOT CACHED | NOT CACHED | TEXT | false | hdfs://localhost:20500/test-warehouse/test/b=1 |
> | Total | -1 | 2 | 4B | 0B | | | | |
> +-------+-------+--------+------+--------------+-------------------+--------+-------------------+------------------------------------------------+
> // Insert new data into one of the partitions
> insert into test partition(b=1) values (2);
> // Newly added file is reflected only in the added partition files.
> show files in test;
> Query: show files in test
> +----------------------------------------------------------------------------------------------------+------+-----------+
> | Path | Size | Partition |
> +----------------------------------------------------------------------------------------------------+------+-----------+
> | hdfs://localhost:20500/test-warehouse/test/b=1/2e44cd49e8c3d30d-572fc97800000000_627280230_data.0. | 2B | b=1 |
> | hdfs://localhost:20500/test-warehouse/test/b=1/e44245ad5c0ef020-a08716d00000000_1244237483_data.0. | 2B | b=1 |
> | hdfs://localhost:20500/test-warehouse/test/b=1/e44245ad5c0ef020-a08716d00000000_1244237483_data.0. | 2B | b=2 |
> +----------------------------------------------------------------------------------------------------+------+-----------+
> invalidate metadata test;
> show files in test;
> // After invalidation, the newly added file now shows up in both the partitions.
> Query: show files in test
> +----------------------------------------------------------------------------------------------------+------+-----------+
> | Path | Size | Partition |
> +----------------------------------------------------------------------------------------------------+------+-----------+
> | hdfs://localhost:20500/test-warehouse/test/b=1/2e44cd49e8c3d30d-572fc97800000000_627280230_data.0. | 2B | b=1 |
> | hdfs://localhost:20500/test-warehouse/test/b=1/e44245ad5c0ef020-a08716d00000000_1244237483_data.0. | 2B | b=1 |
> | hdfs://localhost:20500/test-warehouse/test/b=1/2e44cd49e8c3d30d-572fc97800000000_627280230_data.0. | 2B | b=2 |
> | hdfs://localhost:20500/test-warehouse/test/b=1/e44245ad5c0ef020-a08716d00000000_1244237483_data.0. | 2B | b=2 |
> +----------------------------------------------------------------------------------------------------+------+-----------+
> {noformat}
> So, depending whether the user invalidates the table, they can see different results. The bug is in the following code.
> {noformat}
> private FileMetadataLoadStats resetAndLoadFileMetadata(
> Path partDir, List<HdfsPartition> partitions) throws IOException {
> FileMetadataLoadStats loadStats = new FileMetadataLoadStats(partDir);
> ....
> ....
> ....
> for (HdfsPartition partition: partitions) partition.setFileDescriptors(newFileDescs); <======
> {noformat}
> We only update the added file metadata for the new partition (copy-on-write way). Instead we should update the source descriptors so that it is reflected in the other partitions too.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org