You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@impala.apache.org by mi...@apache.org on 2026/08/24 22:05:03 UTC
(impala) 02/03: IMPALA-15229: Derive the EC Policy column from file metadata, not RPCs
This is an automated email from the ASF dual-hosted git repository.
asf-gitbox-commits pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git
commit 892f15d128ab082224db5d1349c0d8cea57848b4
Author: Yubi Lee <eu...@gmail.com>
AuthorDate: Wed Jul 29 16:43:29 2026 +0900
IMPALA-15229: Derive the EC Policy column from file metadata, not RPCs
SHOW TABLE STATS, SHOW PARTITIONS and SHOW FILES report an "EC Policy"
column that is populated by FileSystemUtil.getErasureCodingPolicy(),
which issues one NameNode RPC per path, uncached, from a serial loop.
The statements therefore cost O(#partitions) or O(#files) round trips
to populate a display-only column: on a table with ~105k partitions a
single SHOW TABLE STATS produced ~105k serial getErasureCodingPolicy
RPCs, and NameNode audit logs showed 952k such calls in a five minute
window, tripping RPC-spike alerts on clusters that do not use erasure
coding at all (where every call returns NONE).
The information is already available for free: the directory listing
that loads the file metadata returns HdfsFileStatus objects that carry
the erasure coding policy of each file. This change records the policy
id (a single byte) in the file descriptor at load time and derives the
EC Policy column from it, eliminating the per-path RPCs. The recorded
value also describes how the data is actually stored, rather than which
policy the directory would assign to new files.
Details:
- FbFileDesc gains 'ec_policy_id: byte = 0 (id: 8)'. 0 is the id of
the replication policy, i.e. "not erasure-coded or unknown". The
field is additive with an explicit id, so catalog objects remain
compatible in both directions and rolling upgrades keep working:
descriptors written by older versions read as 0, which makes the new
code fall back to the old per-path lookup for erasure-coded files.
'is_ec' is untouched and remains what scheduling relies on.
- FileDescriptor records the id extracted from the FileStatus (a purely
local call, no RPC) and exposes getEcPolicyId().
- Policy ids resolve to names locally through
SystemErasureCodingPolicies for the built-in policies. Names of
user-defined policies are fetched with a single
getAllErasureCodingPolicies() call per filesystem and cached. The
cache is refreshed once when an id missing from it is looked up, so
policies added to the NameNode while Impala is running still resolve
without falling back to per-path lookups.
- SHOW TABLE STATS / SHOW PARTITIONS aggregate the descriptors of the
partition: NONE if no file is erasure-coded, the policy name if all
files share one policy, and the new label MIXED otherwise, including
when an erasure-coded file with an unknown policy id is mixed with
plain files. Partitions with no files fall back to the previous
per-path lookup, as do partitions where every file is erasure-coded
but the policy id of some is unknown (older metadata, or filesystems
like Ozone whose listing does not carry the policy).
- SHOW FILES resolves the policy from each file's descriptor directly.
- DESCRIBE FORMATTED and the Iceberg table-level row keep the existing
lookup, which is one RPC per statement. The per-file lookups in the
Paimon SHOW FILES path iterate Paimon file handles that have no
Impala file descriptor and are left for a follow-up.
Behaviour change: for a partition whose directory has an erasure coding
policy set but whose files are not erasure-coded (or the reverse), the
column now describes the files instead of the directory attribute, and
mixed contents report MIXED.
Testing:
- FileSystemUtilTest covers the policy id extraction from plain and
HDFS file statuses and the local resolution of system policy names.
- FeFsTableTest covers the aggregation of a partition's descriptors:
NONE, a single policy resolved locally, and the MIXED combinations,
including erasure-coded files with unknown policy ids.
- Existing EC end-to-end tests exercise the single-policy path on an
erasure-coded minicluster via $ERASURECODE_POLICY.
Change-Id: I35fc413c4a1f4730baab55406b13f5541f379707
Reviewed-on: http://gerrit.cloudera.org:8080/24642
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
common/fbs/CatalogObjects.fbs | 6 ++
.../java/org/apache/impala/catalog/FeFsTable.java | 69 +++++++++++++++-
.../org/apache/impala/catalog/FeIcebergTable.java | 6 +-
.../org/apache/impala/catalog/FileDescriptor.java | 11 +++
.../org/apache/impala/common/FileSystemUtil.java | 91 ++++++++++++++++++++++
.../org/apache/impala/catalog/FeFsTableTest.java | 79 +++++++++++++++++++
.../catalog/IcebergContentFileStoreTest.java | 4 +-
.../apache/impala/common/FileSystemUtilTest.java | 38 +++++++++
8 files changed, 298 insertions(+), 6 deletions(-)
diff --git a/common/fbs/CatalogObjects.fbs b/common/fbs/CatalogObjects.fbs
index e2a35a091..36ad40f9f 100644
--- a/common/fbs/CatalogObjects.fbs
+++ b/common/fbs/CatalogObjects.fbs
@@ -86,6 +86,12 @@ table FbFileDesc {
// Whether this file is encrypted
is_encrypted: bool = false (id: 7);
+
+ // The HDFS erasure coding policy id of this file, taken from the file status
+ // returned by the directory listing. 0 (the id of the replication policy) means
+ // the file is not erasure-coded or the policy is unknown; 'is_ec' remains the
+ // authoritative flag for whether the file is erasure-coded.
+ ec_policy_id: byte = 0 (id: 8);
}
// Additional file-related metadata
diff --git a/fe/src/main/java/org/apache/impala/catalog/FeFsTable.java b/fe/src/main/java/org/apache/impala/catalog/FeFsTable.java
index f26ea698a..294b2d7fc 100644
--- a/fe/src/main/java/org/apache/impala/catalog/FeFsTable.java
+++ b/fe/src/main/java/org/apache/impala/catalog/FeFsTable.java
@@ -82,6 +82,10 @@ public interface FeFsTable extends FeTable {
// and its usage in getFileSystem suggests it should be.
public static final Configuration CONF = new Configuration();
+ // Label reported in the EC Policy column of a partition whose files do not all
+ // share the same erasure coding policy.
+ public static final String MIXED_ERASURE_CODE_LABEL = "MIXED";
+
// Internal table property that specifies the number of files in the table.
public static final String NUM_FILES = "numFiles";
@@ -708,12 +712,73 @@ public interface FeFsTable extends FeTable {
rowBuilder.add(p.getFileFormat().toString());
rowBuilder.add(String.valueOf(p.hasIncrementalStats()));
rowBuilder.add(p.getLocation());
- rowBuilder.add(FileSystemUtil.getErasureCodingPolicy(p.getLocationPath()));
+ rowBuilder.add(getErasureCodingPolicy(p));
result.addToRows(rowBuilder.get());
return new long[] { cachedBytes, size, numFiles, numRows };
}
+ /**
+ * Returns the erasure coding policy to report for partition 'p', derived from the
+ * policies of its files, which are known from the directory listing that loaded them.
+ * This avoids the per-partition NameNode RPC that looking up the policy of the
+ * partition directory would cost, and reflects how the data is actually stored
+ * rather than which policy new files would get. Returns NONE if no file is
+ * erasure-coded, the policy name if all files share one policy, and MIXED if the
+ * partition mixes plain and erasure-coded files or several policies. Falls back to
+ * the per-directory lookup when the partition has no files or when it cannot be
+ * decided from the descriptors alone: every file is erasure-coded but the policy id
+ * of some of them is unknown (e.g. file metadata written by an older version, or a
+ * filesystem whose listing does not carry the policy, like Ozone).
+ */
+ static String getErasureCodingPolicy(FeFsPartition p) {
+ List<FileDescriptor> fds = p.getFileDescriptors();
+ if (fds.isEmpty()) {
+ return FileSystemUtil.getErasureCodingPolicy(p.getLocationPath());
+ }
+ byte policyId = 0;
+ boolean sawEc = false;
+ boolean sawNonEc = false;
+ boolean sawUnknownEc = false;
+ for (FileDescriptor fd : fds) {
+ if (!fd.getIsEc()) {
+ sawNonEc = true;
+ } else if (fd.getEcPolicyId() == 0) {
+ // Erasure-coded, but the policy id was not recorded: the descriptor was
+ // written by an older version or the listing did not carry the policy.
+ sawUnknownEc = true;
+ } else if (sawEc && fd.getEcPolicyId() != policyId) {
+ return MIXED_ERASURE_CODE_LABEL;
+ } else {
+ sawEc = true;
+ policyId = fd.getEcPolicyId();
+ }
+ // A mix of plain and erasure-coded files is MIXED even if the policy of the
+ // erasure-coded files is unknown; no later file can change that.
+ if (sawNonEc && (sawEc || sawUnknownEc)) return MIXED_ERASURE_CODE_LABEL;
+ }
+ if (sawUnknownEc) {
+ // Every file is erasure-coded but some policy ids are unknown, so whether they
+ // all share one policy cannot be decided from the descriptors.
+ return FileSystemUtil.getErasureCodingPolicy(p.getLocationPath());
+ }
+ if (!sawEc) return FileSystemUtil.NO_ERASURE_CODE_LABEL;
+ return FileSystemUtil.getErasureCodingPolicyName(policyId, p.getLocationPath());
+ }
+
+ /**
+ * Returns the erasure coding policy to report for the file with descriptor 'fd' at
+ * path 'p': NONE if the file is not erasure-coded, otherwise the policy name
+ * resolved from the policy id recorded when the file metadata was loaded. Neither
+ * costs an RPC in the common case. Falls back to the per-path lookup when the file
+ * is erasure-coded but its policy id is unknown (see getErasureCodingPolicy above).
+ */
+ static String getErasureCodingPolicy(FileDescriptor fd, Path p) {
+ if (!fd.getIsEc()) return FileSystemUtil.NO_ERASURE_CODE_LABEL;
+ if (fd.getEcPolicyId() == 0) return FileSystemUtil.getErasureCodingPolicy(p);
+ return FileSystemUtil.getErasureCodingPolicyName(fd.getEcPolicyId(), p);
+ }
+
/**
* Helper method to build the "Total" summary row for partitioned tables.
* @param totalCachedBytes Total cached bytes across all partitions
@@ -916,7 +981,7 @@ public interface FeFsTable extends FeTable {
rowBuilder.add(absPath);
rowBuilder.add(PrintUtils.printBytes(fd.getFileLength()));
rowBuilder.add(p.getPartitionName());
- rowBuilder.add(FileSystemUtil.getErasureCodingPolicy(new Path(absPath)));
+ rowBuilder.add(getErasureCodingPolicy(fd, new Path(absPath)));
result.addToRows(rowBuilder.get());
}
}
diff --git a/fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java b/fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java
index 15db1cd81..7d05217ed 100644
--- a/fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java
+++ b/fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java
@@ -532,7 +532,7 @@ public interface FeIcebergTable extends FeFsTable {
rowBuilder.add(absPath);
rowBuilder.add(PrintUtils.printBytes(fd.getFileLength()));
rowBuilder.add("");
- rowBuilder.add(FileSystemUtil.getErasureCodingPolicy(new Path(absPath)));
+ rowBuilder.add(FeFsTable.getErasureCodingPolicy(fd, new Path(absPath)));
result.addToRows(rowBuilder.get());
}
return result;
@@ -567,7 +567,9 @@ public interface FeIcebergTable extends FeFsTable {
}
rowBuilder.add(""); // Partition - empty for now
- rowBuilder.add(FileSystemUtil.getErasureCodingPolicy(new Path(filePath)));
+ rowBuilder.add(fd != null
+ ? FeFsTable.getErasureCodingPolicy(fd, new Path(filePath))
+ : FileSystemUtil.getErasureCodingPolicy(new Path(filePath)));
result.addToRows(rowBuilder.get());
}
diff --git a/fe/src/main/java/org/apache/impala/catalog/FileDescriptor.java b/fe/src/main/java/org/apache/impala/catalog/FileDescriptor.java
index 46be0b216..92f465059 100644
--- a/fe/src/main/java/org/apache/impala/catalog/FileDescriptor.java
+++ b/fe/src/main/java/org/apache/impala/catalog/FileDescriptor.java
@@ -37,6 +37,7 @@ import org.apache.commons.lang3.StringUtils;
import org.apache.hadoop.fs.BlockLocation;
import org.apache.hadoop.fs.FileStatus;
import org.apache.hadoop.fs.Path;
+import org.apache.impala.common.FileSystemUtil;
import org.apache.impala.fb.FbCompression;
import org.apache.impala.fb.FbFileBlock;
import org.apache.impala.fb.FbFileDesc;
@@ -197,6 +198,7 @@ public class FileDescriptor implements Comparable<FileDescriptor> {
FbFileDesc.addLastModificationTime(fbb, fileStatus.getModificationTime());
FbFileDesc.addIsEncrypted(fbb, isEncrypted);
FbFileDesc.addIsEc(fbb, isEc);
+ FbFileDesc.addEcPolicyId(fbb, FileSystemUtil.getErasureCodingPolicyId(fileStatus));
HdfsCompression comp = HdfsCompression.fromFileName(fileStatus.getPath().getName());
FbFileDesc.addCompression(fbb, comp.toFb());
if (blockVectorOffset >= 0) FbFileDesc.addFileBlocks(fbb, blockVectorOffset);
@@ -280,6 +282,15 @@ public class FileDescriptor implements Comparable<FileDescriptor> {
return fbFileDescriptor_.isEc();
}
+ /**
+ * Returns the erasure coding policy id of this file, or 0 (the id of the replication
+ * policy) if the file is not erasure-coded or the policy is unknown, e.g. when the
+ * descriptor was created by a version that did not populate the field.
+ */
+ public byte getEcPolicyId() {
+ return fbFileDescriptor_.ecPolicyId();
+ }
+
public FbFileBlock getFbFileBlock(int idx) {
return fbFileDescriptor_.fileBlocks(idx);
}
diff --git a/fe/src/main/java/org/apache/impala/common/FileSystemUtil.java b/fe/src/main/java/org/apache/impala/common/FileSystemUtil.java
index 7601935a5..8f995231e 100644
--- a/fe/src/main/java/org/apache/impala/common/FileSystemUtil.java
+++ b/fe/src/main/java/org/apache/impala/common/FileSystemUtil.java
@@ -36,6 +36,10 @@ import org.apache.hadoop.hdfs.DistributedFileSystem;
import org.apache.hadoop.hdfs.client.HdfsAdmin;
import org.apache.hadoop.hdfs.protocol.EncryptionZone;
import org.apache.hadoop.hdfs.protocol.ErasureCodingPolicy;
+import org.apache.hadoop.hdfs.protocol.ErasureCodingPolicyInfo;
+import org.apache.hadoop.hdfs.protocol.HdfsFileStatus;
+import org.apache.hadoop.hdfs.protocol.SystemErasureCodingPolicies;
+import org.apache.hadoop.io.erasurecode.ErasureCodeConstants;
import org.apache.hadoop.ozone.client.ObjectStore;
import org.apache.hadoop.ozone.client.OzoneBucket;
import org.apache.hadoop.ozone.client.OzoneVolume;
@@ -51,9 +55,11 @@ import java.io.FileNotFoundException;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
+import java.io.UncheckedIOException;
import java.net.URI;
import java.util.ArrayList;
import java.util.Collections;
+import java.util.HashMap;
import java.util.HashSet;
import java.util.LinkedHashSet;
import java.util.List;
@@ -63,6 +69,8 @@ import java.util.Set;
import java.util.Stack;
import java.util.StringTokenizer;
import java.util.UUID;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
import static org.apache.hadoop.ozone.OzoneConsts.OZONE_URI_DELIMITER;
@@ -322,6 +330,89 @@ public class FileSystemUtil {
return NO_ERASURE_CODE_LABEL;
}
+ /**
+ * Returns the id of the erasure coding policy of the file represented by
+ * 'fileStatus', or 0 (the id of the replication policy) if the file is not
+ * erasure-coded or the listing does not carry the policy (e.g. non-HDFS
+ * filesystems). Purely local, issues no RPC.
+ */
+ public static byte getErasureCodingPolicyId(FileStatus fileStatus) {
+ if (fileStatus.isErasureCoded() && fileStatus instanceof HdfsFileStatus) {
+ ErasureCodingPolicy policy = ((HdfsFileStatus) fileStatus).getErasureCodingPolicy();
+ if (policy != null) return policy.getId();
+ }
+ return ErasureCodeConstants.REPLICATION_POLICY_ID;
+ }
+
+ /**
+ * Cache of the id to name mapping of the erasure coding policies of a filesystem,
+ * keyed by filesystem URI. Only consulted for policies that are not built-in system
+ * policies, i.e. user-defined ones. Populated with a single
+ * getAllErasureCodingPolicies() RPC per filesystem and refreshed once when an id
+ * that is missing from the snapshot is looked up, so policies added to the
+ * NameNode later are still resolved.
+ */
+ private static final ConcurrentMap<String, Map<Byte, String>> EC_POLICY_NAME_CACHE =
+ new ConcurrentHashMap<>();
+
+ /**
+ * Fetches the id to name mapping of the erasure coding policies of 'fs' (including
+ * disabled and removed ones, whose files remain readable) with a single
+ * getAllErasureCodingPolicies() RPC. Wraps the IOException of the RPC so that the
+ * method can be used in the compute functions of EC_POLICY_NAME_CACHE.
+ */
+ private static Map<Byte, String> fetchErasureCodingPolicyNames(
+ DistributedFileSystem fs) {
+ try {
+ Map<Byte, String> result = new HashMap<>();
+ for (ErasureCodingPolicyInfo info : fs.getAllErasureCodingPolicies()) {
+ result.put(info.getPolicy().getId(), info.getPolicy().getName());
+ }
+ return result;
+ } catch (IOException e) {
+ throw new UncheckedIOException(e);
+ }
+ }
+
+ /**
+ * Returns the name of the erasure coding policy with id 'policyId' for a file on the
+ * same filesystem as 'p'. System policies are resolved locally without an RPC. Names
+ * of user-defined policies are fetched with one getAllErasureCodingPolicies() call
+ * per filesystem and cached. As a last resort falls back to a per-path
+ * getErasureCodingPolicy() lookup.
+ */
+ public static String getErasureCodingPolicyName(byte policyId, Path p) {
+ if (policyId == ErasureCodeConstants.REPLICATION_POLICY_ID) {
+ return NO_ERASURE_CODE_LABEL;
+ }
+ ErasureCodingPolicy systemPolicy = SystemErasureCodingPolicies.getByID(policyId);
+ if (systemPolicy != null) return systemPolicy.getName();
+ try {
+ FileSystem fs = p.getFileSystem(CONF);
+ if (fs instanceof DistributedFileSystem) {
+ DistributedFileSystem dfs = (DistributedFileSystem) fs;
+ Map<Byte, String> names = EC_POLICY_NAME_CACHE.computeIfAbsent(
+ dfs.getUri().toString(), uri -> fetchErasureCodingPolicyNames(dfs));
+ String name = names.get(policyId);
+ if (name == null) {
+ // The id is missing from the cached snapshot: a policy may have been added
+ // to the NameNode after the snapshot was taken. Refresh the snapshot once;
+ // callers that raced here reuse a refreshed map that already resolves the
+ // id instead of refetching again.
+ names = EC_POLICY_NAME_CACHE.compute(dfs.getUri().toString(),
+ (uri, old) -> old != null && old.containsKey(policyId)
+ ? old : fetchErasureCodingPolicyNames(dfs));
+ name = names.get(policyId);
+ }
+ if (name != null) return name;
+ }
+ } catch (IOException | UncheckedIOException e) {
+ LOG.warn("Unable to retrieve erasure coding policies of the filesystem of {}",
+ p, e);
+ }
+ return getErasureCodingPolicy(p);
+ }
+
/**
* Relocates all visible (non-hidden) files from a source directory to a destination
* directory. Files are moved (renamed) to the new location unless the source and
diff --git a/fe/src/test/java/org/apache/impala/catalog/FeFsTableTest.java b/fe/src/test/java/org/apache/impala/catalog/FeFsTableTest.java
new file mode 100644
index 000000000..cea9b8e18
--- /dev/null
+++ b/fe/src/test/java/org/apache/impala/catalog/FeFsTableTest.java
@@ -0,0 +1,79 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.impala.catalog;
+
+import static org.junit.Assert.assertEquals;
+
+import com.google.flatbuffers.FlatBufferBuilder;
+import java.nio.ByteBuffer;
+import java.util.Arrays;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hdfs.protocol.SystemErasureCodingPolicies;
+import org.apache.impala.common.FileSystemUtil;
+import org.apache.impala.fb.FbFileDesc;
+import org.junit.Test;
+import org.mockito.Mockito;
+
+public class FeFsTableTest {
+
+ private static FileDescriptor createFd(boolean isEc, byte ecPolicyId) {
+ FlatBufferBuilder fbb = new FlatBufferBuilder(1);
+ fbb.finish(FbFileDesc.createFbFileDesc(
+ fbb, 0, 10L, (byte) 0, 10000L, 0, isEc, 0, false, ecPolicyId));
+ ByteBuffer bb = fbb.dataBuffer().slice();
+ ByteBuffer copy = ByteBuffer.allocate(bb.capacity());
+ copy.put(bb);
+ return new FileDescriptor(FbFileDesc.getRootAsFbFileDesc((ByteBuffer) copy.flip()));
+ }
+
+ private static FeFsPartition mockPartition(FileDescriptor... fds) {
+ FeFsPartition p = Mockito.mock(FeFsPartition.class);
+ Mockito.when(p.getFileDescriptors()).thenReturn(Arrays.asList(fds));
+ Mockito.when(p.getLocationPath())
+ .thenReturn(new Path("hdfs://localhost:20500/dummy"));
+ return p;
+ }
+
+ /**
+ * Tests the cases of FeFsTable.getErasureCodingPolicy() that are decided from the
+ * file descriptors alone, without falling back to a filesystem lookup.
+ */
+ @Test
+ public void testGetErasureCodingPolicy() {
+ byte rs63 = SystemErasureCodingPolicies.RS_6_3_POLICY_ID;
+ byte xor21 = SystemErasureCodingPolicies.XOR_2_1_POLICY_ID;
+ // No erasure-coded files.
+ assertEquals(FileSystemUtil.NO_ERASURE_CODE_LABEL, FeFsTable.getErasureCodingPolicy(
+ mockPartition(createFd(false, (byte) 0), createFd(false, (byte) 0))));
+ // All files share one policy, resolved locally by its id.
+ assertEquals("RS-6-3-1024k", FeFsTable.getErasureCodingPolicy(
+ mockPartition(createFd(true, rs63), createFd(true, rs63))));
+ // Files with two different policies.
+ assertEquals(FeFsTable.MIXED_ERASURE_CODE_LABEL, FeFsTable.getErasureCodingPolicy(
+ mockPartition(createFd(true, rs63), createFd(true, xor21))));
+ // A mix of erasure-coded and plain files.
+ assertEquals(FeFsTable.MIXED_ERASURE_CODE_LABEL, FeFsTable.getErasureCodingPolicy(
+ mockPartition(createFd(true, rs63), createFd(false, (byte) 0))));
+ // A mix of plain files and an erasure-coded file whose policy id is unknown is
+ // MIXED regardless of the unknown policy, in either file order.
+ assertEquals(FeFsTable.MIXED_ERASURE_CODE_LABEL, FeFsTable.getErasureCodingPolicy(
+ mockPartition(createFd(false, (byte) 0), createFd(true, (byte) 0))));
+ assertEquals(FeFsTable.MIXED_ERASURE_CODE_LABEL, FeFsTable.getErasureCodingPolicy(
+ mockPartition(createFd(true, (byte) 0), createFd(false, (byte) 0))));
+ }
+}
diff --git a/fe/src/test/java/org/apache/impala/catalog/IcebergContentFileStoreTest.java b/fe/src/test/java/org/apache/impala/catalog/IcebergContentFileStoreTest.java
index e0a150aa7..13d1233c1 100644
--- a/fe/src/test/java/org/apache/impala/catalog/IcebergContentFileStoreTest.java
+++ b/fe/src/test/java/org/apache/impala/catalog/IcebergContentFileStoreTest.java
@@ -50,8 +50,8 @@ public class IcebergContentFileStoreTest {
private FbFileDesc createFbFileDesc() {
FlatBufferBuilder fbb = new FlatBufferBuilder(1);
- fbb.finish(
- FbFileDesc.createFbFileDesc(fbb, 0, 10L, (byte) 0, 10000L, 0, false, 0, false));
+ fbb.finish(FbFileDesc.createFbFileDesc(
+ fbb, 0, 10L, (byte) 0, 10000L, 0, false, 0, false, (byte) 0));
ByteBuffer bb = fbb.dataBuffer().slice();
ByteBuffer compressedBb = ByteBuffer.allocate(bb.capacity());
compressedBb.put(bb);
diff --git a/fe/src/test/java/org/apache/impala/common/FileSystemUtilTest.java b/fe/src/test/java/org/apache/impala/common/FileSystemUtilTest.java
index d80ff682a..ffe9d945b 100644
--- a/fe/src/test/java/org/apache/impala/common/FileSystemUtilTest.java
+++ b/fe/src/test/java/org/apache/impala/common/FileSystemUtilTest.java
@@ -29,6 +29,9 @@ import static org.junit.Assert.assertEquals;
import org.apache.hadoop.fs.FileStatus;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hdfs.protocol.ErasureCodingPolicy;
+import org.apache.hadoop.hdfs.protocol.HdfsFileStatus;
+import org.apache.hadoop.hdfs.protocol.SystemErasureCodingPolicies;
import org.apache.impala.thrift.TTableName;
import org.junit.Test;
import org.mockito.Mockito;
@@ -206,6 +209,41 @@ public class FileSystemUtilTest {
}
}
+ @Test
+ public void testGetErasureCodingPolicyId() {
+ // A file that is not erasure-coded maps to the id of the replication policy.
+ FileStatus stat = Mockito.mock(FileStatus.class);
+ Mockito.when(stat.isErasureCoded()).thenReturn(false);
+ assertEquals(0, FileSystemUtil.getErasureCodingPolicyId(stat));
+
+ // A file that is erasure-coded but whose status does not carry the policy
+ // (e.g. on a non-HDFS filesystem) maps to 0 as well, meaning "unknown".
+ Mockito.when(stat.isErasureCoded()).thenReturn(true);
+ assertEquals(0, FileSystemUtil.getErasureCodingPolicyId(stat));
+
+ // The status of an erasure-coded HDFS file carries the policy of the file.
+ ErasureCodingPolicy policy = SystemErasureCodingPolicies.getByID(
+ SystemErasureCodingPolicies.RS_6_3_POLICY_ID);
+ FileStatus ecStat = Mockito.mock(FileStatus.class,
+ Mockito.withSettings().extraInterfaces(HdfsFileStatus.class));
+ Mockito.when(ecStat.isErasureCoded()).thenReturn(true);
+ Mockito.when(((HdfsFileStatus) ecStat).getErasureCodingPolicy()).thenReturn(policy);
+ assertEquals(policy.getId(), FileSystemUtil.getErasureCodingPolicyId(ecStat));
+ }
+
+ @Test
+ public void testGetErasureCodingPolicyName() {
+ Path dummyPath = new Path("hdfs://localhost:20500/dummy");
+ // The id of the replication policy means "not erasure-coded".
+ assertEquals(FileSystemUtil.NO_ERASURE_CODE_LABEL,
+ FileSystemUtil.getErasureCodingPolicyName((byte) 0, dummyPath));
+ // System policies are resolved locally, without an RPC.
+ assertEquals("RS-6-3-1024k", FileSystemUtil.getErasureCodingPolicyName(
+ SystemErasureCodingPolicies.RS_6_3_POLICY_ID, dummyPath));
+ assertEquals("XOR-2-1-1024k", FileSystemUtil.getErasureCodingPolicyName(
+ SystemErasureCodingPolicies.XOR_2_1_POLICY_ID, dummyPath));
+ }
+
@Test
public void testWriteableByImpala() throws IOException {
testIsWritableByImpala(mockLocation(FileSystemUtil.SCHEME_ALLUXIO), false);