You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "neuyilan (via GitHub)" <gi...@apache.org> on 2024/12/24 12:26:15 UTC

[PR] [Core] support write to the external path [paimon]

neuyilan opened a new pull request, #4770:
URL: https://github.com/apache/paimon/pull/4770

   <!-- Please specify the module before the PR name: [core] ... or [flink] ... -->
   
   ### Purpose
   https://cwiki.apache.org/confluence/display/PAIMON/PIP-29%3A+Introduce+Table+Multi-Location++Management
   
   
   ### Tests
   
   <!-- List UT and IT cases to verify this change -->
   
   ### API and Format
   
   <!-- Does this change affect API or storage format -->
   
   ### Documentation
   
   <!-- Does this change introduce a new feature -->
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Core] support write to the external path [paimon]

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #4770:
URL: https://github.com/apache/paimon/pull/4770#discussion_r1901442965


##########
paimon-core/src/main/java/org/apache/paimon/io/DataFilePathFactory.java:
##########
@@ -74,7 +81,13 @@ public String newChangelogFileName() {
     }
 
     public Path newPath(String prefix) {
-        return new Path(parent, newFileName(prefix));
+        return externalPathProvider
+                .getNextExternalPath()
+                .map(
+                        externalPath ->
+                                new Path(new Path(externalPath, relativePath), newFileName(prefix)))
+                .orElse(new Path(parent, newFileName(prefix)))
+                .setExternalPath(externalPathProvider.externalPathExists());

Review Comment:
   Maybe `DataFilePathFactory` can provide a method `isExternalPath`?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Core] support write to the external path [paimon]

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #4770:
URL: https://github.com/apache/paimon/pull/4770#discussion_r1901442835


##########
paimon-common/src/main/java/org/apache/paimon/fs/Path.java:
##########
@@ -62,6 +62,8 @@ public class Path implements Comparable<Path>, Serializable {
     /** A hierarchical URI. */
     private URI uri;
 
+    private boolean isExternalPath;

Review Comment:
   We should avoid introducing field in Public API `Path`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Core] support write to the external path [paimon]

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #4770:
URL: https://github.com/apache/paimon/pull/4770#discussion_r1903093823


##########
paimon-common/src/main/java/org/apache/paimon/fs/ExternalPathProvider.java:
##########
@@ -0,0 +1,213 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.paimon.fs;
+
+import org.apache.paimon.CoreOptions.ExternalPathStrategy;
+import org.apache.paimon.annotation.VisibleForTesting;
+
+import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Optional;
+
+/** Provider for external paths. */
+public class ExternalPathProvider implements Serializable {
+    private static final String S3 = "s3";
+    private static final String OSS = "oss";
+
+    private final Map<String, Path> externalPathsMap;
+    private final List<Path> externalPathsList;
+
+    private final ExternalPathStrategy externalPathStrategy;
+    private final String externalFSStrategy;
+    private int currentIndex;
+    private boolean externalPathExists;

Review Comment:
   If no external Paths, ExternalPathProvider should be null.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Core] support write to the external path [paimon]

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #4770:
URL: https://github.com/apache/paimon/pull/4770#discussion_r1903093682


##########
paimon-core/src/main/java/org/apache/paimon/table/AbstractFileStoreTable.java:
##########
@@ -105,11 +106,14 @@ abstract class AbstractFileStoreTable implements FileStoreTable {
     @Nullable protected transient Cache<Path, Snapshot> snapshotCache;
     @Nullable protected transient Cache<String, Statistics> statsCache;
 
+    protected final ExternalPathProvider externalPathProvider;

Review Comment:
   `ExternalPathProvider` is just created from options, it should be created in `AbstractFileStore.pathFactory()`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Core] support write to the external path [paimon]

Posted by "wwj6591812 (via GitHub)" <gi...@apache.org>.
wwj6591812 commented on code in PR #4770:
URL: https://github.com/apache/paimon/pull/4770#discussion_r1898217962


##########
paimon-core/src/main/java/org/apache/paimon/utils/FileStorePathFactory.java:
##########
@@ -38,7 +38,10 @@ public class FileStorePathFactory {
 
     public static final String BUCKET_PATH_PREFIX = "bucket-";
 
+    // this is the table schema root path
     private final Path root;
+    // this is the table data root path
+    private final Path dataRoot;

Review Comment:
   ok



##########
paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/clone/CopyFileOperator.java:
##########
@@ -139,6 +141,37 @@ private Path pathOfTable(Table table) {
         return new Path(table.options().get(CoreOptions.PATH.key()));
     }
 
+    private void copyManifestFile(Path sourcePath, Path targetPath, CloneFileInfo cloneFileInfo)
+            throws IOException, TableNotExistException {
+        Identifier sourceIdentifier = Identifier.fromString(cloneFileInfo.getSourceIdentifier());
+        FileStoreTable sourceTable = (FileStoreTable) sourceCatalog.getTable(sourceIdentifier);
+        FileStore<?> store = sourceTable.store();
+        ManifestFile manifestFile = store.manifestFileFactory().create();
+
+        List<ManifestEntry> manifestEntries =
+                manifestFile.readWithIOException(sourcePath.getName());
+        List<ManifestEntry> targetManifestEntries = new ArrayList<>(manifestEntries.size());
+
+        // clone job will clone the source path to target warehouse path, so the target external
+        // path is null
+        String newExternalPath = null;

Review Comment:
   I means that you can write "manifestEntry.file().copy((String) null))" here.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Core] support write to the external path [paimon]

Posted by "neuyilan (via GitHub)" <gi...@apache.org>.
neuyilan commented on PR #4770:
URL: https://github.com/apache/paimon/pull/4770#issuecomment-2572425499

   > And also, please add ITCase for this feature, Flink ITCase or Spark ITCase. (no need to add both, just pick one engine)
   
   Hi, Jingsong, thanks for the review suggestions.  I have already made all the modifications, PTLA again.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Core] support write to the external path [paimon]

Posted by "wwj6591812 (via GitHub)" <gi...@apache.org>.
wwj6591812 commented on code in PR #4770:
URL: https://github.com/apache/paimon/pull/4770#discussion_r1897940949


##########
paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/clone/CopyFileOperator.java:
##########
@@ -139,6 +141,37 @@ private Path pathOfTable(Table table) {
         return new Path(table.options().get(CoreOptions.PATH.key()));
     }
 
+    private void copyManifestFile(Path sourcePath, Path targetPath, CloneFileInfo cloneFileInfo)
+            throws IOException, TableNotExistException {
+        Identifier sourceIdentifier = Identifier.fromString(cloneFileInfo.getSourceIdentifier());
+        FileStoreTable sourceTable = (FileStoreTable) sourceCatalog.getTable(sourceIdentifier);
+        FileStore<?> store = sourceTable.store();
+        ManifestFile manifestFile = store.manifestFileFactory().create();
+
+        List<ManifestEntry> manifestEntries =
+                manifestFile.readWithIOException(sourcePath.getName());
+        List<ManifestEntry> targetManifestEntries = new ArrayList<>(manifestEntries.size());
+
+        // clone job will clone the source path to target warehouse path, so the target external
+        // path is null
+        String newExternalPath = null;

Review Comment:
   Variable 'newExternalPath' is never used



##########
paimon-core/src/main/java/org/apache/paimon/utils/FileStorePathFactory.java:
##########
@@ -38,7 +38,10 @@ public class FileStorePathFactory {
 
     public static final String BUCKET_PATH_PREFIX = "bucket-";
 
+    // this is the table schema root path
     private final Path root;
+    // this is the table data root path
+    private final Path dataRoot;

Review Comment:
   May be it is better named externalRoot?



##########
paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/clone/CopyFileOperator.java:
##########
@@ -139,6 +141,37 @@ private Path pathOfTable(Table table) {
         return new Path(table.options().get(CoreOptions.PATH.key()));
     }
 
+    private void copyManifestFile(Path sourcePath, Path targetPath, CloneFileInfo cloneFileInfo)

Review Comment:
   Why copy ManifestFile by this new method?



##########
paimon-common/src/main/java/org/apache/paimon/CoreOptions.java:
##########
@@ -2364,6 +2390,10 @@ public boolean statsDenseStore() {
         return options.get(METADATA_STATS_DENSE_STORE);
     }
 
+    public String dataFileExternalPath() {

Review Comment:
   Where this method be called?



##########
paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/clone/CopyFileOperator.java:
##########
@@ -139,6 +141,37 @@ private Path pathOfTable(Table table) {
         return new Path(table.options().get(CoreOptions.PATH.key()));
     }
 
+    private void copyManifestFile(Path sourcePath, Path targetPath, CloneFileInfo cloneFileInfo)
+            throws IOException, TableNotExistException {
+        Identifier sourceIdentifier = Identifier.fromString(cloneFileInfo.getSourceIdentifier());
+        FileStoreTable sourceTable = (FileStoreTable) sourceCatalog.getTable(sourceIdentifier);
+        FileStore<?> store = sourceTable.store();
+        ManifestFile manifestFile = store.manifestFileFactory().create();
+
+        List<ManifestEntry> manifestEntries =
+                manifestFile.readWithIOException(sourcePath.getName());
+        List<ManifestEntry> targetManifestEntries = new ArrayList<>(manifestEntries.size());
+
+        // clone job will clone the source path to target warehouse path, so the target external
+        // path is null
+        String newExternalPath = null;
+        for (ManifestEntry manifestEntry : manifestEntries) {
+            ManifestEntry newManifestEntry =
+                    new ManifestEntry(
+                            manifestEntry.kind(),
+                            manifestEntry.partition(),
+                            manifestEntry.bucket(),
+                            manifestEntry.totalBuckets(),
+                            manifestEntry.file().copy(newExternalPath));

Review Comment:
   Value 'newExternalPath' is always 'null' 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Core] support write to the external path [paimon]

Posted by "neuyilan (via GitHub)" <gi...@apache.org>.
neuyilan commented on code in PR #4770:
URL: https://github.com/apache/paimon/pull/4770#discussion_r1901627785


##########
paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/clone/CloneFileInfo.java:
##########
@@ -53,10 +56,18 @@ public String getTargetIdentifier() {
         return targetIdentifier;
     }
 
+    public FileType getFileType() {

Review Comment:
   OK, In the later PR, I will fix the clone action. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Core] support write to the external path [paimon]

Posted by "neuyilan (via GitHub)" <gi...@apache.org>.
neuyilan commented on code in PR #4770:
URL: https://github.com/apache/paimon/pull/4770#discussion_r1898192584


##########
paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/clone/CopyFileOperator.java:
##########
@@ -139,6 +141,37 @@ private Path pathOfTable(Table table) {
         return new Path(table.options().get(CoreOptions.PATH.key()));
     }
 
+    private void copyManifestFile(Path sourcePath, Path targetPath, CloneFileInfo cloneFileInfo)
+            throws IOException, TableNotExistException {
+        Identifier sourceIdentifier = Identifier.fromString(cloneFileInfo.getSourceIdentifier());
+        FileStoreTable sourceTable = (FileStoreTable) sourceCatalog.getTable(sourceIdentifier);
+        FileStore<?> store = sourceTable.store();
+        ManifestFile manifestFile = store.manifestFileFactory().create();
+
+        List<ManifestEntry> manifestEntries =
+                manifestFile.readWithIOException(sourcePath.getName());
+        List<ManifestEntry> targetManifestEntries = new ArrayList<>(manifestEntries.size());
+
+        // clone job will clone the source path to target warehouse path, so the target external
+        // path is null
+        String newExternalPath = null;

Review Comment:
   It's used in line 165, we first declare a variable of `String` type, and then call the `public DataFileMeta copy(String newExternalPath) `function. The reason why we declare variables of type `String` is because there are many other `copy` functions in `DataMileMeta`, if we directly call DataFileMeta.copy(null), the compiler doesn't know which function is being called.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Core] support write to the external path [paimon]

Posted by "neuyilan (via GitHub)" <gi...@apache.org>.
neuyilan commented on code in PR #4770:
URL: https://github.com/apache/paimon/pull/4770#discussion_r1898192584


##########
paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/clone/CopyFileOperator.java:
##########
@@ -139,6 +141,37 @@ private Path pathOfTable(Table table) {
         return new Path(table.options().get(CoreOptions.PATH.key()));
     }
 
+    private void copyManifestFile(Path sourcePath, Path targetPath, CloneFileInfo cloneFileInfo)
+            throws IOException, TableNotExistException {
+        Identifier sourceIdentifier = Identifier.fromString(cloneFileInfo.getSourceIdentifier());
+        FileStoreTable sourceTable = (FileStoreTable) sourceCatalog.getTable(sourceIdentifier);
+        FileStore<?> store = sourceTable.store();
+        ManifestFile manifestFile = store.manifestFileFactory().create();
+
+        List<ManifestEntry> manifestEntries =
+                manifestFile.readWithIOException(sourcePath.getName());
+        List<ManifestEntry> targetManifestEntries = new ArrayList<>(manifestEntries.size());
+
+        // clone job will clone the source path to target warehouse path, so the target external
+        // path is null
+        String newExternalPath = null;

Review Comment:
   It's used in line 165, we first declare a variable of `String` type, and then call the `public DataFileMeta copy(String newExternalPath) `function. The reason why there are variables of type `String` is because there are many other `copy` functions in `DataMileMeta`.



##########
paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/clone/CopyFileOperator.java:
##########
@@ -139,6 +141,37 @@ private Path pathOfTable(Table table) {
         return new Path(table.options().get(CoreOptions.PATH.key()));
     }
 
+    private void copyManifestFile(Path sourcePath, Path targetPath, CloneFileInfo cloneFileInfo)
+            throws IOException, TableNotExistException {
+        Identifier sourceIdentifier = Identifier.fromString(cloneFileInfo.getSourceIdentifier());
+        FileStoreTable sourceTable = (FileStoreTable) sourceCatalog.getTable(sourceIdentifier);
+        FileStore<?> store = sourceTable.store();
+        ManifestFile manifestFile = store.manifestFileFactory().create();
+
+        List<ManifestEntry> manifestEntries =
+                manifestFile.readWithIOException(sourcePath.getName());
+        List<ManifestEntry> targetManifestEntries = new ArrayList<>(manifestEntries.size());
+
+        // clone job will clone the source path to target warehouse path, so the target external
+        // path is null
+        String newExternalPath = null;
+        for (ManifestEntry manifestEntry : manifestEntries) {
+            ManifestEntry newManifestEntry =
+                    new ManifestEntry(
+                            manifestEntry.kind(),
+                            manifestEntry.partition(),
+                            manifestEntry.bucket(),
+                            manifestEntry.totalBuckets(),
+                            manifestEntry.file().copy(newExternalPath));

Review Comment:
   Yes, the same as above



##########
paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/clone/CopyFileOperator.java:
##########
@@ -139,6 +141,37 @@ private Path pathOfTable(Table table) {
         return new Path(table.options().get(CoreOptions.PATH.key()));
     }
 
+    private void copyManifestFile(Path sourcePath, Path targetPath, CloneFileInfo cloneFileInfo)

Review Comment:
   Because source table's manifest's file entry have `DataFileMeta` field,  in this commit[1], we have add `externalPath` in `DataFileMeta`, so when we clone source table's manifest, we should rewrite the manifest file, rewrite the `externalPath` in `DataMileMeta` to null, (being null means that the datafile exists in the warehouse path).
   [1] https://github.com/apache/paimon/commit/9ff5192a09856aff318597e589f8e8c8f6ff325b



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Core] support write to the external path [paimon]

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #4770:
URL: https://github.com/apache/paimon/pull/4770#discussion_r1903093956


##########
paimon-common/src/main/java/org/apache/paimon/CoreOptions.java:
##########
@@ -2181,6 +2208,18 @@ public PartitionExpireStrategy partitionExpireStrategy() {
         return options.get(PARTITION_EXPIRATION_STRATEGY);
     }
 
+    public String dataFileExternalPaths() {
+        return options.get(DATA_FILE_EXTERNAL_PATHS);
+    }
+
+    public ExternalPathStrategy externalPathStrategy() {
+        return options.get(DATA_FILE_EXTERNAL_PATHS_STRATEGY);
+    }
+
+    public String externalSpecificFSStrategy() {

Review Comment:
   externalPathSpecificFS



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Core] support write to the external path [paimon]

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #4770:
URL: https://github.com/apache/paimon/pull/4770#discussion_r1901626156


##########
paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/clone/CloneFileInfo.java:
##########
@@ -53,10 +56,18 @@ public String getTargetIdentifier() {
         return targetIdentifier;
     }
 
+    public FileType getFileType() {

Review Comment:
   Can you do this in a separate PR?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Core] support write to the external path [paimon]

Posted by "neuyilan (via GitHub)" <gi...@apache.org>.
neuyilan commented on code in PR #4770:
URL: https://github.com/apache/paimon/pull/4770#discussion_r1898291710


##########
paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/clone/CopyFileOperator.java:
##########
@@ -139,6 +141,37 @@ private Path pathOfTable(Table table) {
         return new Path(table.options().get(CoreOptions.PATH.key()));
     }
 
+    private void copyManifestFile(Path sourcePath, Path targetPath, CloneFileInfo cloneFileInfo)

Review Comment:
   However, since we introduced the `externalPath`, which points to the data storage location of the source table, we need to modify this property to the `warehouse path` of the target table when cloning.
   Do you have any good suggestions to solve this problem?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Core] support write to the external path [paimon]

Posted by "wwj6591812 (via GitHub)" <gi...@apache.org>.
wwj6591812 commented on code in PR #4770:
URL: https://github.com/apache/paimon/pull/4770#discussion_r1898218409


##########
paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/clone/CopyFileOperator.java:
##########
@@ -139,6 +141,37 @@ private Path pathOfTable(Table table) {
         return new Path(table.options().get(CoreOptions.PATH.key()));
     }
 
+    private void copyManifestFile(Path sourcePath, Path targetPath, CloneFileInfo cloneFileInfo)
+            throws IOException, TableNotExistException {
+        Identifier sourceIdentifier = Identifier.fromString(cloneFileInfo.getSourceIdentifier());
+        FileStoreTable sourceTable = (FileStoreTable) sourceCatalog.getTable(sourceIdentifier);
+        FileStore<?> store = sourceTable.store();
+        ManifestFile manifestFile = store.manifestFileFactory().create();
+
+        List<ManifestEntry> manifestEntries =
+                manifestFile.readWithIOException(sourcePath.getName());
+        List<ManifestEntry> targetManifestEntries = new ArrayList<>(manifestEntries.size());
+
+        // clone job will clone the source path to target warehouse path, so the target external
+        // path is null
+        String newExternalPath = null;
+        for (ManifestEntry manifestEntry : manifestEntries) {
+            ManifestEntry newManifestEntry =
+                    new ManifestEntry(
+                            manifestEntry.kind(),
+                            manifestEntry.partition(),
+                            manifestEntry.bucket(),
+                            manifestEntry.totalBuckets(),
+                            manifestEntry.file().copy(newExternalPath));

Review Comment:
   ditto



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Core] support write to the external path [paimon]

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on PR #4770:
URL: https://github.com/apache/paimon/pull/4770#issuecomment-2565093440

   Hi Houliang,
   
   How about adding an option to table: external-paths?
   
   And we can introduce an option to table: external-paths.strategy, it
   is an enum, can be ROUND_ROBIN (use all paths, pick one for a file),
   NONE (not use external paths).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Core] support write to the external path [paimon]

Posted by "neuyilan (via GitHub)" <gi...@apache.org>.
neuyilan commented on code in PR #4770:
URL: https://github.com/apache/paimon/pull/4770#discussion_r1898194907


##########
paimon-common/src/main/java/org/apache/paimon/CoreOptions.java:
##########
@@ -2364,6 +2390,10 @@ public boolean statsDenseStore() {
         return options.get(METADATA_STATS_DENSE_STORE);
     }
 
+    public String dataFileExternalPath() {

Review Comment:
   removed



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Core] support write to the external path [paimon]

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #4770:
URL: https://github.com/apache/paimon/pull/4770


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Core] support write to the external path [paimon]

Posted by "neuyilan (via GitHub)" <gi...@apache.org>.
neuyilan commented on PR #4770:
URL: https://github.com/apache/paimon/pull/4770#issuecomment-2567339015

   > Hi Houliang,
   > 
   > How about adding an option to table: external-paths?
   > 
   > And we can introduce an option to table: external-paths.strategy, it is an enum, can be ROUND_ROBIN (use all paths, pick one for a file), NONE (not use external paths).
   
   Good point!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Core] support write to the external path [paimon]

Posted by "neuyilan (via GitHub)" <gi...@apache.org>.
neuyilan commented on code in PR #4770:
URL: https://github.com/apache/paimon/pull/4770#discussion_r1901511040


##########
paimon-common/src/main/java/org/apache/paimon/fs/Path.java:
##########
@@ -62,6 +62,8 @@ public class Path implements Comparable<Path>, Serializable {
     /** A hierarchical URI. */
     private URI uri;
 
+    private boolean isExternalPath;

Review Comment:
   done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Core] support write to the external path [paimon]

Posted by "neuyilan (via GitHub)" <gi...@apache.org>.
neuyilan commented on code in PR #4770:
URL: https://github.com/apache/paimon/pull/4770#discussion_r1898219442


##########
paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/clone/CopyFileOperator.java:
##########
@@ -139,6 +141,37 @@ private Path pathOfTable(Table table) {
         return new Path(table.options().get(CoreOptions.PATH.key()));
     }
 
+    private void copyManifestFile(Path sourcePath, Path targetPath, CloneFileInfo cloneFileInfo)
+            throws IOException, TableNotExistException {
+        Identifier sourceIdentifier = Identifier.fromString(cloneFileInfo.getSourceIdentifier());
+        FileStoreTable sourceTable = (FileStoreTable) sourceCatalog.getTable(sourceIdentifier);
+        FileStore<?> store = sourceTable.store();
+        ManifestFile manifestFile = store.manifestFileFactory().create();
+
+        List<ManifestEntry> manifestEntries =
+                manifestFile.readWithIOException(sourcePath.getName());
+        List<ManifestEntry> targetManifestEntries = new ArrayList<>(manifestEntries.size());
+
+        // clone job will clone the source path to target warehouse path, so the target external
+        // path is null
+        String newExternalPath = null;

Review Comment:
   got it



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Core] support write to the external path [paimon]

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #4770:
URL: https://github.com/apache/paimon/pull/4770#discussion_r1903094382


##########
paimon-common/src/main/java/org/apache/paimon/fs/ExternalPathProvider.java:
##########
@@ -0,0 +1,213 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.paimon.fs;
+
+import org.apache.paimon.CoreOptions.ExternalPathStrategy;
+import org.apache.paimon.annotation.VisibleForTesting;
+
+import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Optional;
+
+/** Provider for external paths. */
+public class ExternalPathProvider implements Serializable {
+    private static final String S3 = "s3";
+    private static final String OSS = "oss";
+
+    private final Map<String, Path> externalPathsMap;
+    private final List<Path> externalPathsList;
+
+    private final ExternalPathStrategy externalPathStrategy;
+    private final String externalFSStrategy;
+    private int currentIndex;
+    private boolean externalPathExists;
+    private final String dbAndTableRelativePath;
+
+    @VisibleForTesting
+    public ExternalPathProvider() {
+        this.externalPathsMap = new HashMap<>();
+        this.externalPathsList = new ArrayList<>();
+        this.externalPathStrategy = ExternalPathStrategy.NONE;
+        this.externalFSStrategy = null;
+        this.dbAndTableRelativePath = null;
+    }
+
+    public ExternalPathProvider(
+            String externalPaths,
+            ExternalPathStrategy externalPathStrategy,
+            String externalFSStrategy,
+            String dbAndTableRelativePath) {
+        this.externalPathsMap = new HashMap<>();
+        this.externalPathsList = new ArrayList<>();
+        this.externalPathStrategy = externalPathStrategy;
+        this.externalFSStrategy = externalFSStrategy;
+        this.dbAndTableRelativePath = dbAndTableRelativePath;
+        this.currentIndex = 0;

Review Comment:
   first index should be random index.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Core] support write to the external path [paimon]

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #4770:
URL: https://github.com/apache/paimon/pull/4770#discussion_r1903094157


##########
paimon-core/src/main/java/org/apache/paimon/io/DataFilePathFactory.java:
##########
@@ -43,14 +45,18 @@ public class DataFilePathFactory {
     private final String changelogFilePrefix;
     private final boolean fileSuffixIncludeCompression;
     private final String fileCompression;
+    private final ExternalPathProvider externalPathProvider;

Review Comment:
   You should create a new class to this class, for example, DataFileExternalProvider here, which contains `relativePath` already.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Core] support write to the external path [paimon]

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #4770:
URL: https://github.com/apache/paimon/pull/4770#discussion_r1903093936


##########
paimon-common/src/main/java/org/apache/paimon/CoreOptions.java:
##########
@@ -2181,6 +2208,18 @@ public PartitionExpireStrategy partitionExpireStrategy() {
         return options.get(PARTITION_EXPIRATION_STRATEGY);
     }
 
+    public String dataFileExternalPaths() {

Review Comment:
   Please add nullable for all nullable methods and fields. ALL, not just this one.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Core] support write to the external path [paimon]

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #4770:
URL: https://github.com/apache/paimon/pull/4770#discussion_r1901527536


##########
paimon-common/src/main/java/org/apache/paimon/CoreOptions.java:
##########
@@ -124,6 +124,32 @@ public class CoreOptions implements Serializable {
                                                     + "if there is no primary key, the full row will be used.")
                                     .build());
 
+    public static final ConfigOption<String> DATA_FILE_EXTERNAL_PATHS =
+            key("data-file.external-paths")
+                    .stringType()
+                    .noDefaultValue()
+                    .withDescription(
+                            "The external paths where the data of this table will be written, "
+                                    + "multiple elements separated by commas.");
+
+    public static final ConfigOption<ExternalPathStrategy> DATA_FILE_EXTERNAL_PATHS_STRATEGY =
+            key("data-file.external-paths.strategy")
+                    .enumType(ExternalPathStrategy.class)
+                    .defaultValue(ExternalPathStrategy.NONE)
+                    .withDescription(
+                            "The strategy of selecting an external path when writing data.");
+
+    public static final ConfigOption<ExternalFSStrategy> DATA_FILE_EXTERNAL_PATHS_SPECIFIC_FS =

Review Comment:
   Maybe just use String instead of an enum?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Core] support write to the external path [paimon]

Posted by "neuyilan (via GitHub)" <gi...@apache.org>.
neuyilan commented on code in PR #4770:
URL: https://github.com/apache/paimon/pull/4770#discussion_r1901572218


##########
paimon-common/src/main/java/org/apache/paimon/CoreOptions.java:
##########
@@ -124,6 +124,32 @@ public class CoreOptions implements Serializable {
                                                     + "if there is no primary key, the full row will be used.")
                                     .build());
 
+    public static final ConfigOption<String> DATA_FILE_EXTERNAL_PATHS =
+            key("data-file.external-paths")
+                    .stringType()
+                    .noDefaultValue()
+                    .withDescription(
+                            "The external paths where the data of this table will be written, "
+                                    + "multiple elements separated by commas.");
+
+    public static final ConfigOption<ExternalPathStrategy> DATA_FILE_EXTERNAL_PATHS_STRATEGY =
+            key("data-file.external-paths.strategy")
+                    .enumType(ExternalPathStrategy.class)
+                    .defaultValue(ExternalPathStrategy.NONE)
+                    .withDescription(
+                            "The strategy of selecting an external path when writing data.");
+
+    public static final ConfigOption<ExternalFSStrategy> DATA_FILE_EXTERNAL_PATHS_SPECIFIC_FS =

Review Comment:
   Yes, string is enough



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Core] support write to the external path [paimon]

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #4770:
URL: https://github.com/apache/paimon/pull/4770#discussion_r1903093807


##########
paimon-common/src/main/java/org/apache/paimon/fs/ExternalPathProvider.java:
##########
@@ -0,0 +1,213 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.paimon.fs;
+
+import org.apache.paimon.CoreOptions.ExternalPathStrategy;
+import org.apache.paimon.annotation.VisibleForTesting;
+
+import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Optional;
+
+/** Provider for external paths. */
+public class ExternalPathProvider implements Serializable {
+    private static final String S3 = "s3";
+    private static final String OSS = "oss";
+
+    private final Map<String, Path> externalPathsMap;
+    private final List<Path> externalPathsList;
+
+    private final ExternalPathStrategy externalPathStrategy;
+    private final String externalFSStrategy;
+    private int currentIndex;
+    private boolean externalPathExists;
+    private final String dbAndTableRelativePath;
+
+    @VisibleForTesting
+    public ExternalPathProvider() {
+        this.externalPathsMap = new HashMap<>();
+        this.externalPathsList = new ArrayList<>();
+        this.externalPathStrategy = ExternalPathStrategy.NONE;
+        this.externalFSStrategy = null;
+        this.dbAndTableRelativePath = null;
+    }
+
+    public ExternalPathProvider(
+            String externalPaths,
+            ExternalPathStrategy externalPathStrategy,
+            String externalFSStrategy,
+            String dbAndTableRelativePath) {
+        this.externalPathsMap = new HashMap<>();
+        this.externalPathsList = new ArrayList<>();
+        this.externalPathStrategy = externalPathStrategy;
+        this.externalFSStrategy = externalFSStrategy;
+        this.dbAndTableRelativePath = dbAndTableRelativePath;
+        this.currentIndex = 0;
+        initExternalPaths(externalPaths);
+    }
+
+    private void initExternalPaths(String externalPaths) {
+        if (externalPaths == null) {
+            return;
+        }
+
+        if (externalPathStrategy != null
+                && externalPathStrategy.equals(ExternalPathStrategy.SPECIFIC_FS)) {
+            if (externalFSStrategy == null) {
+                throw new IllegalArgumentException("external fs strategy should not be null: ");
+            }
+        }
+
+        String[] tmpArray = externalPaths.split(",");
+        for (String part : tmpArray) {
+            String path = part.trim();
+            if (path.toLowerCase().startsWith(OSS)) {

Review Comment:
   You don't need to bind oss or s3, you can just pick the paths by starting with SpecificFS.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Core] support write to the external path [paimon]

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #4770:
URL: https://github.com/apache/paimon/pull/4770#discussion_r1903094334


##########
paimon-common/src/main/java/org/apache/paimon/fs/ExternalPathProvider.java:
##########
@@ -0,0 +1,213 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.paimon.fs;
+
+import org.apache.paimon.CoreOptions.ExternalPathStrategy;
+import org.apache.paimon.annotation.VisibleForTesting;
+
+import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Optional;
+
+/** Provider for external paths. */
+public class ExternalPathProvider implements Serializable {
+    private static final String S3 = "s3";
+    private static final String OSS = "oss";
+
+    private final Map<String, Path> externalPathsMap;
+    private final List<Path> externalPathsList;
+
+    private final ExternalPathStrategy externalPathStrategy;
+    private final String externalFSStrategy;
+    private int currentIndex;
+    private boolean externalPathExists;
+    private final String dbAndTableRelativePath;
+
+    @VisibleForTesting

Review Comment:
   Please not add this method in class, just pass null in tests.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Core] support write to the external path [paimon]

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #4770:
URL: https://github.com/apache/paimon/pull/4770#discussion_r1903093720


##########
paimon-common/src/main/java/org/apache/paimon/fs/ExternalPathProvider.java:
##########
@@ -0,0 +1,213 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.paimon.fs;
+
+import org.apache.paimon.CoreOptions.ExternalPathStrategy;
+import org.apache.paimon.annotation.VisibleForTesting;
+
+import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Optional;
+
+/** Provider for external paths. */
+public class ExternalPathProvider implements Serializable {
+    private static final String S3 = "s3";
+    private static final String OSS = "oss";
+
+    private final Map<String, Path> externalPathsMap;
+    private final List<Path> externalPathsList;
+
+    private final ExternalPathStrategy externalPathStrategy;
+    private final String externalFSStrategy;
+    private int currentIndex;
+    private boolean externalPathExists;
+    private final String dbAndTableRelativePath;

Review Comment:
   Path should be table path, so we don't need this `dbAndTableRelativePath`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Core] support write to the external path [paimon]

Posted by "wwj6591812 (via GitHub)" <gi...@apache.org>.
wwj6591812 commented on code in PR #4770:
URL: https://github.com/apache/paimon/pull/4770#discussion_r1898220263


##########
paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/clone/CopyFileOperator.java:
##########
@@ -139,6 +141,37 @@ private Path pathOfTable(Table table) {
         return new Path(table.options().get(CoreOptions.PATH.key()));
     }
 
+    private void copyManifestFile(Path sourcePath, Path targetPath, CloneFileInfo cloneFileInfo)

Review Comment:
   I think in clone action, maybe it is better to don't change file's content.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Core] support write to the external path [paimon]

Posted by "neuyilan (via GitHub)" <gi...@apache.org>.
neuyilan commented on code in PR #4770:
URL: https://github.com/apache/paimon/pull/4770#discussion_r1898194404


##########
paimon-core/src/main/java/org/apache/paimon/utils/FileStorePathFactory.java:
##########
@@ -38,7 +38,10 @@ public class FileStorePathFactory {
 
     public static final String BUCKET_PATH_PREFIX = "bucket-";
 
+    // this is the table schema root path
     private final Path root;
+    // this is the table data root path
+    private final Path dataRoot;

Review Comment:
   This is the current location where data is written, but it may not necessarily be the external path, it could also be the warehouse root, so it is called `dataRoot` here.  It is the same as `CoreOptions.TABLE_DATA_PATH`, as it description "The data file path of this table in the filesystem. if `data-file.external-path` is not set, it will be same with `PATH.key`"
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org