You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "neuyilan (via GitHub)" <gi...@apache.org> on 2024/12/23 13:20:40 UTC
[PR] [Core]Support read external path in DataFileMeta [paimon]
neuyilan opened a new pull request, #4761:
URL: https://github.com/apache/paimon/pull/4761
<!-- Please specify the module before the PR name: [core] ... or [flink] ... -->
### Purpose
https://cwiki.apache.org/confluence/display/PAIMON/PIP-29%3A+Introduce+Table+Multi-Location++Management
<!-- What is the purpose of the change -->
### Tests
<!-- List UT and IT cases to verify this change -->
### API and Format
<!-- Does this change affect API or storage format -->
### Documentation
<!-- Does this change introduce a new feature -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Core]Support read external path in DataFileMeta [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #4761:
URL: https://github.com/apache/paimon/pull/4761#discussion_r1896325555
##########
paimon-core/src/main/java/org/apache/paimon/io/KeyValueFileWriterFactory.java:
##########
@@ -142,14 +142,15 @@ private KeyValueDataFileWriter createDataFileWriter(
fileIndexOptions);
}
- public void deleteFile(String filename, int level) {
- fileIO.deleteQuietly(formatContext.pathFactory(level).toPath(filename));
+ public void deleteFile(DataFileMeta meta, int level) {
+ fileIO.deleteQuietly(formatContext.pathFactory(level).toPath(meta));
}
- public void copyFile(String sourceFileName, String targetFileName, int level)
+ public void copyFile(
+ String sourceFileName, String targetFileName, int level, String externalPath)
Review Comment:
We can pass two data file metas to this method.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Core]Support read external path in DataFileMeta [paimon]
Posted by "neuyilan (via GitHub)" <gi...@apache.org>.
neuyilan commented on code in PR #4761:
URL: https://github.com/apache/paimon/pull/4761#discussion_r1896344107
##########
paimon-core/src/main/java/org/apache/paimon/io/KeyValueDataFileWriter.java:
##########
@@ -116,6 +117,7 @@ public KeyValueDataFileWriter(
this.dataFileIndexWriter =
DataFileIndexWriter.create(
fileIO, dataFileToFileIndexPath(path), valueType, fileIndexOptions);
+ this.isExternalPath = false;
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Core]Support read external path in DataFileMeta [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #4761:
URL: https://github.com/apache/paimon/pull/4761#discussion_r1896325691
##########
paimon-core/src/main/java/org/apache/paimon/io/RowDataFileWriter.java:
##########
@@ -84,6 +85,7 @@ public RowDataFileWriter(
DataFileIndexWriter.create(
fileIO, dataFileToFileIndexPath(path), writeSchema, fileIndexOptions);
this.fileSource = fileSource;
+ this.isExternalPath = false;
Review Comment:
We don't need to introduce this bool now, remove it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Core]Support read external path in DataFileMeta [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #4761:
URL: https://github.com/apache/paimon/pull/4761
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Core]Support read external path in DataFileMeta [paimon]
Posted by "wwj6591812 (via GitHub)" <gi...@apache.org>.
wwj6591812 commented on code in PR #4761:
URL: https://github.com/apache/paimon/pull/4761#discussion_r1896352169
##########
paimon-core/src/main/java/org/apache/paimon/io/KeyValueDataFileWriter.java:
##########
@@ -195,7 +195,8 @@ public DataFileMeta result() throws IOException {
deleteRecordCount,
indexResult.embeddedIndexBytes(),
fileSource,
- valueStatsPair.getKey());
+ valueStatsPair.getKey(),
+ null);
Review Comment:
When externalPath is not null in DataFileMeta?
The config of externalPath will add in other pr?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Core]Support read external path in DataFileMeta [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #4761:
URL: https://github.com/apache/paimon/pull/4761#discussion_r1896326309
##########
paimon-core/src/main/java/org/apache/paimon/io/DataFilePathFactory.java:
##########
@@ -78,10 +78,39 @@ private Path newPath(String prefix) {
return new Path(parent, name);
}
+ @VisibleForTesting
public Path toPath(String fileName) {
return new Path(parent + "/" + fileName);
}
+ /**
+ * for read purpose.
+ *
+ * @param fileName the file name
+ * @param externalPath the external path, if null, it will use the parent path
+ * @return the file's path
+ */
+ public Path toPath(String fileName, String externalPath) {
+ if (externalPath == null) {
+ return new Path(parent + "/" + fileName);
+ }
+ return new Path(externalPath + "/" + fileName);
+ }
+
+ public Path toPath(DataFileMeta dataFileMeta) {
+ if (dataFileMeta.externalPath() == null) {
+ return new Path(parent + "/" + dataFileMeta.fileName());
+ }
+ return new Path(dataFileMeta.externalPath() + "/" + dataFileMeta.fileName());
+ }
+
+ public Path toIndexPath(DataFileMeta dataFileMeta, String indexFileName) {
Review Comment:
`toExtraFilePath(DataFileMeta dataFileMeta, String extraFile)`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Core]Support read external path in DataFileMeta [paimon]
Posted by "neuyilan (via GitHub)" <gi...@apache.org>.
neuyilan commented on code in PR #4761:
URL: https://github.com/apache/paimon/pull/4761#discussion_r1896380035
##########
paimon-core/src/main/java/org/apache/paimon/io/KeyValueDataFileWriter.java:
##########
@@ -177,6 +179,7 @@ public DataFileMeta result() throws IOException {
? DataFileIndexWriter.EMPTY_RESULT
: dataFileIndexWriter.result();
+ String externalPath = isExternalPath ? path.getParent().toString() : null;
Review Comment:
have been removed, will be introduced in next PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Core]Support read external path in DataFileMeta [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on PR #4761:
URL: https://github.com/apache/paimon/pull/4761#issuecomment-2560684793
+1 Thanks @neuyilan for updating!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Core]Support read external path in DataFileMeta [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #4761:
URL: https://github.com/apache/paimon/pull/4761#discussion_r1896324028
##########
paimon-core/src/main/java/org/apache/paimon/manifest/SimpleFileEntry.java:
##########
@@ -38,6 +38,7 @@ public class SimpleFileEntry implements FileEntry {
@Nullable private final byte[] embeddedIndex;
private final BinaryRow minKey;
private final BinaryRow maxKey;
+ private final String externalPath;
Review Comment:
Don't forget to add Nullable annotation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Core]Support read external path in DataFileMeta [paimon]
Posted by "neuyilan (via GitHub)" <gi...@apache.org>.
neuyilan commented on code in PR #4761:
URL: https://github.com/apache/paimon/pull/4761#discussion_r1896325253
##########
paimon-core/src/main/java/org/apache/paimon/manifest/SimpleFileEntry.java:
##########
@@ -38,6 +38,7 @@ public class SimpleFileEntry implements FileEntry {
@Nullable private final byte[] embeddedIndex;
private final BinaryRow minKey;
private final BinaryRow maxKey;
+ private final String externalPath;
Review Comment:
got it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Core]Support read external path in DataFileMeta [paimon]
Posted by "neuyilan (via GitHub)" <gi...@apache.org>.
neuyilan commented on code in PR #4761:
URL: https://github.com/apache/paimon/pull/4761#discussion_r1896343987
##########
paimon-core/src/main/java/org/apache/paimon/io/DataFileMeta.java:
##########
@@ -135,7 +135,8 @@ public static DataFileMeta forAppend(
List<String> extraFiles,
Review Comment:
done
##########
paimon-core/src/main/java/org/apache/paimon/io/KeyValueFileWriterFactory.java:
##########
@@ -142,14 +142,15 @@ private KeyValueDataFileWriter createDataFileWriter(
fileIndexOptions);
}
- public void deleteFile(String filename, int level) {
- fileIO.deleteQuietly(formatContext.pathFactory(level).toPath(filename));
+ public void deleteFile(DataFileMeta meta, int level) {
+ fileIO.deleteQuietly(formatContext.pathFactory(level).toPath(meta));
}
- public void copyFile(String sourceFileName, String targetFileName, int level)
+ public void copyFile(
+ String sourceFileName, String targetFileName, int level, String externalPath)
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Core]Support read external path in DataFileMeta [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #4761:
URL: https://github.com/apache/paimon/pull/4761#discussion_r1895889421
##########
paimon-core/src/main/java/org/apache/paimon/io/DataFilePathFactory.java:
##########
@@ -82,6 +82,20 @@ public Path toPath(String fileName) {
return new Path(parent + "/" + fileName);
}
+ /**
+ * for read purpose.
+ *
+ * @param fileName the file name
+ * @param externalPath the external path, if null, it will use the parent path
+ * @return the file's path
+ */
+ public Path toPath(String fileName, String externalPath) {
Review Comment:
Can we introduce `toPath(DataFileMeta)` and remove `toPath(String fileName)`?
- For Index file, you can add a `Optional<Path> toIndexPath(DataFileMeta)`, index file should be the same directory of data file.
- To `ChangelogCompactTask`, changelog file should be the same directory of base file too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Core]Support read external path in DataFileMeta [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #4761:
URL: https://github.com/apache/paimon/pull/4761#discussion_r1896329421
##########
paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/compact/changelog/ChangelogCompactTask.java:
##########
@@ -171,7 +171,8 @@ private List<Committable> produceNewCommittables(
realName
+ "."
+ CompactedChangelogReadOnlyFormat.getIdentifier(
- baseResult.meta.fileFormat())));
+ baseResult.meta.fileFormat()),
+ baseResult.meta.externalPath()));
Review Comment:
Here you should use `toExtraFilePath`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Core]Support read external path in DataFileMeta [paimon]
Posted by "neuyilan (via GitHub)" <gi...@apache.org>.
neuyilan commented on code in PR #4761:
URL: https://github.com/apache/paimon/pull/4761#discussion_r1896380271
##########
paimon-core/src/main/java/org/apache/paimon/io/KeyValueDataFileWriter.java:
##########
@@ -195,7 +195,8 @@ public DataFileMeta result() throws IOException {
deleteRecordCount,
indexResult.embeddedIndexBytes(),
fileSource,
- valueStatsPair.getKey());
+ valueStatsPair.getKey(),
+ null);
Review Comment:
Yes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Core]Support read external path in DataFileMeta [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #4761:
URL: https://github.com/apache/paimon/pull/4761#discussion_r1896326551
##########
paimon-core/src/main/java/org/apache/paimon/io/KeyValueDataFileWriter.java:
##########
@@ -116,6 +117,7 @@ public KeyValueDataFileWriter(
this.dataFileIndexWriter =
DataFileIndexWriter.create(
fileIO, dataFileToFileIndexPath(path), valueType, fileIndexOptions);
+ this.isExternalPath = false;
Review Comment:
Remove it, we don't need to introduce this now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Core]Support read external path in DataFileMeta [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #4761:
URL: https://github.com/apache/paimon/pull/4761#discussion_r1896325228
##########
paimon-core/src/main/java/org/apache/paimon/io/DataFileMeta.java:
##########
@@ -135,7 +135,8 @@ public static DataFileMeta forAppend(
List<String> extraFiles,
Review Comment:
rename method should be changed to new path.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Core]Support read external path in DataFileMeta [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #4761:
URL: https://github.com/apache/paimon/pull/4761#discussion_r1895889421
##########
paimon-core/src/main/java/org/apache/paimon/io/DataFilePathFactory.java:
##########
@@ -82,6 +82,20 @@ public Path toPath(String fileName) {
return new Path(parent + "/" + fileName);
}
+ /**
+ * for read purpose.
+ *
+ * @param fileName the file name
+ * @param externalPath the external path, if null, it will use the parent path
+ * @return the file's path
+ */
+ public Path toPath(String fileName, String externalPath) {
Review Comment:
Can we introduce `toPath(DataFileMeta)` and remove `toPath(String fileName)`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Core]Support read external path in DataFileMeta [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #4761:
URL: https://github.com/apache/paimon/pull/4761#discussion_r1896324216
##########
paimon-core/src/main/java/org/apache/paimon/manifest/FileEntry.java:
##########
@@ -56,6 +56,8 @@ public interface FileEntry {
Identifier identifier();
+ String externalPath();
Review Comment:
I think externalPath should also be a field for Identifier?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Core]Support read external path in DataFileMeta [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #4761:
URL: https://github.com/apache/paimon/pull/4761#discussion_r1896327979
##########
paimon-core/src/main/java/org/apache/paimon/manifest/SimpleFileEntrySerializer.java:
##########
@@ -63,6 +63,7 @@ public SimpleFileEntry convertFrom(int version, InternalRow row) {
fromStringArrayData(file.getArray(11)),
Review Comment:
this class is useless now, remove it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Core]Support read external path in DataFileMeta [paimon]
Posted by "neuyilan (via GitHub)" <gi...@apache.org>.
neuyilan commented on code in PR #4761:
URL: https://github.com/apache/paimon/pull/4761#discussion_r1896325933
##########
paimon-core/src/main/java/org/apache/paimon/io/RowDataFileWriter.java:
##########
@@ -84,6 +85,7 @@ public RowDataFileWriter(
DataFileIndexWriter.create(
fileIO, dataFileToFileIndexPath(path), writeSchema, fileIndexOptions);
this.fileSource = fileSource;
+ this.isExternalPath = false;
Review Comment:
ok
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Core]Support read external path in DataFileMeta [paimon]
Posted by "wwj6591812 (via GitHub)" <gi...@apache.org>.
wwj6591812 commented on code in PR #4761:
URL: https://github.com/apache/paimon/pull/4761#discussion_r1896340665
##########
paimon-core/src/main/java/org/apache/paimon/io/DataFilePathFactory.java:
##########
@@ -78,10 +78,39 @@ private Path newPath(String prefix) {
return new Path(parent, name);
}
+ @VisibleForTesting
public Path toPath(String fileName) {
return new Path(parent + "/" + fileName);
}
+ /**
+ * for read purpose.
+ *
+ * @param fileName the file name
+ * @param externalPath the external path, if null, it will use the parent path
+ * @return the file's path
+ */
+ public Path toPath(String fileName, String externalPath) {
+ if (externalPath == null) {
+ return new Path(parent + "/" + fileName);
+ }
+ return new Path(externalPath + "/" + fileName);
+ }
+
+ public Path toPath(DataFileMeta dataFileMeta) {
Review Comment:
String externalPath = dataFileMeta.externalPath();
String fileName = dataFileMeta.fileName();
return new Path((externalPath == null ? parent : externalPath) + "/" + fileName);
##########
paimon-core/src/main/java/org/apache/paimon/io/DataFilePathFactory.java:
##########
@@ -78,10 +78,39 @@ private Path newPath(String prefix) {
return new Path(parent, name);
}
+ @VisibleForTesting
public Path toPath(String fileName) {
return new Path(parent + "/" + fileName);
}
+ /**
+ * for read purpose.
+ *
+ * @param fileName the file name
+ * @param externalPath the external path, if null, it will use the parent path
+ * @return the file's path
+ */
+ public Path toPath(String fileName, String externalPath) {
+ if (externalPath == null) {
+ return new Path(parent + "/" + fileName);
+ }
+ return new Path(externalPath + "/" + fileName);
+ }
+
+ public Path toPath(DataFileMeta dataFileMeta) {
+ if (dataFileMeta.externalPath() == null) {
+ return new Path(parent + "/" + dataFileMeta.fileName());
+ }
+ return new Path(dataFileMeta.externalPath() + "/" + dataFileMeta.fileName());
+ }
+
+ public Path toIndexPath(DataFileMeta dataFileMeta, String indexFileName) {
Review Comment:
String externalPath = dataFileMeta.externalPath();
return new Path((externalPath == null ? parent : externalPath) + "/" + indexFileName);
##########
paimon-core/src/main/java/org/apache/paimon/io/KeyValueDataFileWriter.java:
##########
@@ -177,6 +179,7 @@ public DataFileMeta result() throws IOException {
? DataFileIndexWriter.EMPTY_RESULT
: dataFileIndexWriter.result();
+ String externalPath = isExternalPath ? path.getParent().toString() : null;
Review Comment:
When isExternalPath is true?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org