You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2020/08/20 11:12:59 UTC
[GitHub] [hudi] xushiyan opened a new pull request #1997: [HUDI-781] Introduce HoodieTestTable for test preparation
xushiyan opened a new pull request #1997:
URL: https://github.com/apache/hudi/pull/1997
Redo #1871
## Committer checklist
- [ ] Has a corresponding JIRA in PR title & commit
- [ ] Commit message is descriptive of the change
- [ ] CI is green
- [ ] Necessary doc changes done or have another open PR
- [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yanghua commented on a change in pull request #1997: [HUDI-781] Introduce HoodieTestTable for test preparation
Posted by GitBox <gi...@apache.org>.
yanghua commented on a change in pull request #1997:
URL: https://github.com/apache/hudi/pull/1997#discussion_r474344290
##########
File path: hudi-client/src/test/java/org/apache/hudi/testutils/HoodieClientTestUtils.java
##########
@@ -227,6 +174,8 @@ public static SparkConf getSparkConfForTest(String appName) {
/**
* Find total basefiles for passed in paths.
+ * <p>
+ * TODO move to {@link FileCreateUtils}.
Review comment:
And here?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yanghua commented on a change in pull request #1997: [HUDI-781] Introduce HoodieTestTable for test preparation
Posted by GitBox <gi...@apache.org>.
yanghua commented on a change in pull request #1997:
URL: https://github.com/apache/hudi/pull/1997#discussion_r474071930
##########
File path: hudi-client/src/test/java/org/apache/hudi/testutils/HoodieClientTestUtils.java
##########
@@ -289,27 +245,9 @@ public static String writeParquetFile(String basePath, String partitionPath, Lis
createCommitTime);
}
- public static String createNewMarkerFile(String basePath, String partitionPath, String instantTime)
- throws IOException {
- return createMarkerFile(basePath, partitionPath, instantTime);
- }
-
- public static String createMarkerFile(String basePath, String partitionPath, String instantTime)
- throws IOException {
- return createMarkerFile(basePath, partitionPath, instantTime, UUID.randomUUID().toString(), IOType.MERGE);
- }
-
- public static String createMarkerFile(String basePath, String partitionPath, String instantTime, String fileID, IOType ioType)
- throws IOException {
- String folderPath = basePath + "/" + HoodieTableMetaClient.TEMPFOLDER_NAME + "/" + instantTime + "/" + partitionPath + "/";
- new File(folderPath).mkdirs();
- String markerFileName = String.format("%s_%s_%s%s%s.%s", fileID, DEFAULT_WRITE_TOKEN, instantTime,
- HoodieFileFormat.PARQUET.getFileExtension(), HoodieTableMetaClient.MARKER_EXTN, ioType);
- File f = new File(folderPath + markerFileName);
- f.createNewFile();
- return f.getAbsolutePath();
- }
-
+ /**
+ * TODO move to {@link FileCreateUtils}.
Review comment:
Why do not move it right now?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yanghua merged pull request #1997: [HUDI-781] Introduce HoodieTestTable for test preparation
Posted by GitBox <gi...@apache.org>.
yanghua merged pull request #1997:
URL: https://github.com/apache/hudi/pull/1997
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan commented on a change in pull request #1997: [HUDI-781] Introduce HoodieTestTable for test preparation
Posted by GitBox <gi...@apache.org>.
xushiyan commented on a change in pull request #1997:
URL: https://github.com/apache/hudi/pull/1997#discussion_r474349878
##########
File path: hudi-client/src/test/java/org/apache/hudi/testutils/HoodieClientTestUtils.java
##########
@@ -227,6 +174,8 @@ public static SparkConf getSparkConfForTest(String appName) {
/**
* Find total basefiles for passed in paths.
+ * <p>
+ * TODO move to {@link FileCreateUtils}.
Review comment:
yup good catch.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan commented on pull request #1997: [HUDI-781] Introduce HoodieTestTable for test preparation
Posted by GitBox <gi...@apache.org>.
xushiyan commented on pull request #1997:
URL: https://github.com/apache/hudi/pull/1997#issuecomment-677546876
@yanghua This is to redo #1871, mostly resolving conflicts in `TestCleaner.java`. Thanks.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan commented on a change in pull request #1997: [HUDI-781] Introduce HoodieTestTable for test preparation
Posted by GitBox <gi...@apache.org>.
xushiyan commented on a change in pull request #1997:
URL: https://github.com/apache/hudi/pull/1997#discussion_r474145442
##########
File path: hudi-client/src/test/java/org/apache/hudi/testutils/HoodieClientTestUtils.java
##########
@@ -289,27 +245,9 @@ public static String writeParquetFile(String basePath, String partitionPath, Lis
createCommitTime);
}
- public static String createNewMarkerFile(String basePath, String partitionPath, String instantTime)
- throws IOException {
- return createMarkerFile(basePath, partitionPath, instantTime);
- }
-
- public static String createMarkerFile(String basePath, String partitionPath, String instantTime)
- throws IOException {
- return createMarkerFile(basePath, partitionPath, instantTime, UUID.randomUUID().toString(), IOType.MERGE);
- }
-
- public static String createMarkerFile(String basePath, String partitionPath, String instantTime, String fileID, IOType ioType)
- throws IOException {
- String folderPath = basePath + "/" + HoodieTableMetaClient.TEMPFOLDER_NAME + "/" + instantTime + "/" + partitionPath + "/";
- new File(folderPath).mkdirs();
- String markerFileName = String.format("%s_%s_%s%s%s.%s", fileID, DEFAULT_WRITE_TOKEN, instantTime,
- HoodieFileFormat.PARQUET.getFileExtension(), HoodieTableMetaClient.MARKER_EXTN, ioType);
- File f = new File(folderPath + markerFileName);
- f.createNewFile();
- return f.getAbsolutePath();
- }
-
+ /**
+ * TODO move to {@link FileCreateUtils}.
Review comment:
yea good candidate to be moved.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org