You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2020/01/27 23:35:14 UTC
[GitHub] [incubator-hudi] prashantwason opened a new pull request #1286:
[HUDI-562] Added new test cases for HoodieLogFormat and
HoodieLogFormatVersion.
prashantwason opened a new pull request #1286: [HUDI-562] Added new test cases for HoodieLogFormat and HoodieLogFormatVersion.
URL: https://github.com/apache/incubator-hudi/pull/1286
## What is the purpose of the pull request
Added new test cases for HoodieLogFormat and HoodieLogFormatVersion.
## Verify this pull request
This change added tests and can be verified as follows:
This change added tests and can be verified as follows:
mvn test --also-make -DfailIfNoTests=false -Dtest=TestHoodieLogFormat,TestHoodieLogFormatVersion -pl hudi-common
## Committer checklist
- [ ] Has a corresponding JIRA in PR title & commit
- [ ] Commit message is descriptive of the change
- [ ] CI is green
- [ ] Necessary doc changes done or have another open PR
- [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] n3nash merged pull request #1286: [HUDI-564] Added
new test cases for HoodieLogFormat and HoodieLogFormatVersion.
Posted by GitBox <gi...@apache.org>.
n3nash merged pull request #1286: [HUDI-564] Added new test cases for HoodieLogFormat and HoodieLogFormatVersion.
URL: https://github.com/apache/incubator-hudi/pull/1286
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] n3nash commented on a change in pull request
#1286: [HUDI-564] Added new test cases for HoodieLogFormat and
HoodieLogFormatVersion.
Posted by GitBox <gi...@apache.org>.
n3nash commented on a change in pull request #1286: [HUDI-564] Added new test cases for HoodieLogFormat and HoodieLogFormatVersion.
URL: https://github.com/apache/incubator-hudi/pull/1286#discussion_r372566780
##########
File path: hudi-common/src/test/java/org/apache/hudi/common/table/log/TestHoodieLogFormat.java
##########
@@ -1296,4 +1297,36 @@ public void testBasicAppendAndTraverseInReverse() throws IOException, URISyntaxE
assertFalse(reader.hasPrev());
reader.close();
}
+
+ @Test
+ public void testV0Format() throws IOException, InterruptedException, URISyntaxException {
+ // HoodieLogFormatVersion.DEFAULT_VERSION has been deprecated so we cannot
+ // create a writer for it. So these tests are only for the HoodieAvroDataBlock
+ // of older version.
+ Schema schema = getSimpleSchema();
+ List<IndexedRecord> records = SchemaTestUtil.generateTestRecords(0, 100);
+ List<IndexedRecord> recordsCopy = new ArrayList<>(records);
+ assertEquals(records.size(), 100);
+ assertEquals(recordsCopy.size(), 100);
+ HoodieAvroDataBlock dataBlock = new HoodieAvroDataBlock(records, schema);
+ byte[] content = dataBlock.getBytes(schema);
+ assertTrue(content.length > 0);
+
+ HoodieLogBlock logBlock = HoodieAvroDataBlock.getBlock(content, schema);
+ assertEquals(logBlock.getBlockType(), HoodieLogBlockType.AVRO_DATA_BLOCK);
+ List<IndexedRecord> readRecords = ((HoodieAvroDataBlock)logBlock).getRecords();
+ assertEquals(readRecords.size(), recordsCopy.size());
+ for (int i = 0; i < recordsCopy.size(); ++i) {
+ assertEquals(recordsCopy.get(i), readRecords.get(i));
+ }
+
+ // Reader scheme is optional if it is same as write schema
Review comment:
nit : s/scheme/schema/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] n3nash commented on issue #1286: [HUDI-564] Added
new test cases for HoodieLogFormat and HoodieLogFormatVersion.
Posted by GitBox <gi...@apache.org>.
n3nash commented on issue #1286: [HUDI-564] Added new test cases for HoodieLogFormat and HoodieLogFormatVersion.
URL: https://github.com/apache/incubator-hudi/pull/1286#issuecomment-579903831
@yanghua I think the spark-avro library downloading is flaky which is causing the builds to fail intermittently, we need to look into it
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] yanghua commented on issue #1286: [HUDI-564] Added
new test cases for HoodieLogFormat and HoodieLogFormatVersion.
Posted by GitBox <gi...@apache.org>.
yanghua commented on issue #1286: [HUDI-564] Added new test cases for HoodieLogFormat and HoodieLogFormatVersion.
URL: https://github.com/apache/incubator-hudi/pull/1286#issuecomment-579053963
@prashantwason The Travis is red, please recheck your change and the failed reason.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] prashantwason commented on issue #1286: [HUDI-564]
Added new test cases for HoodieLogFormat and HoodieLogFormatVersion.
Posted by GitBox <gi...@apache.org>.
prashantwason commented on issue #1286: [HUDI-564] Added new test cases for HoodieLogFormat and HoodieLogFormatVersion.
URL: https://github.com/apache/incubator-hudi/pull/1286#issuecomment-579381907
The failure is in hudi-integ-test which seems unrelated to my change. I will investigate the cause.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] prashantwason commented on a change in pull
request #1286: [HUDI-564] Added new test cases for HoodieLogFormat and
HoodieLogFormatVersion.
Posted by GitBox <gi...@apache.org>.
prashantwason commented on a change in pull request #1286: [HUDI-564] Added new test cases for HoodieLogFormat and HoodieLogFormatVersion.
URL: https://github.com/apache/incubator-hudi/pull/1286#discussion_r372571648
##########
File path: hudi-common/src/test/java/org/apache/hudi/common/table/log/TestHoodieLogFormat.java
##########
@@ -1296,4 +1297,36 @@ public void testBasicAppendAndTraverseInReverse() throws IOException, URISyntaxE
assertFalse(reader.hasPrev());
reader.close();
}
+
+ @Test
+ public void testV0Format() throws IOException, InterruptedException, URISyntaxException {
+ // HoodieLogFormatVersion.DEFAULT_VERSION has been deprecated so we cannot
+ // create a writer for it. So these tests are only for the HoodieAvroDataBlock
+ // of older version.
+ Schema schema = getSimpleSchema();
+ List<IndexedRecord> records = SchemaTestUtil.generateTestRecords(0, 100);
+ List<IndexedRecord> recordsCopy = new ArrayList<>(records);
+ assertEquals(records.size(), 100);
+ assertEquals(recordsCopy.size(), 100);
+ HoodieAvroDataBlock dataBlock = new HoodieAvroDataBlock(records, schema);
+ byte[] content = dataBlock.getBytes(schema);
+ assertTrue(content.length > 0);
+
+ HoodieLogBlock logBlock = HoodieAvroDataBlock.getBlock(content, schema);
+ assertEquals(logBlock.getBlockType(), HoodieLogBlockType.AVRO_DATA_BLOCK);
+ List<IndexedRecord> readRecords = ((HoodieAvroDataBlock)logBlock).getRecords();
+ assertEquals(readRecords.size(), recordsCopy.size());
+ for (int i = 0; i < recordsCopy.size(); ++i) {
+ assertEquals(recordsCopy.get(i), readRecords.get(i));
+ }
+
+ // Reader scheme is optional if it is same as write schema
Review comment:
Fixed
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] yanghua edited a comment on issue #1286:
[HUDI-564] Added new test cases for HoodieLogFormat and
HoodieLogFormatVersion.
Posted by GitBox <gi...@apache.org>.
yanghua edited a comment on issue #1286: [HUDI-564] Added new test cases for HoodieLogFormat and HoodieLogFormatVersion.
URL: https://github.com/apache/incubator-hudi/pull/1286#issuecomment-579053963
@prashantwason The Travis is red, please recheck your change and the failure reason.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services