You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2020/09/28 21:46:30 UTC
[GitHub] [hudi] prashantwason opened a new pull request #2128: [HUDI-1303] Some improvements for the HUDI Test Suite.
prashantwason opened a new pull request #2128:
URL: https://github.com/apache/hudi/pull/2128
## What is the purpose of the pull request
1. Use the DAG Node's label from the yaml as its name instead of UUID names which are not descriptive when debugging issues from logs.
2. Fix CleanNode constructor which is not correctly implemented
3. When generating upsets, allows more granualar control over the number of inserts and upserts - zero or more inserts and upserts can be specified instead of always requiring both inserts and upserts.
4. Fixed generation of records of specific size
- The current code was using a class variable "shouldAddMore" which was reset to false after the first record generation causing subsequent records to be of minimum size.
- In this change, we pre-calculate the extra size of the complex fields. When generating records, for complex fields we read the field size from this map.
5. Refresh the timeline of the DeltaSync service before calling readFromSource. This ensures that only the newest generated data is read and data generated in the older Dag Nodes is ignored (as their AVRO files will have an older timestamp).
6. Making --workload-generator-classname an optional parameter as most probably the default will be used
## Brief change log
## Verify this pull request
This pull request is a trivial rework / code cleanup without any test coverage.
## Committer checklist
- [ ] Has a corresponding JIRA in PR title & commit
- [ ] Commit message is descriptive of the change
- [ ] CI is green
- [ ] Necessary doc changes done or have another open PR
- [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] prashantwason commented on a change in pull request #2128: [HUDI-1303] Some improvements for the HUDI Test Suite.
Posted by GitBox <gi...@apache.org>.
prashantwason commented on a change in pull request #2128:
URL: https://github.com/apache/hudi/pull/2128#discussion_r499827933
##########
File path: hudi-integ-test/src/main/java/org/apache/hudi/integ/testsuite/generator/GenericRecordFullPayloadGenerator.java
##########
@@ -333,23 +312,37 @@ private int getSize(Schema elementSchema) {
* @param elementSchema
* @return Number of entries to add
*/
- private int numEntriesToAdd(Schema elementSchema) {
- // Find the size of the primitive data type in bytes
- int primitiveDataTypeSize = getSize(elementSchema);
- int numEntriesToAdd = numberOfBytesToAdd / primitiveDataTypeSize;
- // If more than 10 entries are being added for this same complex field and there are still more complex fields to
- // be visited in the schema, reduce the number of entries to add by a factor of 10 to allow for other complex
- // fields to pack some entries
- if (numEntriesToAdd % 10 > 0 && this.numberOfComplexFields > 1) {
- numEntriesToAdd = numEntriesToAdd / 10;
- numberOfBytesToAdd -= numEntriesToAdd * primitiveDataTypeSize;
- this.shouldAddMore = true;
- } else {
- this.numberOfBytesToAdd = 0;
- this.shouldAddMore = false;
+ private void determineExtraEntriesRequired(int numberOfComplexFields, int numberOfBytesToAdd) {
+ for (Schema.Field f : baseSchema.getFields()) {
+ Schema elementSchema = f.schema();
+ // Find the size of the primitive data type in bytes
+ int primitiveDataTypeSize = 0;
+ if (elementSchema.getType() == Type.ARRAY && isPrimitive(elementSchema.getElementType())) {
+ primitiveDataTypeSize = getSize(elementSchema.getElementType());
+ } else if (elementSchema.getType() == Type.MAP && isPrimitive(elementSchema.getValueType())) {
+ primitiveDataTypeSize = getSize(elementSchema.getValueType());
+ } else {
+ continue;
+ }
+
+ int numEntriesToAdd = numberOfBytesToAdd / primitiveDataTypeSize;
+ // If more than 10 entries are being added for this same complex field and there are still more complex fields to
+ // be visited in the schema, reduce the number of entries to add by a factor of 10 to allow for other complex
+ // fields to pack some entries
+ if (numEntriesToAdd > 10 && numberOfComplexFields > 1) {
Review comment:
The idea here is to distribute the extra-entries (ins some way) across all complex fields.
- If there is only one complex field then all extra bytes will be added to that single entry.
int numEntriesToAdd = numberOfBytesToAdd / primitiveDataTypeSize;
...
extraEntriesMap.put(f.name(), numEntriesToAdd);
- If there are more than 1 complex field, then we add atleast 10 entries to the first one, and so on....
The perfect solution is to equally divide the extra bytes across all complex fields but I have not covered that yet. (slightly complicated as each field may have different size, etc).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] vinothchandar commented on pull request #2128: [HUDI-1303] Some improvements for the HUDI Test Suite.
Posted by GitBox <gi...@apache.org>.
vinothchandar commented on pull request #2128:
URL: https://github.com/apache/hudi/pull/2128#issuecomment-703192490
@nsivabalan are you able to review and land this?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] prashantwason commented on a change in pull request #2128: [HUDI-1303] Some improvements for the HUDI Test Suite.
Posted by GitBox <gi...@apache.org>.
prashantwason commented on a change in pull request #2128:
URL: https://github.com/apache/hudi/pull/2128#discussion_r499825998
##########
File path: hudi-integ-test/src/main/java/org/apache/hudi/integ/testsuite/generator/GenericRecordFullPayloadGenerator.java
##########
@@ -205,45 +193,36 @@ private Object typeConvert(Schema schema) {
case LONG:
return getNextConstrainedLong();
case STRING:
- return UUID.randomUUID().toString();
+ return UUID.randomUUID().toString();
case ENUM:
- List<String> enumSymbols = localSchema.getEnumSymbols();
- return new GenericData.EnumSymbol(localSchema, enumSymbols.get(random.nextInt(enumSymbols.size() - 1)));
+ List<String> enumSymbols = fieldSchema.getEnumSymbols();
+ return new GenericData.EnumSymbol(fieldSchema, enumSymbols.get(random.nextInt(enumSymbols.size() - 1)));
case RECORD:
- return convert(localSchema);
+ return getNewPayload(fieldSchema);
case ARRAY:
- Schema elementSchema = localSchema.getElementType();
+ Schema.Field elementField = new Schema.Field(field.name(), fieldSchema.getElementType(), "", null);
List listRes = new ArrayList();
- if (isPrimitive(elementSchema) && this.shouldAddMore) {
- int numEntriesToAdd = numEntriesToAdd(elementSchema);
- while (numEntriesToAdd > 0) {
- listRes.add(typeConvert(elementSchema));
- numEntriesToAdd--;
- }
- } else {
- listRes.add(typeConvert(elementSchema));
+ int numEntriesToAdd = extraEntriesMap.getOrDefault(field.name(), 1);
+ while (numEntriesToAdd > 0) {
Review comment:
Done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan merged pull request #2128: [HUDI-1303] Some improvements for the HUDI Test Suite.
Posted by GitBox <gi...@apache.org>.
nsivabalan merged pull request #2128:
URL: https://github.com/apache/hudi/pull/2128
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a change in pull request #2128: [HUDI-1303] Some improvements for the HUDI Test Suite.
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on a change in pull request #2128:
URL: https://github.com/apache/hudi/pull/2128#discussion_r499276941
##########
File path: hudi-integ-test/src/main/java/org/apache/hudi/integ/testsuite/generator/GenericRecordFullPayloadGenerator.java
##########
@@ -333,23 +312,37 @@ private int getSize(Schema elementSchema) {
* @param elementSchema
* @return Number of entries to add
*/
- private int numEntriesToAdd(Schema elementSchema) {
- // Find the size of the primitive data type in bytes
- int primitiveDataTypeSize = getSize(elementSchema);
- int numEntriesToAdd = numberOfBytesToAdd / primitiveDataTypeSize;
- // If more than 10 entries are being added for this same complex field and there are still more complex fields to
- // be visited in the schema, reduce the number of entries to add by a factor of 10 to allow for other complex
- // fields to pack some entries
- if (numEntriesToAdd % 10 > 0 && this.numberOfComplexFields > 1) {
- numEntriesToAdd = numEntriesToAdd / 10;
- numberOfBytesToAdd -= numEntriesToAdd * primitiveDataTypeSize;
- this.shouldAddMore = true;
- } else {
- this.numberOfBytesToAdd = 0;
- this.shouldAddMore = false;
+ private void determineExtraEntriesRequired(int numberOfComplexFields, int numberOfBytesToAdd) {
+ for (Schema.Field f : baseSchema.getFields()) {
+ Schema elementSchema = f.schema();
+ // Find the size of the primitive data type in bytes
+ int primitiveDataTypeSize = 0;
+ if (elementSchema.getType() == Type.ARRAY && isPrimitive(elementSchema.getElementType())) {
+ primitiveDataTypeSize = getSize(elementSchema.getElementType());
+ } else if (elementSchema.getType() == Type.MAP && isPrimitive(elementSchema.getValueType())) {
+ primitiveDataTypeSize = getSize(elementSchema.getValueType());
+ } else {
+ continue;
+ }
+
+ int numEntriesToAdd = numberOfBytesToAdd / primitiveDataTypeSize;
+ // If more than 10 entries are being added for this same complex field and there are still more complex fields to
+ // be visited in the schema, reduce the number of entries to add by a factor of 10 to allow for other complex
+ // fields to pack some entries
+ if (numEntriesToAdd > 10 && numberOfComplexFields > 1) {
Review comment:
what happens if there is only one complexField and numEntriesToAdd is > 10 ?
##########
File path: hudi-integ-test/src/main/java/org/apache/hudi/integ/testsuite/generator/GenericRecordFullPayloadGenerator.java
##########
@@ -205,45 +193,36 @@ private Object typeConvert(Schema schema) {
case LONG:
return getNextConstrainedLong();
case STRING:
- return UUID.randomUUID().toString();
+ return UUID.randomUUID().toString();
case ENUM:
- List<String> enumSymbols = localSchema.getEnumSymbols();
- return new GenericData.EnumSymbol(localSchema, enumSymbols.get(random.nextInt(enumSymbols.size() - 1)));
+ List<String> enumSymbols = fieldSchema.getEnumSymbols();
+ return new GenericData.EnumSymbol(fieldSchema, enumSymbols.get(random.nextInt(enumSymbols.size() - 1)));
case RECORD:
- return convert(localSchema);
+ return getNewPayload(fieldSchema);
case ARRAY:
- Schema elementSchema = localSchema.getElementType();
+ Schema.Field elementField = new Schema.Field(field.name(), fieldSchema.getElementType(), "", null);
List listRes = new ArrayList();
- if (isPrimitive(elementSchema) && this.shouldAddMore) {
- int numEntriesToAdd = numEntriesToAdd(elementSchema);
- while (numEntriesToAdd > 0) {
- listRes.add(typeConvert(elementSchema));
- numEntriesToAdd--;
- }
- } else {
- listRes.add(typeConvert(elementSchema));
+ int numEntriesToAdd = extraEntriesMap.getOrDefault(field.name(), 1);
+ while (numEntriesToAdd > 0) {
Review comment:
minor. we might as well do while (numEntriesToAdd-- > 0) {
and remove line 208.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org