You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2020/03/11 00:00:00 UTC
[GitHub] [incubator-hudi] nsivabalan opened a new pull request #1395:
[HUDI-667] Fixing delete tests for DeltaStreamer
nsivabalan opened a new pull request #1395: [HUDI-667] Fixing delete tests for DeltaStreamer
URL: https://github.com/apache/incubator-hudi/pull/1395
PR fixes a bug in delete record generation for tests in hoodie delta streamer.
## Brief change log
- Fixing a bug in delete record generation for tests in hoodie delta streamer
## Verify this pull request
This pull request is already covered by existing tests. Most tests in TestHoodieDeltaStreamer tests deletes. Have to make some fixes to continuous tests as part of the bug fix.
## Committer checklist
- [ ] Has a corresponding JIRA in PR title & commit
- [ ] Commit message is descriptive of the change
- [ ] CI is green
- [ ] Necessary doc changes done or have another open PR
- [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] pratyakshsharma commented on a change in pull
request #1395: [HUDI-667] Fixing delete tests for DeltaStreamer
Posted by GitBox <gi...@apache.org>.
pratyakshsharma commented on a change in pull request #1395: [HUDI-667] Fixing delete tests for DeltaStreamer
URL: https://github.com/apache/incubator-hudi/pull/1395#discussion_r392120925
##########
File path: hudi-client/src/test/java/org/apache/hudi/common/HoodieTestDataGenerator.java
##########
@@ -429,24 +428,24 @@ public HoodieRecord generateUpdateRecord(HoodieKey key, String commitTime) throw
*/
public Stream<HoodieKey> generateUniqueDeleteStream(Integer n) {
final Set<KeyPartition> used = new HashSet<>();
-
if (n > numExistingKeys) {
throw new IllegalArgumentException("Requested unique deletes is greater than number of available keys");
}
- return IntStream.range(0, n).boxed().map(i -> {
- int index = numExistingKeys == 1 ? 0 : RAND.nextInt(numExistingKeys - 1);
- KeyPartition kp = existingKeys.get(index);
- // Find the available keyPartition starting from randomly chosen one.
- while (used.contains(kp)) {
+ List<HoodieKey> result = new ArrayList<>();
+ for (int i = 0; i < n; i++) {
+ int index = RAND.nextInt(numExistingKeys);
+ while (!existingKeys.containsKey(index)) {
index = (index + 1) % numExistingKeys;
- kp = existingKeys.get(index);
}
- existingKeys.remove(kp);
+ KeyPartition kp = existingKeys.remove(index);
+ existingKeys.put(index, existingKeys.get(numExistingKeys - 1));
+ existingKeys.remove(numExistingKeys - 1);
Review comment:
this line is solving the use case that you pointed out in jira @prashantwason . I guess the fix is complete and good to merge.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] bvaradar commented on issue #1395: [HUDI-667]
Fixing delete tests for DeltaStreamer
Posted by GitBox <gi...@apache.org>.
bvaradar commented on issue #1395: [HUDI-667] Fixing delete tests for DeltaStreamer
URL: https://github.com/apache/incubator-hudi/pull/1395#issuecomment-597434678
@lamber-ken : Can you review this PR.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] prashantwason commented on issue #1395: [HUDI-667]
Fixing delete tests for DeltaStreamer
Posted by GitBox <gi...@apache.org>.
prashantwason commented on issue #1395: [HUDI-667] Fixing delete tests for DeltaStreamer
URL: https://github.com/apache/incubator-hudi/pull/1395#issuecomment-598024231
I dont think the fix is complete. Please see my comments.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] bvaradar merged pull request #1395: [HUDI-667]
Fixing delete tests for DeltaStreamer
Posted by GitBox <gi...@apache.org>.
bvaradar merged pull request #1395: [HUDI-667] Fixing delete tests for DeltaStreamer
URL: https://github.com/apache/incubator-hudi/pull/1395
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] prashantwason commented on a change in pull
request #1395: [HUDI-667] Fixing delete tests for DeltaStreamer
Posted by GitBox <gi...@apache.org>.
prashantwason commented on a change in pull request #1395: [HUDI-667] Fixing delete tests for DeltaStreamer
URL: https://github.com/apache/incubator-hudi/pull/1395#discussion_r392403324
##########
File path: hudi-client/src/test/java/org/apache/hudi/common/HoodieTestDataGenerator.java
##########
@@ -429,24 +428,24 @@ public HoodieRecord generateUpdateRecord(HoodieKey key, String commitTime) throw
*/
public Stream<HoodieKey> generateUniqueDeleteStream(Integer n) {
final Set<KeyPartition> used = new HashSet<>();
-
if (n > numExistingKeys) {
throw new IllegalArgumentException("Requested unique deletes is greater than number of available keys");
}
- return IntStream.range(0, n).boxed().map(i -> {
- int index = numExistingKeys == 1 ? 0 : RAND.nextInt(numExistingKeys - 1);
- KeyPartition kp = existingKeys.get(index);
- // Find the available keyPartition starting from randomly chosen one.
- while (used.contains(kp)) {
+ List<HoodieKey> result = new ArrayList<>();
+ for (int i = 0; i < n; i++) {
+ int index = RAND.nextInt(numExistingKeys);
+ while (!existingKeys.containsKey(index)) {
index = (index + 1) % numExistingKeys;
- kp = existingKeys.get(index);
}
- existingKeys.remove(kp);
+ KeyPartition kp = existingKeys.remove(index);
+ existingKeys.put(index, existingKeys.get(numExistingKeys - 1));
+ existingKeys.remove(numExistingKeys - 1);
Review comment:
Looks good. I didnt read the fix correctly.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services