You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "LsomeYeah (via GitHub)" <gi...@apache.org> on 2023/06/21 03:40:32 UTC
[GitHub] [incubator-paimon] LsomeYeah opened a new pull request, #1416: [core] implement adjustProjection in PartialUpdateMergeFunction
LsomeYeah opened a new pull request, #1416:
URL: https://github.com/apache/incubator-paimon/pull/1416
<!-- Please specify the module before the PR name: [core] ... or [flink] ... -->
### Purpose
When user provides a projection, we should expand the projection with the necessary sequence fields.
And the FieldSequenceGroup should be adjusted if the projection is applied.
<!-- Linking this pull request to the issue -->
Linked issue: close #xxx
<!-- What is the purpose of the change -->
### Tests
<!-- List UT and IT cases to verify this change -->
UT case : add a test `testAdjustProjection` in `PartialUpdateMergeFunctionTest`.
IT case : enrich the content of `testSequenceGroup` in `PartialUpdateITCase`
### API and Format
<!-- Does this change affect API or storage format -->
### Documentation
<!-- Does this change introduce a new feature -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-paimon] JingsongLi commented on a diff in pull request #1416: [core] implement adjustProjection in PartialUpdateMergeFunction
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #1416:
URL: https://github.com/apache/incubator-paimon/pull/1416#discussion_r1236663101
##########
paimon-core/src/main/java/org/apache/paimon/mergetree/compact/PartialUpdateMergeFunction.java:
##########
@@ -190,11 +195,36 @@ private Factory(Options options, RowType rowType) {
@Override
public MergeFunction<KeyValue> create(@Nullable int[][] projection) {
List<DataType> fieldTypes = tableTypes;
+ Map<Integer, SequenceGenerator> projectedFieldSequences = new HashMap<>();
+
if (projection != null) {
fieldTypes = Projection.of(projection).project(tableTypes);
+ int[] projects = Projection.of(projection).toTopLevelIndexes();
+ Map<Integer, Integer> indexMap = new HashMap<>();
+ for (int i = 0; i < projects.length; i++) {
+ indexMap.put(projects[i], i);
+ }
+ fieldSequences.forEach(
+ (field, sequence) -> {
+ int newField = indexMap.getOrDefault(field, -1);
+ if (newField != -1) {
+ int newSequenceId = indexMap.getOrDefault(sequence.index(), -1);
+ if (newSequenceId != -1) {
Review Comment:
newSequenceId == -1 should throws a exception, you should add test for this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-paimon] JingsongLi commented on a diff in pull request #1416: [core] implement adjustProjection in PartialUpdateMergeFunction
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #1416:
URL: https://github.com/apache/incubator-paimon/pull/1416#discussion_r1236666639
##########
paimon-core/src/main/java/org/apache/paimon/mergetree/compact/PartialUpdateMergeFunction.java:
##########
@@ -190,11 +195,36 @@ private Factory(Options options, RowType rowType) {
@Override
public MergeFunction<KeyValue> create(@Nullable int[][] projection) {
List<DataType> fieldTypes = tableTypes;
+ Map<Integer, SequenceGenerator> projectedFieldSequences = new HashMap<>();
+
if (projection != null) {
fieldTypes = Projection.of(projection).project(tableTypes);
+ int[] projects = Projection.of(projection).toTopLevelIndexes();
+ Map<Integer, Integer> indexMap = new HashMap<>();
+ for (int i = 0; i < projects.length; i++) {
+ indexMap.put(projects[i], i);
+ }
+ fieldSequences.forEach(
+ (field, sequence) -> {
+ int newField = indexMap.getOrDefault(field, -1);
+ if (newField != -1) {
+ int newSequenceId = indexMap.getOrDefault(sequence.index(), -1);
+ if (newSequenceId != -1) {
+ projectedFieldSequences.put(
+ newField,
+ new SequenceGenerator(
+ newSequenceId, sequence.fieldType()));
+ }
+ }
+ });
Review Comment:
`return new PartialUpdateMergeFunction(createFieldGetters(fieldTypes), ignoreDelete, projectedFieldSequences);`
here.
##########
paimon-core/src/main/java/org/apache/paimon/mergetree/compact/PartialUpdateMergeFunction.java:
##########
@@ -190,11 +195,36 @@ private Factory(Options options, RowType rowType) {
@Override
public MergeFunction<KeyValue> create(@Nullable int[][] projection) {
List<DataType> fieldTypes = tableTypes;
+ Map<Integer, SequenceGenerator> projectedFieldSequences = new HashMap<>();
+
if (projection != null) {
Review Comment:
```
if (project != null) {
List<DataType> fieldTypes = tableTypes;
Map<Integer, SequenceGenerator> projectedFieldSequences = new HashMap<>();
.....
return new PartialUpdateMergeFunction(createFieldGetters(fieldTypes), ignoreDelete, projectedFieldSequences);
} else {
return new PartialUpdateMergeFunction(createFieldGetters(tableTypes), ignoreDelete, fieldSequences);
}
```
here.
##########
paimon-core/src/test/java/org/apache/paimon/mergetree/compact/PartialUpdateMergeFunctionTest.java:
##########
@@ -113,6 +310,46 @@ private void add(
GenericRow.of(f0, f1, f2, f3, f4, f5, f6)));
}
+ private void add(
+ MergeFunction<KeyValue> function,
+ Integer f0,
Review Comment:
Object... values
##########
paimon-core/src/test/java/org/apache/paimon/mergetree/compact/PartialUpdateMergeFunctionTest.java:
##########
@@ -113,6 +310,46 @@ private void add(
GenericRow.of(f0, f1, f2, f3, f4, f5, f6)));
}
+ private void add(
+ MergeFunction<KeyValue> function,
+ Integer f0,
+ Integer f1,
+ Integer f2,
+ Integer f3,
+ Integer f4,
+ Integer f5,
+ Integer f6,
+ Integer f7) {
+ function.add(
+ new KeyValue()
+ .replace(
+ GenericRow.of(1),
+ sequence++,
+ RowKind.INSERT,
+ GenericRow.of(f0, f1, f2, f3, f4, f5, f6, f7)));
+ }
+
+ private void validate(
+ MergeFunction<KeyValue> function,
+ Integer f0,
Review Comment:
Object... values
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-paimon] JingsongLi merged pull request #1416: [core] implement adjustProjection in PartialUpdateMergeFunction
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #1416:
URL: https://github.com/apache/incubator-paimon/pull/1416
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-paimon] LsomeYeah commented on a diff in pull request #1416: [core] implement adjustProjection in PartialUpdateMergeFunction
Posted by "LsomeYeah (via GitHub)" <gi...@apache.org>.
LsomeYeah commented on code in PR #1416:
URL: https://github.com/apache/incubator-paimon/pull/1416#discussion_r1236380077
##########
paimon-core/src/main/java/org/apache/paimon/mergetree/compact/PartialUpdateMergeFunction.java:
##########
@@ -203,7 +231,35 @@ public AdjustedProjection adjustProjection(@Nullable int[][] projection) {
if (fieldSequences.isEmpty()) {
return new AdjustedProjection(projection, null);
}
- // TODO implement this, just keep required fields and adjust fieldSequences too.
+
+ if (projection != null) {
+ int[] projIndexs = Projection.of(projection).toTopLevelIndexes();
Review Comment:
There may be some sequence fields in `topProjects`, we don't need to add these fields to `extraFields`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-paimon] JingsongLi commented on a diff in pull request #1416: [core] implement adjustProjection in PartialUpdateMergeFunction
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #1416:
URL: https://github.com/apache/incubator-paimon/pull/1416#discussion_r1236287176
##########
paimon-core/src/main/java/org/apache/paimon/mergetree/compact/PartialUpdateMergeFunction.java:
##########
@@ -203,7 +231,35 @@ public AdjustedProjection adjustProjection(@Nullable int[][] projection) {
if (fieldSequences.isEmpty()) {
return new AdjustedProjection(projection, null);
}
- // TODO implement this, just keep required fields and adjust fieldSequences too.
+
+ if (projection != null) {
+ int[] projIndexs = Projection.of(projection).toTopLevelIndexes();
Review Comment:
```
if (projection == null) {
return new AdjustedProjection(null, null);
}
LinkedHashSet<Integer> extraFields = new LinkedHashSet<>();
int[] topProjects = Projection.of(projection).toTopLevelIndexes();
for (int index : topProjects) {
SequenceGenerator generator = fieldSequences.get(index);
if (generator != null) {
extraFields.add(generator.index());
}
}
int[] allProjects = Stream.concat(
Arrays.stream(topProjects).boxed(),
extraFields.stream()).mapToInt(Integer::intValue).toArray();
int[][] pushdown = Projection.of(allProjects).toNestedIndexes();
int[][] outer = Projection.of(IntStream.range(0, topProjects.length).toArray()).toNestedIndexes();
return new AdjustedProjection(pushdown, outer);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-paimon] JingsongLi commented on a diff in pull request #1416: [core] implement adjustProjection in PartialUpdateMergeFunction
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #1416:
URL: https://github.com/apache/incubator-paimon/pull/1416#discussion_r1236288694
##########
paimon-core/src/main/java/org/apache/paimon/mergetree/compact/PartialUpdateMergeFunction.java:
##########
@@ -190,11 +193,36 @@ private Factory(Options options, RowType rowType) {
@Override
public MergeFunction<KeyValue> create(@Nullable int[][] projection) {
List<DataType> fieldTypes = tableTypes;
+ Map<Integer, SequenceGenerator> projFieldSequences = new HashMap<>();
Review Comment:
naming: projectedFieldSequences
##########
paimon-core/src/test/java/org/apache/paimon/mergetree/compact/PartialUpdateMergeFunctionTest.java:
##########
@@ -95,6 +95,38 @@ public void testSequenceGroupRepeatDefine() {
.hasMessageContaining("is defined repeatedly by multiple groups");
}
+ @Test
+ public void testAdjustProjection() {
Review Comment:
This test is too simple, you should cover more.
##########
paimon-core/src/main/java/org/apache/paimon/mergetree/compact/PartialUpdateMergeFunction.java:
##########
@@ -190,11 +193,36 @@ private Factory(Options options, RowType rowType) {
@Override
public MergeFunction<KeyValue> create(@Nullable int[][] projection) {
List<DataType> fieldTypes = tableTypes;
+ Map<Integer, SequenceGenerator> projFieldSequences = new HashMap<>();
+
if (projection != null) {
fieldTypes = Projection.of(projection).project(tableTypes);
+ int[] fieldOldIndex = Projection.of(projection).toTopLevelIndexes();
Review Comment:
naming: fieldOldIndex -> projects
##########
paimon-core/src/main/java/org/apache/paimon/mergetree/compact/PartialUpdateMergeFunction.java:
##########
@@ -190,11 +193,36 @@ private Factory(Options options, RowType rowType) {
@Override
public MergeFunction<KeyValue> create(@Nullable int[][] projection) {
List<DataType> fieldTypes = tableTypes;
+ Map<Integer, SequenceGenerator> projFieldSequences = new HashMap<>();
+
if (projection != null) {
fieldTypes = Projection.of(projection).project(tableTypes);
+ int[] fieldOldIndex = Projection.of(projection).toTopLevelIndexes();
+ Map<Integer, Integer> indexMap = new HashMap<>();
+ for (int i = 0; i < fieldOldIndex.length; i++) {
+ indexMap.put(fieldOldIndex[i], i);
+ }
+ for (int index = 0; index < fieldOldIndex.length; index++) {
+ int oldIndex = fieldOldIndex[index];
+ if (fieldSequences.get(oldIndex) != null) {
+ int oldKeyIndex = fieldSequences.get(oldIndex).index();
+ int keyIndex = indexMap.get(oldKeyIndex);
+ if (!projFieldSequences.containsKey(keyIndex)) {
+ SequenceGenerator newSequenceGen =
+ new SequenceGenerator(keyIndex, fieldTypes.get(keyIndex));
Review Comment:
keyIndex is not new index of oldIndex.
There should be a bug! You should add tests to cover.
##########
paimon-core/src/main/java/org/apache/paimon/mergetree/compact/PartialUpdateMergeFunction.java:
##########
@@ -190,11 +193,36 @@ private Factory(Options options, RowType rowType) {
@Override
public MergeFunction<KeyValue> create(@Nullable int[][] projection) {
List<DataType> fieldTypes = tableTypes;
+ Map<Integer, SequenceGenerator> projFieldSequences = new HashMap<>();
+
if (projection != null) {
fieldTypes = Projection.of(projection).project(tableTypes);
+ int[] fieldOldIndex = Projection.of(projection).toTopLevelIndexes();
+ Map<Integer, Integer> indexMap = new HashMap<>();
+ for (int i = 0; i < fieldOldIndex.length; i++) {
+ indexMap.put(fieldOldIndex[i], i);
+ }
+ for (int index = 0; index < fieldOldIndex.length; index++) {
Review Comment:
loop `fieldSequences` like this:
```
fieldSequences.forEach((field, sequence) -> {
int newField = indexMap.get(field);
checkArguement(newField != -1);
int newSequenceId = indexMap.get(sequence.index());
projectedFieldSequences.put(newField, new SequenceGenerator(newSequenceId, ...));
});
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-paimon] JingsongLi commented on a diff in pull request #1416: [core] implement adjustProjection in PartialUpdateMergeFunction
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #1416:
URL: https://github.com/apache/incubator-paimon/pull/1416#discussion_r1236290098
##########
paimon-core/src/main/java/org/apache/paimon/mergetree/compact/PartialUpdateMergeFunction.java:
##########
@@ -190,11 +193,36 @@ private Factory(Options options, RowType rowType) {
@Override
public MergeFunction<KeyValue> create(@Nullable int[][] projection) {
List<DataType> fieldTypes = tableTypes;
+ Map<Integer, SequenceGenerator> projFieldSequences = new HashMap<>();
+
if (projection != null) {
fieldTypes = Projection.of(projection).project(tableTypes);
+ int[] fieldOldIndex = Projection.of(projection).toTopLevelIndexes();
+ Map<Integer, Integer> indexMap = new HashMap<>();
+ for (int i = 0; i < fieldOldIndex.length; i++) {
+ indexMap.put(fieldOldIndex[i], i);
+ }
+ for (int index = 0; index < fieldOldIndex.length; index++) {
Review Comment:
loop `fieldSequences` like this:
```
fieldSequences.forEach((field, sequence) -> {
int newField = indexMap.get(field);
if (newField != -1) {
int newSequenceId = indexMap.get(sequence.index());
checkArguement(newSequenceId != -1);
projectedFieldSequences.put(newField, new SequenceGenerator(newSequenceId, ...));
}
});
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org