You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "rohityadav1993 (via GitHub)" <gi...@apache.org> on 2023/09/13 17:35:07 UTC
[GitHub] [pinot] rohityadav1993 opened a new pull request, #11584: custom partial upsert row merger
rohityadav1993 opened a new pull request, #11584:
URL: https://github.com/apache/pinot/pull/11584
Instructions:
1. The PR has to be tagged with at least one of the following labels (*):
1. `feature`
2. `bugfix`
3. `performance`
4. `ui`
5. `backward-incompat`
6. `release-notes` (**)
2. Remove these instructions before publishing the PR.
(*) Other labels to consider:
- `testing`
- `dependencies`
- `docker`
- `kubernetes`
- `observability`
- `security`
- `code-style`
- `extension-point`
- `refactor`
- `cleanup`
(**) Use `release-notes` label for scenarios like:
- New configuration options
- Deprecation of configurations
- Signature changes to public methods/interfaces
- New plugins added or old plugins removed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] deemoliu commented on a diff in pull request #11584: custom partial upsert row merger
Posted by "deemoliu (via GitHub)" <gi...@apache.org>.
deemoliu commented on code in PR #11584:
URL: https://github.com/apache/pinot/pull/11584#discussion_r1325212461
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/upsert/PartialUpsertHandler.java:
##########
@@ -41,16 +44,33 @@ public class PartialUpsertHandler {
private final PartialUpsertMerger _defaultPartialUpsertMerger;
private final List<String> _comparisonColumns;
private final List<String> _primaryKeyColumns;
+ private final PartialUpsertRowMergeEvaluator _rowMerger;
+ private LazyRow _reusePreviousLazyRow;
+ private LazyRow _reuseNewLazyRow;
+ private Map<String, Object> _reuseRowMergerResult;
- public PartialUpsertHandler(Schema schema, Map<String, UpsertConfig.Strategy> partialUpsertStrategies,
- UpsertConfig.Strategy defaultPartialUpsertStrategy, List<String> comparisonColumns) {
- _defaultPartialUpsertMerger = PartialUpsertMergerFactory.getMerger(defaultPartialUpsertStrategy);
+ public PartialUpsertHandler(Schema schema, UpsertConfig upsertConfig, List<String> comparisonColumns) {
+ _defaultPartialUpsertMerger = PartialUpsertMergerFactory.getMerger(upsertConfig.getDefaultPartialUpsertStrategy());
_comparisonColumns = comparisonColumns;
_primaryKeyColumns = schema.getPrimaryKeyColumns();
- for (Map.Entry<String, UpsertConfig.Strategy> entry : partialUpsertStrategies.entrySet()) {
+ for (Map.Entry<String, UpsertConfig.Strategy> entry : upsertConfig.getPartialUpsertStrategies().entrySet()) {
_column2Mergers.put(entry.getKey(), PartialUpsertMergerFactory.getMerger(entry.getValue()));
}
+
+ String rowMergerCustomImplementation = upsertConfig.getRowMergerCustomImplementation();
+ if (rowMergerCustomImplementation != null && !rowMergerCustomImplementation.equals("")) {
+ try {
+ _rowMerger = PartialUpsertRowMergeEvaluatorFactory.getInstance(rowMergerCustomImplementation);
+ } catch (Exception e) {
+ throw new RuntimeException(e);
+ }
+ _reusePreviousLazyRow = new LazyRow();
+ _reuseNewLazyRow = new LazyRow();
+ _reuseRowMergerResult = new HashMap<>();
Review Comment:
concurrent hashmap?
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/readers/LazyRow.java:
##########
@@ -0,0 +1,94 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.readers;
+
+import java.io.IOException;
+import java.util.HashMap;
+import org.apache.pinot.segment.spi.IndexSegment;
+import org.apache.pinot.spi.data.readers.GenericRow;
+
+
+/**
+ * A wrapper class to read column values of a row. The wrapper can either be over a {@link IndexSegment} and docId or
+ * over a {@link GenericRow}<br>
+ * The advantage of having wrapper over segment and docId is column values are read only when
+ * {@link LazyRow#getValue(String)} is invoked.
+ * This is useful to reduce the disk reads incurred due to loading the previous row during merge step.
+ * There isn't any advantage to have a LazyRow wrap a GenericRow but has been kept for syntactic sugar.
+ */
+public class LazyRow {
+ private IndexSegment _segment;
+ private int _docId;
+ private GenericRow _row;
+
+ private HashMap<String, Object> _fieldToValueMap = new HashMap<>();
Review Comment:
concurrentHashMap ?
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/upsert/merger/PartialUpsertRowMergeEvaluatorFactory.java:
##########
@@ -0,0 +1,50 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.upsert.merger;
+
+import org.apache.commons.lang3.StringUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+public class PartialUpsertRowMergeEvaluatorFactory {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(PartialUpsertRowMergeEvaluatorFactory.class);
+
+ private PartialUpsertRowMergeEvaluatorFactory() {
+ }
+
+ public static PartialUpsertRowMergeEvaluator getInstance(String implementationClassName) {
+ LOGGER.info("Creating BiFunctionEvaluator for class {}", implementationClassName);
+ if (StringUtils.isBlank(implementationClassName)) {
+ throw new IllegalArgumentException("Empty implementationClassName");
+ }
+ try {
+ Class<?> aClass = Class.forName(implementationClassName);
+ if (!PartialUpsertRowMergeEvaluator.class.isAssignableFrom(aClass)) {
+ throw new IllegalArgumentException(
+ "The provided class is not an implementation of BiFunctionEvaluator");
Review Comment:
BiFunctionEvaluator --> PartialUpsertRowMergeEvaluator
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/upsert/BaseTableUpsertMetadataManager.java:
##########
@@ -145,6 +137,32 @@ public void init(TableConfig tableConfig, Schema schema, TableDataManager tableD
}
}
+ /**
+ * Reuses a PartialUpsertHandler if there is no custom row merger otherwise creates a new instance to avoid
+ * concurrent modification of state(LazyRow) between segments of the table
+ * @return partial upsert handler to merge previous and new row
+ */
+ protected PartialUpsertHandler getPartialUpsertHandler() {
+ UpsertConfig upsertConfig = _tableConfig.getUpsertConfig();
+ if (upsertConfig.getMode() == UpsertConfig.Mode.PARTIAL) {
+ Map<String, UpsertConfig.Strategy> partialUpsertStrategies = upsertConfig.getPartialUpsertStrategies();
+ Preconditions.checkArgument(partialUpsertStrategies != null,
+ "Partial-upsert strategies must be configured for partial-upsert enabled table: %s",
+ _tableNameWithType);
+ if (upsertConfig.getRowMergerCustomImplementation() != null) {
+ // In case of custom merger, the PUH is dependent on LazyRow object to be reused. LazyRow is stateful and
+ // cause concurrent modification issues, hence a new PUH is created per partition
+ return new PartialUpsertHandler(_schema, upsertConfig, _comparisonColumns);
Review Comment:
Got it. currently the partial upsert handler are initialized in the upsertTableManager and shared by partitionUpsertManager. After lazyRow passed in the partialUpsertHandler it become stateful, and multiple partitions can access the partial upsert handler. this leads to concurrent modification issue. So we have to avoid sharing the partial upsert handler among partitions.
@Jackie-Jiang do you see any improvement on this approach?
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/upsert/merger/PartialUpsertRowMergeEvaluatorFactory.java:
##########
@@ -0,0 +1,50 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.upsert.merger;
+
+import org.apache.commons.lang3.StringUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+public class PartialUpsertRowMergeEvaluatorFactory {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(PartialUpsertRowMergeEvaluatorFactory.class);
+
+ private PartialUpsertRowMergeEvaluatorFactory() {
+ }
+
+ public static PartialUpsertRowMergeEvaluator getInstance(String implementationClassName) {
+ LOGGER.info("Creating BiFunctionEvaluator for class {}", implementationClassName);
Review Comment:
BiFunctionEvaluator --> PartialUpsertRowMergeEvaluator
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] rohityadav1993 commented on a diff in pull request #11584: custom partial upsert row merger
Posted by "rohityadav1993 (via GitHub)" <gi...@apache.org>.
rohityadav1993 commented on code in PR #11584:
URL: https://github.com/apache/pinot/pull/11584#discussion_r1330441934
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/readers/LazyRow.java:
##########
@@ -0,0 +1,94 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.readers;
+
+import java.io.IOException;
+import java.util.HashMap;
+import org.apache.pinot.segment.spi.IndexSegment;
+import org.apache.pinot.spi.data.readers.GenericRow;
+
+
+/**
+ * A wrapper class to read column values of a row. The wrapper can either be over a {@link IndexSegment} and docId or
+ * over a {@link GenericRow}<br>
+ * The advantage of having wrapper over segment and docId is column values are read only when
+ * {@link LazyRow#getValue(String)} is invoked.
+ * This is useful to reduce the disk reads incurred due to loading the previous row during merge step.
+ * There isn't any advantage to have a LazyRow wrap a GenericRow but has been kept for syntactic sugar.
+ */
+public class LazyRow {
+ private IndexSegment _segment;
+ private int _docId;
+ private GenericRow _row;
+
+ private HashMap<String, Object> _fieldToValueMap = new HashMap<>();
Review Comment:
At least in the current usage, I don't see a requirement to have a ConcurrentHashMap as the merger happens per row per segment in a single thread manner. Do you see any usage where we want to use LazyRow and there can be multiple threads referencing it?
I am taking reference from GenericRow which also uses hashmap.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] rohityadav1993 commented on a diff in pull request #11584: custom partial upsert row merger
Posted by "rohityadav1993 (via GitHub)" <gi...@apache.org>.
rohityadav1993 commented on code in PR #11584:
URL: https://github.com/apache/pinot/pull/11584#discussion_r1330441934
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/readers/LazyRow.java:
##########
@@ -0,0 +1,94 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.readers;
+
+import java.io.IOException;
+import java.util.HashMap;
+import org.apache.pinot.segment.spi.IndexSegment;
+import org.apache.pinot.spi.data.readers.GenericRow;
+
+
+/**
+ * A wrapper class to read column values of a row. The wrapper can either be over a {@link IndexSegment} and docId or
+ * over a {@link GenericRow}<br>
+ * The advantage of having wrapper over segment and docId is column values are read only when
+ * {@link LazyRow#getValue(String)} is invoked.
+ * This is useful to reduce the disk reads incurred due to loading the previous row during merge step.
+ * There isn't any advantage to have a LazyRow wrap a GenericRow but has been kept for syntactic sugar.
+ */
+public class LazyRow {
+ private IndexSegment _segment;
+ private int _docId;
+ private GenericRow _row;
+
+ private HashMap<String, Object> _fieldToValueMap = new HashMap<>();
Review Comment:
At least in the current usage, I don't see a requirement to have a ConcurrentHashMap as the merger happens per row per segment in a single thread manner. Do you see any usage where we want to use LazyRow and there can be multiple threads referencing it?
I am taking reference from GenericRow which also using hashmap.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] custom partial upsert row merger [pinot]
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on PR #11584:
URL: https://github.com/apache/pinot/pull/11584#issuecomment-1758692248
My suggestion is to not add a new interface, but modify the existing `PartialUpsertMerger` interface
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] [DRAFT] custom partial upsert row merger [pinot]
Posted by "rohityadav1993 (via GitHub)" <gi...@apache.org>.
rohityadav1993 closed pull request #11584: [DRAFT] custom partial upsert row merger
URL: https://github.com/apache/pinot/pull/11584
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] custom partial upsert row merger [pinot]
Posted by "rohityadav1993 (via GitHub)" <gi...@apache.org>.
rohityadav1993 commented on code in PR #11584:
URL: https://github.com/apache/pinot/pull/11584#discussion_r1343656583
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/upsert/PartialUpsertHandler.java:
##########
@@ -41,16 +44,33 @@ public class PartialUpsertHandler {
private final PartialUpsertMerger _defaultPartialUpsertMerger;
private final List<String> _comparisonColumns;
private final List<String> _primaryKeyColumns;
+ private final PartialUpsertRowMergeEvaluator _rowMerger;
+ private LazyRow _reusePreviousLazyRow;
+ private LazyRow _reuseNewLazyRow;
+ private Map<String, Object> _reuseRowMergerResult;
- public PartialUpsertHandler(Schema schema, Map<String, UpsertConfig.Strategy> partialUpsertStrategies,
- UpsertConfig.Strategy defaultPartialUpsertStrategy, List<String> comparisonColumns) {
- _defaultPartialUpsertMerger = PartialUpsertMergerFactory.getMerger(defaultPartialUpsertStrategy);
+ public PartialUpsertHandler(Schema schema, UpsertConfig upsertConfig, List<String> comparisonColumns) {
+ _defaultPartialUpsertMerger = PartialUpsertMergerFactory.getMerger(upsertConfig.getDefaultPartialUpsertStrategy());
_comparisonColumns = comparisonColumns;
_primaryKeyColumns = schema.getPrimaryKeyColumns();
- for (Map.Entry<String, UpsertConfig.Strategy> entry : partialUpsertStrategies.entrySet()) {
+ for (Map.Entry<String, UpsertConfig.Strategy> entry : upsertConfig.getPartialUpsertStrategies().entrySet()) {
_column2Mergers.put(entry.getKey(), PartialUpsertMergerFactory.getMerger(entry.getValue()));
}
+
+ String rowMergerCustomImplementation = upsertConfig.getRowMergerCustomImplementation();
+ if (rowMergerCustomImplementation != null && !rowMergerCustomImplementation.equals("")) {
+ try {
+ _rowMerger = PartialUpsertRowMergeEvaluatorFactory.getInstance(rowMergerCustomImplementation);
+ } catch (Exception e) {
+ throw new RuntimeException(e);
+ }
+ _reusePreviousLazyRow = new LazyRow();
+ _reuseNewLazyRow = new LazyRow();
+ _reuseRowMergerResult = new HashMap<>();
Review Comment:
Should not be needed since this is per partition.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] custom partial upsert row merger [pinot]
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on PR #11584:
URL: https://github.com/apache/pinot/pull/11584#issuecomment-1755887036
IIUC, `LazyRow` is introduced to avoid reading the same value from the segment multiple times. I think it is a good abstraction (suggestion making it a wrapper over `IndexSegment` and allow setting a `docId` so that the internal `fieldToValueMap` can be reused, no need to allow initializing it with `GenericRow`), and we can make `PartialUpsertMerger` merge API: `void merge(LazyRow previousRow, GenericRow currentRow, Map<String, Object> mergedValues)`. This way we can even support generating multiple values in one merger
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] custom partial upsert row merger [pinot]
Posted by "rohityadav1993 (via GitHub)" <gi...@apache.org>.
rohityadav1993 commented on PR #11584:
URL: https://github.com/apache/pinot/pull/11584#issuecomment-1755999768
> IIUC, `LazyRow` is introduced to avoid reading the same value from the segment multiple times. I think it is a good abstraction (suggestion making it a wrapper over `IndexSegment` and allow setting a `docId` so that the internal `fieldToValueMap` can be reused, no need to allow initializing it with `GenericRow`), and we can make `PartialUpsertMerger` merge API: `void merge(LazyRow previousRow, GenericRow currentRow, Map<String, Object> mergedValues)`. This way we can even support generating multiple values in one merger
It is also used for defining new interface:
```
public interface PartialUpsertRowMergeEvaluator {
void evaluate(LazyRow previousRow, LazyRow newRow, Map<String, Object> result);
}
```
which represents the row merge logic, which requires LazyRow to wrap GenericRecord.
So this contract should change as well similar to `void merge(LazyRow previousRow, GenericRow currentRow, Map<String, Object> mergedValues)`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] custom partial upsert row merger [pinot]
Posted by "rohityadav1993 (via GitHub)" <gi...@apache.org>.
rohityadav1993 commented on PR #11584:
URL: https://github.com/apache/pinot/pull/11584#issuecomment-1752604357
Thanks for the feedback @Jackie-Jiang
> We should not mix value based merger and row based merger in the same class because it will be very hard to maintain, and the sequence of applying the mergers can be very confusing.
I agree with the maintenance perspective. There could be a requirement to have both row merger and column level merger for values which were not computed in row merger. Hence was the motivation to keep them in the same class.
> I'm leaning towards the second approach because that allows us to reuse the existing mergers.
This is a cleaner approach if we can stick to a good contract for merge. from the design doc [Approach 2](https://docs.google.com/document/d/1bBTCYZFP2stvzc6xZUOEh-XweVgC9WfD7uiSPbKtaZY/edit#heading=h.qkglmejmd9qa), if we can define merge as `merge(GenericRow previousRecord, GenericRow newRecord, Map<String, Object> mergerResult)` and pass an instance of `LazyRow extends GenericRow`
> Since each column can reference the value from other columns, the contract should be to not change the column value before all the mergers are applied
`mergerResult` will be used to store the intermediate merger results and avoid modification to newRecord until all mergers are applied. We can ensure that results are applied after column level mergers are executed.
If we don't want to have `Map<String, Object> mergerResult` in merge method then we have to initialise PartialUpsertHandler at `PartitionUpsertMetadataManager` instead of `TableUpsertMetadataManager` to avoid concurrent modification.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] custom partial upsert row merger [pinot]
Posted by "rohityadav1993 (via GitHub)" <gi...@apache.org>.
rohityadav1993 commented on code in PR #11584:
URL: https://github.com/apache/pinot/pull/11584#discussion_r1343655671
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/upsert/merger/PartialUpsertRowMergeEvaluatorFactory.java:
##########
@@ -0,0 +1,50 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.upsert.merger;
+
+import org.apache.commons.lang3.StringUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+public class PartialUpsertRowMergeEvaluatorFactory {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(PartialUpsertRowMergeEvaluatorFactory.class);
+
+ private PartialUpsertRowMergeEvaluatorFactory() {
+ }
+
+ public static PartialUpsertRowMergeEvaluator getInstance(String implementationClassName) {
+ LOGGER.info("Creating BiFunctionEvaluator for class {}", implementationClassName);
+ if (StringUtils.isBlank(implementationClassName)) {
+ throw new IllegalArgumentException("Empty implementationClassName");
+ }
+ try {
+ Class<?> aClass = Class.forName(implementationClassName);
+ if (!PartialUpsertRowMergeEvaluator.class.isAssignableFrom(aClass)) {
+ throw new IllegalArgumentException(
+ "The provided class is not an implementation of BiFunctionEvaluator");
Review Comment:
Done
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/upsert/merger/PartialUpsertRowMergeEvaluatorFactory.java:
##########
@@ -0,0 +1,50 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.upsert.merger;
+
+import org.apache.commons.lang3.StringUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+public class PartialUpsertRowMergeEvaluatorFactory {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(PartialUpsertRowMergeEvaluatorFactory.class);
+
+ private PartialUpsertRowMergeEvaluatorFactory() {
+ }
+
+ public static PartialUpsertRowMergeEvaluator getInstance(String implementationClassName) {
+ LOGGER.info("Creating BiFunctionEvaluator for class {}", implementationClassName);
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] custom partial upsert row merger [pinot]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #11584:
URL: https://github.com/apache/pinot/pull/11584#issuecomment-1749725725
## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/11584?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
> Merging [#11584](https://app.codecov.io/gh/apache/pinot/pull/11584?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (8951cc9) into [master](https://app.codecov.io/gh/apache/pinot/commit/d1021dfd60911c57fe124e6882ebcd51bcc012ed?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (d1021df) will **decrease** coverage by `48.68%`.
> Report is 1 commits behind head on master.
> The diff coverage is `0.00%`.
```diff
@@ Coverage Diff @@
## master #11584 +/- ##
=============================================
- Coverage 63.09% 14.42% -48.68%
+ Complexity 1117 201 -916
=============================================
Files 2342 2344 +2
Lines 125900 125986 +86
Branches 19362 19375 +13
=============================================
- Hits 79437 18170 -61267
- Misses 40817 106281 +65464
+ Partials 5646 1535 -4111
```
| [Flag](https://app.codecov.io/gh/apache/pinot/pull/11584/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [integration](https://app.codecov.io/gh/apache/pinot/pull/11584/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [integration1](https://app.codecov.io/gh/apache/pinot/pull/11584/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [integration2](https://app.codecov.io/gh/apache/pinot/pull/11584/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [java-11](https://app.codecov.io/gh/apache/pinot/pull/11584/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `14.41% <0.00%> (-35.73%)` | :arrow_down: |
| [java-17](https://app.codecov.io/gh/apache/pinot/pull/11584/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `14.41% <0.00%> (-48.55%)` | :arrow_down: |
| [java-20](https://app.codecov.io/gh/apache/pinot/pull/11584/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [temurin](https://app.codecov.io/gh/apache/pinot/pull/11584/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `14.42% <0.00%> (-48.68%)` | :arrow_down: |
| [unittests](https://app.codecov.io/gh/apache/pinot/pull/11584/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `14.42% <0.00%> (-48.67%)` | :arrow_down: |
| [unittests1](https://app.codecov.io/gh/apache/pinot/pull/11584/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [unittests2](https://app.codecov.io/gh/apache/pinot/pull/11584/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `14.42% <0.00%> (+<0.01%)` | :arrow_up: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files](https://app.codecov.io/gh/apache/pinot/pull/11584?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...psert/ConcurrentMapTableUpsertMetadataManager.java](https://app.codecov.io/gh/apache/pinot/pull/11584?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC91cHNlcnQvQ29uY3VycmVudE1hcFRhYmxlVXBzZXJ0TWV0YWRhdGFNYW5hZ2VyLmphdmE=) | `0.00% <0.00%> (-33.34%)` | :arrow_down: |
| [...rg/apache/pinot/spi/config/table/UpsertConfig.java](https://app.codecov.io/gh/apache/pinot/pull/11584?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3Qtc3BpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcGkvY29uZmlnL3RhYmxlL1Vwc2VydENvbmZpZy5qYXZh) | `0.00% <0.00%> (-85.72%)` | :arrow_down: |
| [.../merger/PartialUpsertRowMergeEvaluatorFactory.java](https://app.codecov.io/gh/apache/pinot/pull/11584?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC91cHNlcnQvbWVyZ2VyL1BhcnRpYWxVcHNlcnRSb3dNZXJnZUV2YWx1YXRvckZhY3RvcnkuamF2YQ==) | `0.00% <0.00%> (ø)` | |
| [...t/local/upsert/BaseTableUpsertMetadataManager.java](https://app.codecov.io/gh/apache/pinot/pull/11584?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC91cHNlcnQvQmFzZVRhYmxlVXBzZXJ0TWV0YWRhdGFNYW5hZ2VyLmphdmE=) | `0.00% <0.00%> (-76.54%)` | :arrow_down: |
| [...not/segment/local/upsert/PartialUpsertHandler.java](https://app.codecov.io/gh/apache/pinot/pull/11584?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC91cHNlcnQvUGFydGlhbFVwc2VydEhhbmRsZXIuamF2YQ==) | `0.00% <0.00%> (-83.79%)` | :arrow_down: |
| [...e/pinot/segment/local/segment/readers/LazyRow.java](https://app.codecov.io/gh/apache/pinot/pull/11584?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L3JlYWRlcnMvTGF6eVJvdy5qYXZh) | `0.00% <0.00%> (ø)` | |
... and [1509 files with indirect coverage changes](https://app.codecov.io/gh/apache/pinot/pull/11584/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
:mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] custom partial upsert row merger [pinot]
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on code in PR #11584:
URL: https://github.com/apache/pinot/pull/11584#discussion_r1349353163
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/readers/LazyRow.java:
##########
@@ -0,0 +1,94 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.readers;
+
+import java.io.IOException;
+import java.util.HashMap;
+import org.apache.pinot.segment.spi.IndexSegment;
+import org.apache.pinot.spi.data.readers.GenericRow;
+
+
+/**
+ * A wrapper class to read column values of a row. The wrapper can either be over a {@link IndexSegment} and docId or
+ * over a {@link GenericRow}<br>
+ * The advantage of having wrapper over segment and docId is column values are read only when
+ * {@link LazyRow#getValue(String)} is invoked.
+ * This is useful to reduce the disk reads incurred due to loading the previous row during merge step.
+ * There isn't any advantage to have a LazyRow wrap a GenericRow but has been kept for syntactic sugar.
+ */
+public class LazyRow {
+ private IndexSegment _segment;
Review Comment:
(code format) Please follow [Pinot Style](https://docs.pinot.apache.org/developers/developers-and-contributors/code-setup#set-up-ide)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org