You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2021/03/08 22:53:07 UTC
[GitHub] [incubator-pinot] daniellavoie opened a new pull request #6658: Add context on data type transform failure
daniellavoie opened a new pull request #6658:
URL: https://github.com/apache/incubator-pinot/pull/6658
## Description
Bubbles up the column name on failure when generating segments with invalid schema.
Fixes #6656
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] codecov-io commented on pull request #6658: Add context on data type transform failure
Posted by GitBox <gi...@apache.org>.
codecov-io commented on pull request #6658:
URL: https://github.com/apache/incubator-pinot/pull/6658#issuecomment-793187627
# [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/6658?src=pr&el=h1) Report
> Merging [#6658](https://codecov.io/gh/apache/incubator-pinot/pull/6658?src=pr&el=desc) (7df578c) into [master](https://codecov.io/gh/apache/incubator-pinot/commit/1beaab59b73f26c4e35f3b9bc856b03806cddf5a?el=desc) (1beaab5) will **increase** coverage by `7.41%`.
> The diff coverage is `80.36%`.
[](https://codecov.io/gh/apache/incubator-pinot/pull/6658?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #6658 +/- ##
==========================================
+ Coverage 66.44% 73.86% +7.41%
==========================================
Files 1075 1361 +286
Lines 54773 66520 +11747
Branches 8168 9702 +1534
==========================================
+ Hits 36396 49135 +12739
+ Misses 15700 14180 -1520
- Partials 2677 3205 +528
```
| Flag | Coverage Δ | |
|---|---|---|
| integration | `44.20% <42.81%> (?)` | |
| unittests | `65.86% <62.61%> (?)` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/incubator-pinot/pull/6658?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [...ot/broker/broker/AllowAllAccessControlFactory.java](https://codecov.io/gh/apache/incubator-pinot/pull/6658/diff?src=pr&el=tree#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvYnJva2VyL0FsbG93QWxsQWNjZXNzQ29udHJvbEZhY3RvcnkuamF2YQ==) | `100.00% <ø> (ø)` | |
| [.../helix/BrokerUserDefinedMessageHandlerFactory.java](https://codecov.io/gh/apache/incubator-pinot/pull/6658/diff?src=pr&el=tree#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvYnJva2VyL2hlbGl4L0Jyb2tlclVzZXJEZWZpbmVkTWVzc2FnZUhhbmRsZXJGYWN0b3J5LmphdmE=) | `52.83% <0.00%> (-13.84%)` | :arrow_down: |
| [...ker/routing/instanceselector/InstanceSelector.java](https://codecov.io/gh/apache/incubator-pinot/pull/6658/diff?src=pr&el=tree#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvcm91dGluZy9pbnN0YW5jZXNlbGVjdG9yL0luc3RhbmNlU2VsZWN0b3IuamF2YQ==) | `100.00% <ø> (ø)` | |
| [...ava/org/apache/pinot/client/AbstractResultSet.java](https://codecov.io/gh/apache/incubator-pinot/pull/6658/diff?src=pr&el=tree#diff-cGlub3QtY2xpZW50cy9waW5vdC1qYXZhLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY2xpZW50L0Fic3RyYWN0UmVzdWx0U2V0LmphdmE=) | `73.33% <ø> (+16.19%)` | :arrow_up: |
| [...n/java/org/apache/pinot/client/BrokerResponse.java](https://codecov.io/gh/apache/incubator-pinot/pull/6658/diff?src=pr&el=tree#diff-cGlub3QtY2xpZW50cy9waW5vdC1qYXZhLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY2xpZW50L0Jyb2tlclJlc3BvbnNlLmphdmE=) | `100.00% <ø> (ø)` | |
| [.../main/java/org/apache/pinot/client/Connection.java](https://codecov.io/gh/apache/incubator-pinot/pull/6658/diff?src=pr&el=tree#diff-cGlub3QtY2xpZW50cy9waW5vdC1qYXZhLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY2xpZW50L0Nvbm5lY3Rpb24uamF2YQ==) | `44.44% <ø> (-4.40%)` | :arrow_down: |
| [...org/apache/pinot/client/DynamicBrokerSelector.java](https://codecov.io/gh/apache/incubator-pinot/pull/6658/diff?src=pr&el=tree#diff-cGlub3QtY2xpZW50cy9waW5vdC1qYXZhLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY2xpZW50L0R5bmFtaWNCcm9rZXJTZWxlY3Rvci5qYXZh) | `88.57% <ø> (+15.84%)` | :arrow_up: |
| [...n/java/org/apache/pinot/client/ExecutionStats.java](https://codecov.io/gh/apache/incubator-pinot/pull/6658/diff?src=pr&el=tree#diff-cGlub3QtY2xpZW50cy9waW5vdC1qYXZhLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY2xpZW50L0V4ZWN1dGlvblN0YXRzLmphdmE=) | `68.88% <ø> (ø)` | |
| [...va/org/apache/pinot/client/ExternalViewReader.java](https://codecov.io/gh/apache/incubator-pinot/pull/6658/diff?src=pr&el=tree#diff-cGlub3QtY2xpZW50cy9waW5vdC1qYXZhLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY2xpZW50L0V4dGVybmFsVmlld1JlYWRlci5qYXZh) | `81.69% <ø> (+31.69%)` | :arrow_up: |
| [...inot/client/JsonAsyncHttpPinotClientTransport.java](https://codecov.io/gh/apache/incubator-pinot/pull/6658/diff?src=pr&el=tree#diff-cGlub3QtY2xpZW50cy9waW5vdC1qYXZhLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY2xpZW50L0pzb25Bc3luY0h0dHBQaW5vdENsaWVudFRyYW5zcG9ydC5qYXZh) | `56.36% <ø> (-5.64%)` | :arrow_down: |
| ... and [1206 more](https://codecov.io/gh/apache/incubator-pinot/pull/6658/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/6658?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/6658?src=pr&el=footer). Last update [1dbdd67...f72b2a5](https://codecov.io/gh/apache/incubator-pinot/pull/6658?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] Jackie-Jiang commented on a change in pull request #6658: Add context on data type transform failure
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on a change in pull request #6658:
URL: https://github.com/apache/incubator-pinot/pull/6658#discussion_r589836707
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/data/recordtransformer/DataTypeTransformer.java
##########
@@ -80,39 +80,43 @@ public DataTypeTransformer(Schema schema) {
public GenericRow transform(GenericRow record) {
for (Map.Entry<String, PinotDataType> entry : _dataTypes.entrySet()) {
String column = entry.getKey();
- Object value = record.getValue(column);
- if (value == null) {
- continue;
- }
- PinotDataType dest = entry.getValue();
- value = standardize(column, value, dest.isSingleValue());
- // NOTE: The standardized value could be null for empty Collection/Map/Object[].
- if (value == null) {
- record.putValue(column, null);
- continue;
- }
+ try {
+ Object value = record.getValue(column);
+ if (value == null) {
+ continue;
+ }
+ PinotDataType dest = entry.getValue();
+ value = standardize(column, value, dest.isSingleValue());
+ // NOTE: The standardized value could be null for empty Collection/Map/Object[].
+ if (value == null) {
+ record.putValue(column, null);
+ continue;
+ }
- // Convert data type if necessary
- PinotDataType source;
- if (value instanceof Object[]) {
- // Multi-value column
- Object[] values = (Object[]) value;
- source = MULTI_VALUE_TYPE_MAP.get(values[0].getClass());
- if (source == null) {
- source = PinotDataType.OBJECT_ARRAY;
+ // Convert data type if necessary
+ PinotDataType source;
+ if (value instanceof Object[]) {
+ // Multi-value column
+ Object[] values = (Object[]) value;
+ source = MULTI_VALUE_TYPE_MAP.get(values[0].getClass());
+ if (source == null) {
+ source = PinotDataType.OBJECT_ARRAY;
+ }
+ } else {
+ // Single-value column
+ source = SINGLE_VALUE_TYPE_MAP.get(value.getClass());
+ if (source == null) {
+ source = PinotDataType.OBJECT;
+ }
}
- } else {
- // Single-value column
- source = SINGLE_VALUE_TYPE_MAP.get(value.getClass());
- if (source == null) {
- source = PinotDataType.OBJECT;
+ if (source != dest) {
+ value = dest.convert(value, source);
}
- }
- if (source != dest) {
- value = dest.convert(value, source);
- }
- record.putValue(column, value);
+ record.putValue(column, value);
+ } catch (Exception e) {
+ throw new RuntimeException("failed to transform column " + column, e);
Review comment:
```suggestion
throw new RuntimeException("Caught exception while transforming data type for column: " + column, e);
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] Jackie-Jiang merged pull request #6658: Add context on data type transform failure
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang merged pull request #6658:
URL: https://github.com/apache/incubator-pinot/pull/6658
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org