You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2021/11/03 21:29:03 UTC
[GitHub] [pinot] mapshen opened a new issue #7694: Data schema mismatch between merged block
mapshen opened a new issue #7694:
URL: https://github.com/apache/pinot/issues/7694
We run a realtime table `table1` with fields `X` in **upsert** mode. When a new field `Y` is added to the schema, a simple `select` query in the Pinot explorer will return the following error:
```
[
{
"message": "MergeResponseError:\nData schema mismatch between merged block: [X(DOUBLE)] and block to merge: [X(DOUBLE),Y(DOUBLE)], drop block to merge",
"errorCode": 500
}
]
```
The root cause seems to be that not all records are updated to have the `Y` field after the schema change. In the worst case, some record will never receive an update to contain that `Y` field.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on issue #7694: Data schema mismatch between merged block
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on issue #7694:
URL: https://github.com/apache/pinot/issues/7694#issuecomment-960316879
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] mapshen commented on issue #7694: Data schema mismatch between merged block
Posted by GitBox <gi...@apache.org>.
mapshen commented on issue #7694:
URL: https://github.com/apache/pinot/issues/7694#issuecomment-961270251
@Jackie-Jiang it doesn't seem explain why we don't experience such an issue with tables without having `upsert` enabled.
Even in this `table1`, we only receive the error with
```
select * from table1 limit 10
```
The error will not show up if we do
```
select * from table1 limit 10 option (skipUpsert=True)
```
That said, maybe we are doing the reload incorrectly? Can you let us know the right way to do it?
Also, what is the right way to restart the server so that a consuming segment can be closed and persisted to disk?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] kkrugler commented on issue #7694: Data schema mismatch between merged block
Posted by GitBox <gi...@apache.org>.
kkrugler commented on issue #7694:
URL: https://github.com/apache/pinot/issues/7694#issuecomment-961279578
Hi @mapshen - I'd suggest you repost this to the Slack #troubleshooting channel, as that will make it visible to the wider Pinot community, and provides faster/more effective resolution. It also exposes the conversation to other Pinot users, which is often very helpful as they deal with similar problems, thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] mapshen commented on issue #7694: Data schema mismatch between merged block
Posted by GitBox <gi...@apache.org>.
mapshen commented on issue #7694:
URL: https://github.com/apache/pinot/issues/7694#issuecomment-960345865
@Jackie-Jiang Yes we did. We don't see this error with a regular table. This only manifests when `upsert` is on. Does `reload` rebuild the index `upsert` uses?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on issue #7694: Data schema mismatch between merged block
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on issue #7694:
URL: https://github.com/apache/pinot/issues/7694#issuecomment-960352473
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] mapshen commented on issue #7694: Data schema mismatch between merged block
Posted by GitBox <gi...@apache.org>.
mapshen commented on issue #7694:
URL: https://github.com/apache/pinot/issues/7694#issuecomment-961270251
@Jackie-Jiang it doesn't seem explain why we don't experience such an issue with tables without having `upsert` enabled.
Even in this `table1`, we only receive the error with
```
select * from table1 limit 10
```
The error will not show up if we do
```
select * from table1 limit 10 option (skipUpsert=True)
```
That said, maybe we are doing the reload incorrectly? Can you let us know the right way to do it?
Also, what is the right way to restart the server so that a consuming segment can be closed and persisted to disk?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] mapshen commented on issue #7694: Data schema mismatch between merged block
Posted by GitBox <gi...@apache.org>.
mapshen commented on issue #7694:
URL: https://github.com/apache/pinot/issues/7694#issuecomment-960345865
@Jackie-Jiang Yes we did. We don't see this error with a regular table. This only manifests when `upsert` is on. Does `reload` rebuild the index `upsert` uses?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] kkrugler commented on issue #7694: Data schema mismatch between merged block
Posted by GitBox <gi...@apache.org>.
kkrugler commented on issue #7694:
URL: https://github.com/apache/pinot/issues/7694#issuecomment-961279578
Hi @mapshen - I'd suggest you repost this to the Slack #troubleshooting channel, as that will make it visible to the wider Pinot community, and provides faster/more effective resolution. It also exposes the conversation to other Pinot users, which is often very helpful as they deal with similar problems, thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] kkrugler commented on issue #7694: Data schema mismatch between merged block
Posted by GitBox <gi...@apache.org>.
kkrugler commented on issue #7694:
URL: https://github.com/apache/pinot/issues/7694#issuecomment-961279578
Hi @mapshen - I'd suggest you repost this to the Slack #troubleshooting channel, as that will make it visible to the wider Pinot community, and provides faster/more effective resolution. It also exposes the conversation to other Pinot users, which is often very helpful as they deal with similar problems, thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org