You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2022/05/04 15:26:05 UTC
[GitHub] [pulsar] Raven888888 opened a new issue, #12284: Pulsar SQL BufferUnderflowException after upgrading from 2.7.0 to 2.8.1
Raven888888 opened a new issue, #12284:
URL: https://github.com/apache/pulsar/issues/12284
**Describe the bug**
Following [this upgrade guide](https://pulsar.apache.org/docs/en/administration-upgrade/) to upgrade pulsar cluster node by node, from `2.7.0` to `2.8.1`.
Presto CLI version `332` is after the upgrade.
Trying to query in Presto CLI from one of the topics,
`apache-pulsar/bin/pulsar sql`
`select * from pulsar."public/default"."test";`
returns
`Query 20211006_093830_00017_59f2r failed: java.nio.BufferUnderflowException`
(See below for full logs)
However, this only affects some topics. Other topics in the same tenant and namespace can be queried just fine.
Also, using [`pulsar flink connector`](https://github.com/streamnative/pulsar-flink) and [`pulsar client python api`](https://pulsar.apache.org/docs/en/client-libraries-python/#reader-interface-example) to consume data from the problematic topics are working as expected, no issues.
Full logs
```
2021-10-06T09:38:30.837Z ERROR remote-task-callback-89 io.prestosql.execution.StageStateMachine Stage 20211006_093830_00017_59f2r.1 failed
com.google.common.util.concurrent.UncheckedExecutionException: java.nio.BufferUnderflowException
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2051)
at com.google.common.cache.LocalCache.get(LocalCache.java:3951)
at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3974)
at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4935)
at org.apache.pulsar.sql.presto.PulsarSqlSchemaInfoProvider.getSchemaByVersion(PulsarSqlSchemaInfoProvider.java:76)
at org.apache.pulsar.sql.presto.PulsarRecordCursor.advanceNextPosition(PulsarRecordCursor.java:485)
at io.prestosql.spi.connector.RecordPageSource.getNextPage(RecordPageSource.java:90)
at io.prestosql.operator.TableScanOperator.getOutput(TableScanOperator.java:302)
at io.prestosql.operator.Driver.processInternal(Driver.java:379)
at io.prestosql.operator.Driver.lambda$processFor$8(Driver.java:283)
at io.prestosql.operator.Driver.tryWithLock(Driver.java:675)
at io.prestosql.operator.Driver.processFor(Driver.java:276)
at io.prestosql.execution.SqlTaskExecution$DriverSplitRunner.processFor(SqlTaskExecution.java:1075)
at io.prestosql.execution.executor.PrioritizedSplitRunner.process(PrioritizedSplitRunner.java:163)
at io.prestosql.execution.executor.TaskExecutor$TaskRunner.run(TaskExecutor.java:484)
at io.prestosql.$gen.Presto_332__testversion____20211005_104810_2.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.nio.BufferUnderflowException
at java.nio.Buffer.nextGetIndex(Buffer.java:510)
at java.nio.HeapByteBuffer.getLong(HeapByteBuffer.java:427)
at org.apache.pulsar.sql.presto.PulsarSqlSchemaInfoProvider.loadSchema(PulsarSqlSchemaInfoProvider.java:106)
at org.apache.pulsar.sql.presto.PulsarSqlSchemaInfoProvider.access$000(PulsarSqlSchemaInfoProvider.java:49)
at org.apache.pulsar.sql.presto.PulsarSqlSchemaInfoProvider$1.load(PulsarSqlSchemaInfoProvider.java:61)
at org.apache.pulsar.sql.presto.PulsarSqlSchemaInfoProvider$1.load(PulsarSqlSchemaInfoProvider.java:58)
at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3529)
at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2278)
at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2155)
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2045)
... 18 more
```
PS: Saw similar [issue](https://github.com/apache/pulsar/pull/9631), but that one is about **byte schema data**, mine is **string schema data**.
Suspect presto has messed up its cache somehow. Any pointer on how to identify the root cause and/or fix this issue will be greatly appreciated. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codelipenghui commented on issue #12284: Pulsar SQL BufferUnderflowException after upgrading from 2.7.0 to 2.8.1
Posted by GitBox <gi...@apache.org>.
codelipenghui commented on issue #12284:
URL: https://github.com/apache/pulsar/issues/12284#issuecomment-1121192107
@Technoboy- This one should be related to https://lists.apache.org/thread/3js51tq2p3c3oldfrhprn4kcohx7h1wv ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Raven888888 commented on issue #12284: Pulsar SQL BufferUnderflowException after upgrading from 2.7.0 to 2.8.1
Posted by "Raven888888 (via GitHub)" <gi...@apache.org>.
Raven888888 commented on issue #12284:
URL: https://github.com/apache/pulsar/issues/12284#issuecomment-1603478428
Any update?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Raven888888 commented on issue #12284: Pulsar SQL BufferUnderflowException after upgrading from 2.7.0 to 2.8.1
Posted by GitBox <gi...@apache.org>.
Raven888888 commented on issue #12284:
URL: https://github.com/apache/pulsar/issues/12284#issuecomment-1121027262
I noticed that one of the differences of `broker.conf` between pulsar 2.7.0 and pulsar 2.8.0+ is the introduction of this
```
# The schema compatibility strategy in broker level.
# SchemaCompatibilityStrategy : ALWAYS_INCOMPATIBLE, ALWAYS_COMPATIBLE, BACKWARD, FORWARD,
# FULL, BACKWARD_TRANSITIVE, FORWARD_TRANSITIVE, FULL_TRANSITIVE
schemaCompatibilityStrategy=FULL
```
Could it be what causing the `BufferUnderflowException`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] gaoran10 commented on issue #12284: Pulsar SQL BufferUnderflowException after upgrading from 2.7.0 to 2.8.1
Posted by "gaoran10 (via GitHub)" <gi...@apache.org>.
gaoran10 commented on issue #12284:
URL: https://github.com/apache/pulsar/issues/12284#issuecomment-1617275046
@Raven888888 Which version do you use? I test with Pulsar 3.0.0, it works well, could you try to use Pulsar 3.0.0?
#### Test steps
1. Use websocket client to produce a message without schema to a topic
2. Upload string schema to the topic
3. Query with Pulsar SQL
Before upload string schema
```
presto> select * from pulsar."public/default"."t1";
__value__ | __partition__ | __event_time__ | __publish_time__ | __message_id__ | __sequence_id__ | __producer_name__ | __key__ | __properties__
----------------------------------+---------------+----------------+-------------------------+----------------+-----------------+-------------------+---------+-----------------------------------
48 65 6c 6c 6f 20 57 6f 72 6c 64 | -1 | NULL | 2023-07-03 03:56:43.873 | (15,5,0) | 5 | standalone-0-9 | NULL | {"key1":"value1","key2":"value2"}
48 65 6c 6c 6f 20 57 6f 72 6c 64 | -1 | NULL | 2023-07-03 03:56:43.873 | (15,6,0) | 6 | standalone-0-9 | NULL | {"key1":"value1","key2":"value2"}
48 65 6c 6c 6f 20 57 6f 72 6c 64 | -1 | NULL | 2023-07-03 03:56:43.873 | (15,7,0) | 7 | standalone-0-9 | NULL | {"key1":"value1","key2":"value2"}
48 65 6c 6c 6f 20 57 6f 72 6c 64 | -1 | NULL | 2023-07-03 03:56:43.873 | (15,8,0) | 8 | standalone-0-9 | NULL | {"key1":"value1","key2":"value2"}
48 65 6c 6c 6f 20 57 6f 72 6c 64 | -1 | NULL | 2023-07-03 03:56:43.874 | (15,9,0) | 9 | standalone-0-9 | NULL | {"key1":"value1","key2":"value2"}
48 65 6c 6c 6f 20 57 6f 72 6c 64 | -1 | NULL | 2023-07-03 03:56:43.870 | (15,0,0) | 0 | standalone-0-9 | NULL | {"key1":"value1","key2":"value2"}
48 65 6c 6c 6f 20 57 6f 72 6c 64 | -1 | NULL | 2023-07-03 03:56:43.871 | (15,1,0) | 1 | standalone-0-9 | NULL | {"key1":"value1","key2":"value2"}
48 65 6c 6c 6f 20 57 6f 72 6c 64 | -1 | NULL | 2023-07-03 03:56:43.871 | (15,2,0) | 2 | standalone-0-9 | NULL | {"key1":"value1","key2":"value2"}
48 65 6c 6c 6f 20 57 6f 72 6c 64 | -1 | NULL | 2023-07-03 03:56:43.872 | (15,3,0) | 3 | standalone-0-9 | NULL | {"key1":"value1","key2":"value2"}
48 65 6c 6c 6f 20 57 6f 72 6c 64 | -1 | NULL | 2023-07-03 03:56:43.872 | (15,4,0) | 4 | standalone-0-9 | NULL | {"key1":"value1","key2":"value2"}
(10 rows)
Query 20230703_040424_00001_2a95d, FINISHED, 1 node
Splits: 18 total, 18 done (100.00%)
0:00 [10 rows, 1.17KB] [46 rows/s, 5.48KB/s]
```
After upload string schema
```
presto> select * from pulsar."public/default"."t1";
__value__ | __partition__ | __event_time__ | __publish_time__ | __message_id__ | __sequence_id__ | __producer_name__ | __key__ | __properties__
-------------+---------------+----------------+-------------------------+----------------+-----------------+-------------------+---------+-----------------------------------
Hello World | -1 | NULL | 2023-07-03 03:56:43.873 | (15,5,0) | 5 | standalone-0-9 | NULL | {"key1":"value1","key2":"value2"}
Hello World | -1 | NULL | 2023-07-03 03:56:43.873 | (15,6,0) | 6 | standalone-0-9 | NULL | {"key1":"value1","key2":"value2"}
Hello World | -1 | NULL | 2023-07-03 03:56:43.873 | (15,7,0) | 7 | standalone-0-9 | NULL | {"key1":"value1","key2":"value2"}
Hello World | -1 | NULL | 2023-07-03 03:56:43.873 | (15,8,0) | 8 | standalone-0-9 | NULL | {"key1":"value1","key2":"value2"}
Hello World | -1 | NULL | 2023-07-03 03:56:43.874 | (15,9,0) | 9 | standalone-0-9 | NULL | {"key1":"value1","key2":"value2"}
Hello World | -1 | NULL | 2023-07-03 03:56:43.870 | (15,0,0) | 0 | standalone-0-9 | NULL | {"key1":"value1","key2":"value2"}
Hello World | -1 | NULL | 2023-07-03 03:56:43.871 | (15,1,0) | 1 | standalone-0-9 | NULL | {"key1":"value1","key2":"value2"}
Hello World | -1 | NULL | 2023-07-03 03:56:43.871 | (15,2,0) | 2 | standalone-0-9 | NULL | {"key1":"value1","key2":"value2"}
Hello World | -1 | NULL | 2023-07-03 03:56:43.872 | (15,3,0) | 3 | standalone-0-9 | NULL | {"key1":"value1","key2":"value2"}
Hello World | -1 | NULL | 2023-07-03 03:56:43.872 | (15,4,0) | 4 | standalone-0-9 | NULL | {"key1":"value1","key2":"value2"}
(10 rows)
Query 20230703_042603_00002_2a95d, FINISHED, 1 node
Splits: 18 total, 18 done (100.00%)
0:00 [10 rows, 1.17KB] [41 rows/s, 4.9KB/s]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Raven888888 commented on issue #12284: Pulsar SQL BufferUnderflowException after upgrading from 2.7.0 to 2.8.1
Posted by "Raven888888 (via GitHub)" <gi...@apache.org>.
Raven888888 commented on issue #12284:
URL: https://github.com/apache/pulsar/issues/12284#issuecomment-1619260264
Thanks @gaoran10
I have tried through versions 2.7 (works), 2.8-2.10 (all have BufferUnderflowException). I have yet to try version 3.x, which I will in my nearest capacity.
That said, I think your test step should be:
1. Use websocket client to produce a message without schema to a topic
1. **Use pulsar client to produce a message without schema to a topic**
1. Upload string schema to the topic
1. Query with Pulsar SQL
I notice it happens when I have 2 kinds of clients producing message into the same topic, and it breaks pulsar SQL.
I still able to read from the topic just fine using pulsar client or websocket client though.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Raven888888 commented on issue #12284: Pulsar SQL BufferUnderflowException after upgrading from 2.7.0 to 2.8.1
Posted by GitBox <gi...@apache.org>.
Raven888888 commented on issue #12284:
URL: https://github.com/apache/pulsar/issues/12284#issuecomment-1117489085
@gaoran10 Sorry to say but the issue I faced still persists.
I recently only have the time to try your PR #12809 . I used the official 2.10.0 binary, which should already contain your PR, and did a clean install.
However, I still face the exact same `BufferUnderflowException` error (and same error log) when I
1. use websocket client to produce a message *without schema* to a topic
2. upload `string` schema to the topic (`conf/schema_example.conf`)
3. face the error when query from pulsar SQL
Note that:
- I use the default settings in `broker.conf`, not sure if they play a role here. Some of the paramaters are added after pulsar 2.7.0, which leads me to relate that this `BufferUnderflowException` did NOT happen in pulsar 2.7.0 before...
```
isAllowAutoUpdateSchemaEnabled=true
systemTopicSchemaCompatibilityStrategy=ALWAYS_COMPATIBLE
topicLevelPoliciesEnabled=false
isSchemaValidationEnforced=false
schemaCompatibilityStrategy=FULL
```
- I understand that using a pulsar client that supports producing schema, can solve the issue. However, **this requires changes on client side and is unfortunately not possible**. I'd like a solution on the server side.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Raven888888 commented on issue #12284: Pulsar SQL BufferUnderflowException after upgrading from 2.7.0 to 2.8.1
Posted by GitBox <gi...@apache.org>.
Raven888888 commented on issue #12284:
URL: https://github.com/apache/pulsar/issues/12284#issuecomment-1117489285
@gaoran10
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org