You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2021/10/06 10:08:20 UTC
[GitHub] [pulsar] Raven888888 opened a new issue #12284: Pulsar SQL BufferUnderflowException after upgrading from 2.7.0 to 2.8.1
Raven888888 opened a new issue #12284:
URL: https://github.com/apache/pulsar/issues/12284
**Describe the bug**
Following [this upgrade guide](https://pulsar.apache.org/docs/en/administration-upgrade/) to upgrade pulsar cluster node by node, from `2.7.0` to `2.8.1`.
Presto CLI version `332` is after the upgrade.
Trying to query in Presto CLI from one of the topics,
`apache-pulsar/bin/pulsar sql`
`select * from pulsar."public/default"."test";`
returns
`Query 20211006_093830_00017_59f2r failed: java.nio.BufferUnderflowException`
(See below for full logs)
However, this only affects some topics. Other topics in the same tenant and namespace can be queried just fine.
Also, using [`pulsar flink connector`](https://github.com/streamnative/pulsar-flink) and [`pulsar client python api`](https://pulsar.apache.org/docs/en/client-libraries-python/#reader-interface-example) to consume data from the problematic topics are working as expected, no issues.
Full logs
```
2021-10-06T09:38:30.837Z ERROR remote-task-callback-89 io.prestosql.execution.StageStateMachine Stage 20211006_093830_00017_59f2r.1 failed
com.google.common.util.concurrent.UncheckedExecutionException: java.nio.BufferUnderflowException
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2051)
at com.google.common.cache.LocalCache.get(LocalCache.java:3951)
at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3974)
at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4935)
at org.apache.pulsar.sql.presto.PulsarSqlSchemaInfoProvider.getSchemaByVersion(PulsarSqlSchemaInfoProvider.java:76)
at org.apache.pulsar.sql.presto.PulsarRecordCursor.advanceNextPosition(PulsarRecordCursor.java:485)
at io.prestosql.spi.connector.RecordPageSource.getNextPage(RecordPageSource.java:90)
at io.prestosql.operator.TableScanOperator.getOutput(TableScanOperator.java:302)
at io.prestosql.operator.Driver.processInternal(Driver.java:379)
at io.prestosql.operator.Driver.lambda$processFor$8(Driver.java:283)
at io.prestosql.operator.Driver.tryWithLock(Driver.java:675)
at io.prestosql.operator.Driver.processFor(Driver.java:276)
at io.prestosql.execution.SqlTaskExecution$DriverSplitRunner.processFor(SqlTaskExecution.java:1075)
at io.prestosql.execution.executor.PrioritizedSplitRunner.process(PrioritizedSplitRunner.java:163)
at io.prestosql.execution.executor.TaskExecutor$TaskRunner.run(TaskExecutor.java:484)
at io.prestosql.$gen.Presto_332__testversion____20211005_104810_2.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.nio.BufferUnderflowException
at java.nio.Buffer.nextGetIndex(Buffer.java:510)
at java.nio.HeapByteBuffer.getLong(HeapByteBuffer.java:427)
at org.apache.pulsar.sql.presto.PulsarSqlSchemaInfoProvider.loadSchema(PulsarSqlSchemaInfoProvider.java:106)
at org.apache.pulsar.sql.presto.PulsarSqlSchemaInfoProvider.access$000(PulsarSqlSchemaInfoProvider.java:49)
at org.apache.pulsar.sql.presto.PulsarSqlSchemaInfoProvider$1.load(PulsarSqlSchemaInfoProvider.java:61)
at org.apache.pulsar.sql.presto.PulsarSqlSchemaInfoProvider$1.load(PulsarSqlSchemaInfoProvider.java:58)
at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3529)
at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2278)
at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2155)
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2045)
... 18 more
```
PS: Saw similar [issue](https://github.com/apache/pulsar/pull/9631), but that one is about **byte schema data**, mine is **string schema data**.
Suspect presto has messed up its cache somehow. Any pointer on how to identify the root cause and/or fix this issue will be greatly appreciated. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] MarvinCai edited a comment on issue #12284: Pulsar SQL BufferUnderflowException after upgrading from 2.7.0 to 2.8.1
Posted by GitBox <gi...@apache.org>.
MarvinCai edited a comment on issue #12284:
URL: https://github.com/apache/pulsar/issues/12284#issuecomment-957368879
sorry for delay, had some discussion with @gaoran10 offline
The Websocket endpoint doesn't support schema, the internal implementation simply produce raw byte([ref](https://github.com/apache/pulsar/blob/master/pulsar-websocket/src/main/java/org/apache/pulsar/websocket/ProducerHandler.java#L73))
And when the Pulsar topic has schema, Pulsar SQL try to ready the message and will expect a schema, while there'll be no schema on the message, causing NPE.
But when we remove the schema, Pulsar SQL will know that and just use a Byte schema by default([ref](https://github.com/apache/pulsar/blob/master/pulsar-sql/presto-pulsar/src/main/java/org/apache/pulsar/sql/presto/PulsarSplitManager.java#L117-L118)), that should explain why it's working after you remove the schema from the topic.
If you try produce with a client that support passing schema, Pulsar SQL should work as expected.
Actually here's a handy cli tool to ingest NYC taxi data to a Pulsar cluster with Schema as test dataset which make it easy to test Pulsar SQL: https://github.com/streamnative/examples/tree/master/nyctaxi/taxidata , the schema: https://github.com/streamnative/examples/blob/master/nyctaxi/taxidata/pkg/types/yellow.go
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Raven888888 closed issue #12284: Pulsar SQL BufferUnderflowException after upgrading from 2.7.0 to 2.8.1
Posted by GitBox <gi...@apache.org>.
Raven888888 closed issue #12284:
URL: https://github.com/apache/pulsar/issues/12284
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] MarvinCai edited a comment on issue #12284: Pulsar SQL BufferUnderflowException after upgrading from 2.7.0 to 2.8.1
Posted by GitBox <gi...@apache.org>.
MarvinCai edited a comment on issue #12284:
URL: https://github.com/apache/pulsar/issues/12284#issuecomment-957368879
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Raven888888 commented on issue #12284: Pulsar SQL BufferUnderflowException after upgrading from 2.7.0 to 2.8.1
Posted by GitBox <gi...@apache.org>.
Raven888888 commented on issue #12284:
URL: https://github.com/apache/pulsar/issues/12284#issuecomment-958588769
@MarvinCai @gaoran10 Thank you for your explanation, clears up a bit. I have further follow-up questions:
1. In pulsar 2.7.0, I too used websocket to publish data (as you pointed out rightfully, data in raw bytes) to pulsar. And pulsar SQL works fine in that version, where I manually upload `string` schema. Is there changes in schema handling from 2.7.x to 2.8.x, that causes pulsar SQL to break?
2. Is it possible to get back pulsar 2.7.0 behaviour in 2.8.x and beyond? I still want to use websocket to publish data (raw bytes), and sets pulsar topic with `string` schema, and able to query using pulsar SQL.
Thanks a lot.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] gaoran10 commented on issue #12284: Pulsar SQL BufferUnderflowException after upgrading from 2.7.0 to 2.8.1
Posted by GitBox <gi...@apache.org>.
gaoran10 commented on issue #12284:
URL: https://github.com/apache/pulsar/issues/12284#issuecomment-956289288
It seems that the schema version of the message is null, which language client do messages come from?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] MarvinCai commented on issue #12284: Pulsar SQL BufferUnderflowException after upgrading from 2.7.0 to 2.8.1
Posted by GitBox <gi...@apache.org>.
MarvinCai commented on issue #12284:
URL: https://github.com/apache/pulsar/issues/12284#issuecomment-957368879
sorry for delay, had some discussion with @gaoran10 offline
The Websocket endpoint doesn't support schema, the internal simply use byte schema([ref](https://github.com/apache/pulsar/blob/master/pulsar-websocket/src/main/java/org/apache/pulsar/websocket/ProducerHandler.java#L73))
And when the Pulsar topic has schema, Pulsar SQL try to ready the message and will expect a schema, but there'll be no schema, causing NPE.
But when we remove the schema, Pulsar SQL will know that and just use a Byte schema by default([ref](https://github.com/apache/pulsar/blob/master/pulsar-sql/presto-pulsar/src/main/java/org/apache/pulsar/sql/presto/PulsarSplitManager.java#L117-L118)), that should explain why it's working after you remove the schema
If you try produce with a client which support passing schema, Pulsar SQL should work as expected.
Actually here's a handy cli tool to ingest NYC taxi data to a Pulsar cluster with Schema which make it easy to test Pulsar SQL: https://github.com/streamnative/examples/tree/master/nyctaxi/taxidata , the schema: https://github.com/streamnative/examples/blob/master/nyctaxi/taxidata/pkg/types/yellow.go
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] gaoran10 commented on issue #12284: Pulsar SQL BufferUnderflowException after upgrading from 2.7.0 to 2.8.1
Posted by GitBox <gi...@apache.org>.
gaoran10 commented on issue #12284:
URL: https://github.com/apache/pulsar/issues/12284#issuecomment-975010303
We add a PR https://github.com/apache/pulsar/pull/12809 to handle the null schema version problem.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] MarvinCai edited a comment on issue #12284: Pulsar SQL BufferUnderflowException after upgrading from 2.7.0 to 2.8.1
Posted by GitBox <gi...@apache.org>.
MarvinCai edited a comment on issue #12284:
URL: https://github.com/apache/pulsar/issues/12284#issuecomment-957368879
sorry for delay, had some discussion with @gaoran10 offline
The Websocket endpoint doesn't support schema, the internal implementation simply use byte schema([ref](https://github.com/apache/pulsar/blob/master/pulsar-websocket/src/main/java/org/apache/pulsar/websocket/ProducerHandler.java#L73))
And when the Pulsar topic has schema, Pulsar SQL try to ready the message and will expect a schema, but there'll be no schema, causing NPE.
But when we remove the schema, Pulsar SQL will know that and just use a Byte schema by default([ref](https://github.com/apache/pulsar/blob/master/pulsar-sql/presto-pulsar/src/main/java/org/apache/pulsar/sql/presto/PulsarSplitManager.java#L117-L118)), that should explain why it's working after you remove the schema
If you try produce with a client which support passing schema, Pulsar SQL should work as expected.
Actually here's a handy cli tool to ingest NYC taxi data to a Pulsar cluster with Schema which make it easy to test Pulsar SQL: https://github.com/streamnative/examples/tree/master/nyctaxi/taxidata , the schema: https://github.com/streamnative/examples/blob/master/nyctaxi/taxidata/pkg/types/yellow.go
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] MarvinCai commented on issue #12284: Pulsar SQL BufferUnderflowException after upgrading from 2.7.0 to 2.8.1
Posted by GitBox <gi...@apache.org>.
MarvinCai commented on issue #12284:
URL: https://github.com/apache/pulsar/issues/12284#issuecomment-938434580
The exception seems come from Pulsar connector not Presto as Pulsar connector manages the Schema and cache, seems the exceptions comes from this line: https://github.com/apache/pulsar/blob/master/pulsar-sql/presto-pulsar/src/main/java/org/apache/pulsar/sql/presto/PulsarSqlSchemaInfoProvider.java#L106 which is throwing exception when parsing schema version, can you provide schema info using pulsar-admin, so we can verify the schema info is not corrupted.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] MarvinCai commented on issue #12284: Pulsar SQL BufferUnderflowException after upgrading from 2.7.0 to 2.8.1
Posted by GitBox <gi...@apache.org>.
MarvinCai commented on issue #12284:
URL: https://github.com/apache/pulsar/issues/12284#issuecomment-957368879
sorry for delay, had some discussion with @gaoran10 offline
The Websocket endpoint doesn't support schema, the internal simply use byte schema([ref](https://github.com/apache/pulsar/blob/master/pulsar-websocket/src/main/java/org/apache/pulsar/websocket/ProducerHandler.java#L73))
And when the Pulsar topic has schema, Pulsar SQL try to ready the message and will expect a schema, but there'll be no schema, causing NPE.
But when we remove the schema, Pulsar SQL will know that and just use a Byte schema by default([ref](https://github.com/apache/pulsar/blob/master/pulsar-sql/presto-pulsar/src/main/java/org/apache/pulsar/sql/presto/PulsarSplitManager.java#L117-L118)), that should explain why it's working after you remove the schema
If you try produce with a client which support passing schema, Pulsar SQL should work as expected.
Actually here's a handy cli tool to ingest NYC taxi data to a Pulsar cluster with Schema which make it easy to test Pulsar SQL: https://github.com/streamnative/examples/tree/master/nyctaxi/taxidata , the schema: https://github.com/streamnative/examples/blob/master/nyctaxi/taxidata/pkg/types/yellow.go
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] MarvinCai commented on issue #12284: Pulsar SQL BufferUnderflowException after upgrading from 2.7.0 to 2.8.1
Posted by GitBox <gi...@apache.org>.
MarvinCai commented on issue #12284:
URL: https://github.com/apache/pulsar/issues/12284#issuecomment-957368879
sorry for delay, had some discussion with @gaoran10 offline
The Websocket endpoint doesn't support schema, the internal simply use byte schema([ref](https://github.com/apache/pulsar/blob/master/pulsar-websocket/src/main/java/org/apache/pulsar/websocket/ProducerHandler.java#L73))
And when the Pulsar topic has schema, Pulsar SQL try to ready the message and will expect a schema, but there'll be no schema, causing NPE.
But when we remove the schema, Pulsar SQL will know that and just use a Byte schema by default([ref](https://github.com/apache/pulsar/blob/master/pulsar-sql/presto-pulsar/src/main/java/org/apache/pulsar/sql/presto/PulsarSplitManager.java#L117-L118)), that should explain why it's working after you remove the schema
If you try produce with a client which support passing schema, Pulsar SQL should work as expected.
Actually here's a handy cli tool to ingest NYC taxi data to a Pulsar cluster with Schema which make it easy to test Pulsar SQL: https://github.com/streamnative/examples/tree/master/nyctaxi/taxidata , the schema: https://github.com/streamnative/examples/blob/master/nyctaxi/taxidata/pkg/types/yellow.go
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] MarvinCai edited a comment on issue #12284: Pulsar SQL BufferUnderflowException after upgrading from 2.7.0 to 2.8.1
Posted by GitBox <gi...@apache.org>.
MarvinCai edited a comment on issue #12284:
URL: https://github.com/apache/pulsar/issues/12284#issuecomment-957368879
sorry for delay, had some discussion with @gaoran10 offline
The Websocket endpoint doesn't support schema, the internal implementation simply use byte schema([ref](https://github.com/apache/pulsar/blob/master/pulsar-websocket/src/main/java/org/apache/pulsar/websocket/ProducerHandler.java#L73))
And when the Pulsar topic has schema, Pulsar SQL try to ready the message and will expect a schema, while there'll be no schema on the message, causing NPE.
But when we remove the schema, Pulsar SQL will know that and just use a Byte schema by default([ref](https://github.com/apache/pulsar/blob/master/pulsar-sql/presto-pulsar/src/main/java/org/apache/pulsar/sql/presto/PulsarSplitManager.java#L117-L118)), that should explain why it's working after you remove the schema
If you try produce with a client which support passing schema, Pulsar SQL should work as expected.
Actually here's a handy cli tool to ingest NYC taxi data to a Pulsar cluster with Schema which make it easy to test Pulsar SQL: https://github.com/streamnative/examples/tree/master/nyctaxi/taxidata , the schema: https://github.com/streamnative/examples/blob/master/nyctaxi/taxidata/pkg/types/yellow.go
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] MarvinCai edited a comment on issue #12284: Pulsar SQL BufferUnderflowException after upgrading from 2.7.0 to 2.8.1
Posted by GitBox <gi...@apache.org>.
MarvinCai edited a comment on issue #12284:
URL: https://github.com/apache/pulsar/issues/12284#issuecomment-957368879
sorry for delay, had some discussion with @gaoran10 offline
The Websocket endpoint doesn't support schema, the internal implementation simply produce raw byte([ref](https://github.com/apache/pulsar/blob/master/pulsar-websocket/src/main/java/org/apache/pulsar/websocket/ProducerHandler.java#L73))
And when the Pulsar topic has schema, Pulsar SQL try to ready the message and will expect a schema, while there'll be no schema on the message, causing NPE.
But when we remove the schema, Pulsar SQL will know that and just use a Byte schema by default([ref](https://github.com/apache/pulsar/blob/master/pulsar-sql/presto-pulsar/src/main/java/org/apache/pulsar/sql/presto/PulsarSplitManager.java#L117-L118)), that should explain why it's working after you remove the schema
If you try produce with a client which support passing schema, Pulsar SQL should work as expected.
Actually here's a handy cli tool to ingest NYC taxi data to a Pulsar cluster with Schema which make it easy to test Pulsar SQL: https://github.com/streamnative/examples/tree/master/nyctaxi/taxidata , the schema: https://github.com/streamnative/examples/blob/master/nyctaxi/taxidata/pkg/types/yellow.go
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Raven888888 commented on issue #12284: Pulsar SQL BufferUnderflowException after upgrading from 2.7.0 to 2.8.1
Posted by GitBox <gi...@apache.org>.
Raven888888 commented on issue #12284:
URL: https://github.com/apache/pulsar/issues/12284#issuecomment-958588769
@MarvinCai @gaoran10 Thank you for your explanation, clears up a bit. I have further follow-up questions:
1. In pulsar 2.7.0, I too used websocket to publish data (as you pointed out rightfully, data in raw bytes) to pulsar. And pulsar SQL works fine in that version, where I manually upload `string` schema. Is there changes in schema handling from 2.7.x to 2.8.x, that causes pulsar SQL to break?
2. Is it possible to get back pulsar 2.7.0 behaviour in 2.8.x and beyond? I still want to use websocket to publish data (raw bytes), and sets pulsar topic with `string` schema, and able to query using pulsar SQL.
Thanks a lot.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Raven888888 commented on issue #12284: Pulsar SQL BufferUnderflowException after upgrading from 2.7.0 to 2.8.1
Posted by GitBox <gi...@apache.org>.
Raven888888 commented on issue #12284:
URL: https://github.com/apache/pulsar/issues/12284#issuecomment-938536927
Greetings @MarvinCai
`apache-pulsar/bin/pulsar-admin schemas get test`
returns
```
{
"version": 0,
"schemaInfo": {
"name": "test",
"schema": "",
"type": "STRING",
"properties": {}
}
}
```
This is the same schema before the pulsar upgrade process.
Also tried to re-upload schema:
`apache-pulsar/bin/pulsar-admin schemas upload test -f apache-pulsar/conf/schema_example.conf`
same `BufferUnderflowException` error.
Thanks in advance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Raven888888 commented on issue #12284: Pulsar SQL BufferUnderflowException after upgrading from 2.7.0 to 2.8.1
Posted by GitBox <gi...@apache.org>.
Raven888888 commented on issue #12284:
URL: https://github.com/apache/pulsar/issues/12284#issuecomment-943218703
Seeing some more similar issues of pulsar worker in [pulsar-client-go#546](https://github.com/apache/pulsar-client-go/issues/546) and [pulsar#11457](https://github.com/apache/pulsar/issues/11457).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Raven888888 commented on issue #12284: Pulsar SQL BufferUnderflowException after upgrading from 2.7.0 to 2.8.1
Posted by GitBox <gi...@apache.org>.
Raven888888 commented on issue #12284:
URL: https://github.com/apache/pulsar/issues/12284#issuecomment-975013967
Awesome, thank you @gaoran10 and team! 🚀
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Raven888888 commented on issue #12284: Pulsar SQL BufferUnderflowException after upgrading from 2.7.0 to 2.8.1
Posted by GitBox <gi...@apache.org>.
Raven888888 commented on issue #12284:
URL: https://github.com/apache/pulsar/issues/12284#issuecomment-940672006
Hi @MarvinCai have you got any clue?
Update from my side:
I tried deleting its schema,
`apache-pulsar/bin/pulsar-admin schemas delete test`
able to query in pulsar SQL just fine.
Then, tried re-upload schema
`apache-pulsar/bin/pulsar-admin schemas upload test -f apache-pulsar/conf/schema_example.conf`
got the same `BufferUnderflowException` error.
However, I noticed that `GET` schemas now returns
```
{
**"version": 2,**
"schemaInfo": {
"name": "test",
"schema": "",
"type": "STRING",
"properties": {
"key1": "value1"
}
}
}
```
Seems like schema version has been updated from V1 to V2, from pulsar 2.7.0 to 2.8.1 respectively. Is this what causing presto pulsar connection to break?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Raven888888 commented on issue #12284: Pulsar SQL BufferUnderflowException after upgrading from 2.7.0 to 2.8.1
Posted by GitBox <gi...@apache.org>.
Raven888888 commented on issue #12284:
URL: https://github.com/apache/pulsar/issues/12284#issuecomment-956307372
I have broker enabled with websocket service.
```webSocketServiceEnabled=true```
I am using python client WS [example](https://pulsar.apache.org/docs/en/client-libraries-websocket/#python) to publish into the topic.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] MarvinCai edited a comment on issue #12284: Pulsar SQL BufferUnderflowException after upgrading from 2.7.0 to 2.8.1
Posted by GitBox <gi...@apache.org>.
MarvinCai edited a comment on issue #12284:
URL: https://github.com/apache/pulsar/issues/12284#issuecomment-957368879
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Raven888888 commented on issue #12284: Pulsar SQL BufferUnderflowException after upgrading from 2.7.0 to 2.8.1
Posted by GitBox <gi...@apache.org>.
Raven888888 commented on issue #12284:
URL: https://github.com/apache/pulsar/issues/12284#issuecomment-938541864
Also, ref: [link](https://github.com/streamnative/pulsar/issues/3131#issuecomment-936742398)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Raven888888 edited a comment on issue #12284: Pulsar SQL BufferUnderflowException after upgrading from 2.7.0 to 2.8.1
Posted by GitBox <gi...@apache.org>.
Raven888888 edited a comment on issue #12284:
URL: https://github.com/apache/pulsar/issues/12284#issuecomment-940672006
Hi @MarvinCai have you got any clue?
Update from my side:
I tried deleting its schema,
`apache-pulsar/bin/pulsar-admin schemas delete test`
able to query in pulsar SQL just fine.
Then, tried re-upload schema
`apache-pulsar/bin/pulsar-admin schemas upload test -f apache-pulsar/conf/schema_example.conf`
got the same `BufferUnderflowException` error.
However, I noticed that `GET` schemas now returns
```
{
**"version": 2,**
"schemaInfo": {
"name": "test",
"schema": "",
"type": "STRING",
"properties": {
"key1": "value1"
}
}
}
```
Seems like schema version has been updated from V0 to V2, from pulsar 2.7.0 to 2.8.1 respectively. Is this what causing presto pulsar connection to break?
**PS**: Apologies, version is just related to how many times I update the topic schema. Unrelated to the issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Raven888888 edited a comment on issue #12284: Pulsar SQL BufferUnderflowException after upgrading from 2.7.0 to 2.8.1
Posted by GitBox <gi...@apache.org>.
Raven888888 edited a comment on issue #12284:
URL: https://github.com/apache/pulsar/issues/12284#issuecomment-940672006
Hi @MarvinCai have you got any clue?
Update from my side:
I tried deleting its schema,
`apache-pulsar/bin/pulsar-admin schemas delete test`
able to query in pulsar SQL just fine.
Then, tried re-upload schema
`apache-pulsar/bin/pulsar-admin schemas upload test -f apache-pulsar/conf/schema_example.conf`
got the same `BufferUnderflowException` error.
However, I noticed that `GET` schemas now returns
```
{
**"version": 2,**
"schemaInfo": {
"name": "test",
"schema": "",
"type": "STRING",
"properties": {
"key1": "value1"
}
}
}
```
Seems like schema version has been updated from V0 to V2, from pulsar 2.7.0 to 2.8.1 respectively. Is this what causing presto pulsar connection to break?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Raven888888 commented on issue #12284: Pulsar SQL BufferUnderflowException after upgrading from 2.7.0 to 2.8.1
Posted by GitBox <gi...@apache.org>.
Raven888888 commented on issue #12284:
URL: https://github.com/apache/pulsar/issues/12284#issuecomment-958588769
@MarvinCai @gaoran10 Thank you for your explanation, clears up a bit. I have further follow-up questions:
1. In pulsar 2.7.0, I too used websocket to publish data (as you pointed out rightfully, data in raw bytes) to pulsar. And pulsar SQL works fine in that version, where I manually upload `string` schema. Is there changes in schema handling from 2.7.x to 2.8.x, that causes pulsar SQL to break?
2. Is it possible to get back pulsar 2.7.0 behaviour in 2.8.x and beyond? I still want to use websocket to publish data (raw bytes), and sets pulsar topic with `string` schema, and able to query using pulsar SQL.
Thanks a lot.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org