You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2021/10/22 14:44:10 UTC
[GitHub] [pulsar] xl4hub opened a new issue #12465: Pulsar SQL NullPointerException when select a row produced by go client with schema
xl4hub opened a new issue #12465:
URL: https://github.com/apache/pulsar/issues/12465
**Expected:**
Using the go client to send a message with a schema I should be able to use Pulsar SQL to read the message
**Actual:**
When I try this:
```
presto> select * from pulsar."public/default".simple;
Query 20211022_073804_00008_5x42n, FAILED, 1 node
http://pulsar-broker:8081/ui/query.html?20211022_073804_00008_5x42n
Splits: 18 total, 1 done (5.56%)
CPU Time: 0.0s total, 0 rows/s, 0B/s, 2% active
Per Node: 0.0 parallelism, 0 rows/s, 0B/s
Parallelism: 0.0
Peak Memory: 0B
0:00 [0 rows, 0B] [0 rows/s, 0B/s]
Query 20211022_073804_00008_5x42n failed: Internal error
java.lang.NullPointerException
at org.apache.pulsar.sql.presto.PulsarDispatchingRowDecoderFactory.createDecoderFactory(PulsarDispatchingRowDecoderFactory.java:69)
at org.apache.pulsar.sql.presto.PulsarDispatchingRowDecoderFactory.createRowDecoder(PulsarDispatchingRowDecoderFactory.java:58)
at org.apache.pulsar.sql.presto.PulsarRecordCursor.advanceNextPosition(PulsarRecordCursor.java:546)
at io.prestosql.spi.connector.RecordPageSource.getNextPage(RecordPageSource.java:90)
at io.prestosql.operator.TableScanOperator.getOutput(TableScanOperator.java:302)
at io.prestosql.operator.Driver.processInternal(Driver.java:379)
at io.prestosql.operator.Driver.lambda$processFor$8(Driver.java:283)
at io.prestosql.operator.Driver.tryWithLock(Driver.java:675)
at io.prestosql.operator.Driver.processFor(Driver.java:276)
at io.prestosql.execution.SqlTaskExecution$DriverSplitRunner.processFor(SqlTaskExecution.java:1075)
at io.prestosql.execution.executor.PrioritizedSplitRunner.process(PrioritizedSplitRunner.java:163)
at io.prestosql.execution.executor.TaskExecutor$TaskRunner.run(TaskExecutor.java:484)
at io.prestosql.$gen.Presto_332__testversion____20211022_052918_2.run(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
```
I built pulsar master head and 2.8.1 both have this problem. I am using the go client version 0.6.0.
The problem is in org.apache.pulsar.sql.presto.PulsarRecordCursor and appears to originate in getBytesSchemaInfo()
called here:
`482: SchemaInfo schemaInfo = getBytesSchemaInfo(pulsarSplit.getSchemaType(), pulsarSplit.getSchemaName());
`
I added a log message and can see:
pulsarSplit.getSchemaType() returns JSON
pulsarSplit.getSchemaName() returns "public/default"
getBytesSchemaInfo() doesn't appear to handle a schema type of JSON - it returns null as schemaInfo.
This is then dererenced at line 493 and hence the NullPointerException.
This looks like a simple oversight but it is not clear to me how it should be fixed.
This is not a problem with 2.7.2 but I moved to 2.8.1 in that hope that avro schemas with arrays are finally supported. Unfortunately things are worse.
My go code reads a very simple schema from a file and creates the producer like this:
options.Topic = "simple"
options.Schema = pulsar.NewJSONSchema(string(schema), nil)
producer, err := client.CreateProducer(options)
schema is:
```
{
"namespace" : "test",
"name" : "simple",
"type" : "record",
"fields" : [
{"name": "str", "type": "string"},
{"name": "num", "type": "int"}
]
}
```
Send one message {"str":"abcdef", "num":123} and another long term bug is apparent: an sql select of the topic returns 0 rows.
Send another message and sql select gives the NullPointerException.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Raven888888 commented on issue #12465: Pulsar SQL NullPointerException when select a row produced by go client with schema
Posted by GitBox <gi...@apache.org>.
Raven888888 commented on issue #12465:
URL: https://github.com/apache/pulsar/issues/12465#issuecomment-957084389
Related issue: #12284
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] MarvinCai commented on issue #12465: Pulsar SQL NullPointerException when select a row produced by go client with schema
Posted by GitBox <gi...@apache.org>.
MarvinCai commented on issue #12465:
URL: https://github.com/apache/pulsar/issues/12465#issuecomment-958605052
for the first issue, it's not a bug, the behavior is controlled by a flag, please refer this: https://github.com/apache/pulsar/issues/12209, will update the document about it.
for second issue, I also tried go client, seems the schema version is just null, so Pulsar Presto connector is not able to locate schema(the connector already knows there is a schema for the topic by schema admin endpoint, so it expect a schema from message, but it's not there)
need to dig deeper to see why schema is missing from message.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] github-actions[bot] commented on issue #12465: Pulsar SQL NullPointerException when select a row produced by go client with schema
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #12465:
URL: https://github.com/apache/pulsar/issues/12465#issuecomment-1052938495
The issue had no activity for 30 days, mark with Stale label.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Raven888888 commented on issue #12465: Pulsar SQL NullPointerException when select a row produced by go client with schema
Posted by GitBox <gi...@apache.org>.
Raven888888 commented on issue #12465:
URL: https://github.com/apache/pulsar/issues/12465#issuecomment-957084389
Related issue: #12284
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] longtengz commented on issue #12465: Pulsar SQL NullPointerException when select a row produced by go client with schema
Posted by GitBox <gi...@apache.org>.
longtengz commented on issue #12465:
URL: https://github.com/apache/pulsar/issues/12465#issuecomment-985972267
@Raven888888 are you able to work around this? And what version of golang client are you using?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Raven888888 commented on issue #12465: Pulsar SQL NullPointerException when select a row produced by go client with schema
Posted by GitBox <gi...@apache.org>.
Raven888888 commented on issue #12465:
URL: https://github.com/apache/pulsar/issues/12465#issuecomment-957084389
Related issue: #12284
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] halfvector commented on issue #12465: Pulsar SQL NullPointerException when select a row produced by go client with schema
Posted by GitBox <gi...@apache.org>.
halfvector commented on issue #12465:
URL: https://github.com/apache/pulsar/issues/12465#issuecomment-1076895842
@Raven888888 Just tried on a local build of `v2.10.0-candidate-4` and the bug does not reoccur. So looks like its fixed in the upcoming release.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] MarvinCai commented on issue #12465: Pulsar SQL NullPointerException when select a row produced by go client with schema
Posted by GitBox <gi...@apache.org>.
MarvinCai commented on issue #12465:
URL: https://github.com/apache/pulsar/issues/12465#issuecomment-958605052
for the first issue, it's not a bug, the behavior is controlled by a flag, please refer this: https://github.com/apache/pulsar/issues/12209, will update the document about it.
for second issue, I also tried go client, seems the schema version is just null, so Pulsar Presto connector is not able to locate schema(the connector already knows there is a schema for the topic by schema admin endpoint, so it expect a schema from message, but it's not there)
need to dig deeper to see why schema is missing from message.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] MarvinCai commented on issue #12465: Pulsar SQL NullPointerException when select a row produced by go client with schema
Posted by GitBox <gi...@apache.org>.
MarvinCai commented on issue #12465:
URL: https://github.com/apache/pulsar/issues/12465#issuecomment-958605052
for the first issue, it's not a bug, the behavior is controlled by a flag, please refer this: https://github.com/apache/pulsar/issues/12209, will update the document about it.
for second issue, I also tried go client, seems the schema version is just null, so Pulsar Presto connector is not able to locate schema(the connector already knows there is a schema for the topic by schema admin endpoint, so it expect a schema from message, but it's not there)
need to dig deeper to see why schema is missing from message.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] MarvinCai commented on issue #12465: Pulsar SQL NullPointerException when select a row produced by go client with schema
Posted by GitBox <gi...@apache.org>.
MarvinCai commented on issue #12465:
URL: https://github.com/apache/pulsar/issues/12465#issuecomment-958605052
for the first issue, it's not a bug, the behavior is controlled by a flag, please refer this: https://github.com/apache/pulsar/issues/12209, will update the document about it.
for second issue, I also tried go client, seems the schema version is just null, so Pulsar Presto connector is not able to locate schema(the connector already knows there is a schema for the topic by schema admin endpoint, so it expect a schema from message, but it's not there)
need to dig deeper to see why schema is missing from message.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Raven888888 commented on issue #12465: Pulsar SQL NullPointerException when select a row produced by go client with schema
Posted by GitBox <gi...@apache.org>.
Raven888888 commented on issue #12465:
URL: https://github.com/apache/pulsar/issues/12465#issuecomment-1075800758
I was going to try Pulsar 2.9.1 to see if it gets resolved, hearing from @halfvector puts a stop for me...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Raven888888 commented on issue #12465: Pulsar SQL NullPointerException when select a row produced by go client with schema
Posted by GitBox <gi...@apache.org>.
Raven888888 commented on issue #12465:
URL: https://github.com/apache/pulsar/issues/12465#issuecomment-1076967891
@halfvector great to hear! I'll definitely try out v2.10 release then. Thanks a bucket!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] halfvector commented on issue #12465: Pulsar SQL NullPointerException when select a row produced by go client with schema
Posted by GitBox <gi...@apache.org>.
halfvector commented on issue #12465:
URL: https://github.com/apache/pulsar/issues/12465#issuecomment-1075732502
Ran into this same bug using Pulsar 2.9.1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org