You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by "bpereto (via GitHub)" <gi...@apache.org> on 2023/05/07 11:14:16 UTC
[GitHub] [pulsar] bpereto opened a new issue, #20247: [Bug] Pulsar IO JDBC Clickhouse and Protobuf native with "ByteString" throws exception
bpereto opened a new issue, #20247:
URL: https://github.com/apache/pulsar/issues/20247
### Search before asking
- [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.
### Version
Rocky Linux 8
Pulsar 2.11.1
### Minimal reproduce step
- Add ProtobufNative Schema to topic with "bytes" field.
- Use Pulsar IO JDBC Clickhouse Sink
### What did you expect to see?
Event is sinked/inserted to clickhouse.
### What did you see instead?
```
2023-05-07T13:06:54,039+0200 [pool-2-thread-1] ERROR org.apache.pulsar.io.jdbc.JdbcAbstractSink - Update count 0 not match total number of records 1005
2023-05-07T13:06:54,098+0200 [pool-2-thread-1] ERROR org.apache.pulsar.io.jdbc.JdbcAbstractSink - Got exception
java.lang.Exception: Not support value type, need to add it. class com.google.protobuf.ByteString$LiteralByteString
at org.apache.pulsar.io.jdbc.BaseJdbcAutoSchemaSink.setColumnValue(BaseJdbcAutoSchemaSink.java:186) ~[pulsar-io-jdbc-core-2.11.1.jar:?]
at org.apache.pulsar.io.jdbc.BaseJdbcAutoSchemaSink.bindValue(BaseJdbcAutoSchemaSink.java:81) ~[pulsar-io-jdbc-core-2.11.1.jar:?]
at org.apache.pulsar.io.jdbc.JdbcAbstractSink.flush(JdbcAbstractSink.java:247) ~[pulsar-io-jdbc-core-2.11.1.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) ~[?:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
at java.lang.Thread.run(Thread.java:833) ~[?:?]
```
### Anything else?
As i'm aware of, clickhouse supports "ByteString": https://clickhouse.com/docs/en/sql-reference/data-types/fixedstring
Maybe adding `.setBytes` does the job, but i'm not aware how to build the pulsar io `.nar` files to test it. How do I build the connectors?
https://github.com/apache/pulsar/blob/master/pulsar-io/jdbc/core/src/main/java/org/apache/pulsar/io/jdbc/BaseJdbcAutoSchemaSink.java#L170
```
} else if (value instanceof ByteArray) {
statement.setBytes(index, (ByteArray) value);
```
### Are you willing to submit a PR?
- [X] I'm willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] tisonkun closed issue #20247: [Bug] Pulsar IO JDBC Clickhouse and Protobuf native with "ByteString" throws exception
Posted by "tisonkun (via GitHub)" <gi...@apache.org>.
tisonkun closed issue #20247: [Bug] Pulsar IO JDBC Clickhouse and Protobuf native with "ByteString" throws exception
URL: https://github.com/apache/pulsar/issues/20247
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] tisonkun commented on issue #20247: [Bug] Pulsar IO JDBC Clickhouse and Protobuf native with "ByteString" throws exception
Posted by "tisonkun (via GitHub)" <gi...@apache.org>.
tisonkun commented on issue #20247:
URL: https://github.com/apache/pulsar/issues/20247#issuecomment-1538017627
I agree with your change. However, you need to add protobuf-java as dependencies and we should investigate a bit about whether it cause deps issues.
I saw you tick the " I'm willing to submit a PR!" box. Please feel free to ping me once you submit a PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] bpereto commented on issue #20247: [Bug] Pulsar IO JDBC Clickhouse and Protobuf native with "ByteString" throws exception
Posted by "bpereto (via GitHub)" <gi...@apache.org>.
bpereto commented on issue #20247:
URL: https://github.com/apache/pulsar/issues/20247#issuecomment-1538477433
I submitted the pull request: https://github.com/apache/pulsar/pull/20259
I tested it with clickhouse. Had to enable bytes support via jdbc url:
```
jdbc:clickhouse://192.168.1.50:8123/default?use_binary_string=true
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] tisonkun commented on issue #20247: [Bug] Pulsar IO JDBC Clickhouse and Protobuf native with "ByteString" throws exception
Posted by "tisonkun (via GitHub)" <gi...@apache.org>.
tisonkun commented on issue #20247:
URL: https://github.com/apache/pulsar/issues/20247#issuecomment-1538006671
> How do I build the connectors
Run:
```
./mvnw clean install -DskipITs -DskipTests -Dlicense.skip=true -Drat.skip=true -Dcheckstyle.skip=true
```
And the nar file will be located at `distribution/io/target/apache-pulsar-io-connectors-3.1.0-SNAPSHOT-bin` then you can test it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org