You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@pulsar.apache.org by Apache Pulsar Slack <ap...@gmail.com> on 2020/01/08 09:11:03 UTC
Slack digest for #general - 2020-01-08
2020-01-07 09:14:38 UTC - Lykant: @Lykant has joined the channel
----
2020-01-07 14:17:33 UTC - Alexandre DUVAL: There is a simple way to run pulsar standalone without function worker ?
----
2020-01-07 15:11:13 UTC - Sijie Guo: `bin/pulsar standalone -nfw`
----
2020-01-07 15:12:05 UTC - vikash: Hello All,
i am using Apache Pulsar v2.4.2. and running as Standalone.
i am getting the NullPointerException checkNotNull exception in custom connector and connector write() method not getting trigger while sending message to pulsar.
1.I have taken refrence of pulsar-io jdbc built-in connector code and changed driver connection to Apache Ignite JDBC.
2.Created AVRO schema for the pulsar topic and uploaded.
schema like ./connectors/schema_example.conf:
{
"type": "AVRO",
"schema": "{\"type\":\"record\",\"name\":\"Test\",\"fields\":[{\"name\":\"id\",\"type\":[\"null\",\"int\"]},{\"name\":\"name\",\"type\":[\"null\",\"string\"]}]}",
"properties": {}
}
Uploaded Schema:
/apache-pulsar-2.4.2/bin#./pulsar-admin schemas upload myjdbc --filename ./connectors/schema_example.conf
3.created connector on pulsar:
/apache-pulsar-2.4.2/bin#./pulsar-admin sinks localrun --archive ./connectors/pulsar-io-jdbc-2.4.2.nar --inputs myjdbc --name myjdbc --sink-config-file ./connectors/pulsar-io-jdbc-ignite-sink.yaml
4.Open() method triggering and insert statement preparing successfully.
5.If topic already not exist and first time create command executing on pulsar no exception.
but onces sending message to pulsar from producer.sendAsync(byteArrayMessage)
getting the NullPointerException checkNotNull exception
and Write() method not getting call/triggering
here is the exception and log message:
=================================================================================
root@biz-VBox:/opt/pulsar/apache-pulsar-2.4.2/bin#
root@biz-VBox:/opt/pulsar/apache-pulsar-2.4.2/bin#
root@biz-VBox:/opt/pulsar/apache-pulsar-2.4.2/bin# ./pulsar-admin sinks localrun --archive ./connectors/pulsar-io-jdbc-2.4.2.nar --inputs myjdbc --name myjdbc --sink-config-file ./connectors/pulsar-io-jdbc-ignite-sink.yaml
19:48:28.849 [main] INFO <http://org.apache.pulsar.functions.utils.io|org.apache.pulsar.functions.utils.io>.ConnectorUtils - Searching for connectors in /opt/pulsar/apache-pulsar-2.4.2/connectors
19:48:29.003 [main] INFO <http://org.apache.pulsar.functions.utils.io|org.apache.pulsar.functions.utils.io>.ConnectorUtils - Found connector ConnectorDefinition(name=ignite, description=Ignite sink connector, sourceClass=null, sinkClass=<http://org.apache.pulsar.io|org.apache.pulsar.io>.ignite.IgniteBytesSink) from /opt/pulsar/apache-pulsar-2.4.2/connectors/pulsar-io-ignite-2.4.1.nar
19:48:29.612 [main] INFO <http://org.apache.pulsar.functions.utils.io|org.apache.pulsar.functions.utils.io>.ConnectorUtils - Found connector ConnectorDefinition(name=ignitetopictest, description=Jdbc Ignite sink, sourceClass=null, sinkClass=<http://org.apache.pulsar.io|org.apache.pulsar.io>.jdbc.ignite.JdbcAutoSchemaSink) from /opt/pulsar/apache-pulsar-2.4.2/connectors/pulsar-io-jdbc-ignite-2.4.1.nar
19:48:30.113 [main] INFO <http://org.apache.pulsar.functions.utils.io|org.apache.pulsar.functions.utils.io>.ConnectorUtils - Found connector ConnectorDefinition(name=jdbc, description=Jdbc sink, sourceClass=null, sinkClass=<http://org.apache.pulsar.io|org.apache.pulsar.io>.jdbc.JdbcAutoSchemaSink) from /opt/pulsar/apache-pulsar-2.4.2/connectors/pulsar-io-jdbc-2.4.1.nar
19:48:30.520 [main] INFO <http://org.apache.pulsar.functions.utils.io|org.apache.pulsar.functions.utils.io>.ConnectorUtils - Found connector ConnectorDefinition(name=myjdbc, description=Jdbc sink, sourceClass=null, sinkClass=<http://org.apache.pulsar.io|org.apache.pulsar.io>.jdbc.JdbcAutoSchemaSink) from /opt/pulsar/apache-pulsar-2.4.2/connectors/pulsar-io-jdbc-2.4.2.nar
19:48:33.026 [main] INFO org.apache.pulsar.functions.runtime.RuntimeSpawner - public/default/myjdbc-0 RuntimeSpawner starting function
19:48:33.029 [main] INFO org.apache.pulsar.functions.runtime.ThreadRuntime - ThreadContainer starting function with instance config InstanceConfig(instanceId=0, functionId=aa384dfc-da74-4e1a-9739-355f5f57aac4, functionVersion=6c9f3846-7a21-4722-a453-a5f563d101f4, functionDetails=tenant: "public"
namespace: "default"
name: "myjdbc"
className: "org.apache.pulsar.functions.api.utils.IdentityFunction"
autoAck: true
parallelism: 1
source {
typeClassName: "org.apache.pulsar.client.api.schema.GenericRecord"
inputSpecs {
key: "myjdbc"
value {
}
}
cleanupSubscription: true
}
sink {
className: "<http://org.apache.pulsar.io|org.apache.pulsar.io>.jdbc.JdbcAutoSchemaSink"
configs: "{\"jdbcUrl\":\"jdbc:ignite:<thin://localhost/>\",\"tableName\":\"ENTITIESTABLEINFO\"}"
typeClassName: "org.apache.pulsar.client.api.schema.GenericRecord"
}
resources {
cpu: 1.0
ram: 1073741824
disk: 10737418240
}
componentType: SINK
, maxBufferedTuples=1024, functionAuthenticationSpec=null, port=43629, clusterName=local)
2020-01-07 19:48:33,952 - INFO - [public/default/myjdbc-0:JdbcAbstractSink@80] - OPEN INVOKE>>>>>>>>
19:48:36.332 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ConnectionPool - [[id: 0x10072908, L:/127.0.0.1:39486 - R:localhost/127.0.0.1:6650]] Connected to server
19:48:36.500 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.PulsarClientImpl - Configuring schema for topic myjdbc : {
"name": "public/default/myjdbc",
"schema": {
"type": "record",
"name": "Test",
"fields": [
{
"name": "id",
"type": [
"null",
"int"
]
},
{
"name": "name",
"type": [
"null",
"string"
]
}
]
},
"type": "AVRO",
"properties": {}
}
19:48:37.141 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.schema.AutoConsumeSchema - Configure topic schema for topic myjdbc : {"type":"record","name":"Test","fields":[{"name":"id","type":["null","int"]},{"name":"name","type":["null","string"]}]}
19:48:37.319 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ConsumerStatsRecorderImpl - Starting Pulsar consumer perf with config: {
"topicNames" : [ "myjdbc" ],
"topicsPattern" : null,
"subscriptionName" : "public/default/myjdbc",
"subscriptionType" : "Shared",
"receiverQueueSize" : 1000,
"acknowledgementsGroupTimeMicros" : 100000,
"negativeAckRedeliveryDelayMicros" : 60000000,
"maxTotalReceiverQueueSizeAcrossPartitions" : 50000,
"consumerName" : null,
"ackTimeoutMillis" : 0,
"tickDurationMillis" : 1000,
"priorityLevel" : 0,
"cryptoFailureAction" : "CONSUME",
"properties" : {
"application" : "pulsar-sink",
"id" : "public/default/myjdbc",
"instance_id" : "0"
},
"readCompacted" : false,
"subscriptionInitialPosition" : "Latest",
"patternAutoDiscoveryPeriod" : 1,
"regexSubscriptionMode" : "PersistentOnly",
"deadLetterPolicy" : null,
"autoUpdatePartitions" : true,
"replicateSubscriptionState" : false,
"resetIncludeHead" : false
}
19:48:37.343 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ConsumerStatsRecorderImpl - Pulsar client config: {
"serviceUrl" : "<pulsar://localhost:6650>",
"authPluginClassName" : null,
"authParams" : null,
"operationTimeoutMs" : 30000,
"statsIntervalSeconds" : 60,
"numIoThreads" : 1,
"numListenerThreads" : 1,
"connectionsPerBroker" : 1,
"useTcpNoDelay" : true,
"useTls" : false,
"tlsTrustCertsFilePath" : null,
"tlsAllowInsecureConnection" : false,
"tlsHostnameVerificationEnable" : false,
"concurrentLookupRequest" : 5000,
"maxLookupRequest" : 50000,
"maxNumberOfRejectedRequestPerConnection" : 50,
"keepAliveIntervalSeconds" : 30,
"connectionTimeoutMs" : 10000,
"requestTimeoutMs" : 60000,
"initialBackoffIntervalNanos" : 100000000,
"maxBackoffIntervalNanos" : 60000000000
}
19:48:37.434 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ConnectionPool - [[id: 0x557fb218, L:/127.0.0.1:39488 - R:localhost/127.0.0.1:6650]] Connected to server
19:48:37.437 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ClientCnx - [id: 0x557fb218, L:/127.0.0.1:39488 - R:localhost/127.0.0.1:6650] Connected through proxy to target broker at 192.168.1.97:6650
19:48:37.445 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ConsumerImpl - [myjdbc][public/default/myjdbc] Subscribing to topic on cnx [id: 0x557fb218, L:/127.0.0.1:39488 - R:localhost/127.0.0.1:6650]
19:48:37.503 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ConsumerImpl - [myjdbc][public/default/myjdbc] Subscribed to topic on localhost/127.0.0.1:6650 -- consumer: 0
19:48:37.557 [pulsar-client-io-1-1] INFO com.scurrilous.circe.checksum.Crc32cIntChecksum - SSE4.2 CRC32C provider initialized
19:48:37.715 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ConsumerImpl - [myjdbc] [public/default/myjdbc] Closed consumer
19:48:37.942 [main] INFO org.apache.pulsar.functions.LocalRunner - RuntimeSpawner quit because of
java.lang.NullPointerException: null
at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:877) ~[com.google.guava-guava-25.1-jre.jar:?]
at com.google.common.cache.LocalCache.get(LocalCache.java:3950) ~[com.google.guava-guava-25.1-jre.jar:?]
at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3973) ~[com.google.guava-guava-25.1-jre.jar:?]
at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4957) ~[com.google.guava-guava-25.1-jre.jar:?]
at org.apache.pulsar.client.impl.schema.StructSchema.decode(StructSchema.java:94) ~[org.apache.pulsar-pulsar-client-original-2.4.2.jar:2.4.2]
at org.apache.pulsar.client.impl.schema.AutoConsumeSchema.decode(AutoConsumeSchema.java:72) ~[org.apache.pulsar-pulsar-client-original-2.4.2.jar:2.4.2]
at org.apache.pulsar.client.impl.schema.AutoConsumeSchema.decode(AutoConsumeSchema.java:36) ~[org.apache.pulsar-pulsar-client-original-2.4.2.jar:2.4.2]
at org.apache.pulsar.client.api.Schema.decode(Schema.java:97) ~[org.apache.pulsar-pulsar-client-api-2.4.2.jar:2.4.2]
at org.apache.pulsar.client.impl.MessageImpl.getValue(MessageImpl.java:268) ~[org.apache.pulsar-pulsar-client-original-2.4.2.jar:2.4.2]
at org.apache.pulsar.functions.source.PulsarRecord.getValue(PulsarRecord.java:74) ~[org.apache.pulsar-pulsar-functions-instance-2.4.2.jar:2.4.2]
at org.apache.pulsar.functions.instance.JavaInstanceRunnable.readInput(JavaInstanceRunnable.java:473) ~[org.apache.pulsar-pulsar-functions-instance-2.4.2.jar:2.4.2]
at org.apache.pulsar.functions.instance.JavaInstanceRunnable.run(JavaInstanceRunnable.java:246) ~[org.apache.pulsar-pulsar-functions-instance-2.4.2.jar:2.4.2]
at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_232]
19:49:03.105 [function-timer-thread-5-1] ERROR org.apache.pulsar.functions.runtime.RuntimeSpawner - public/default/myjdbc-java.lang.NullPointerException Function Container is dead with exception.. restarting
----
2020-01-07 15:13:23 UTC - Alexandre DUVAL: Ty!
----
2020-01-07 15:45:57 UTC - Roman Popenov: Looks like some of the defined fields are null when the object is being deserialized and sinked as a record
----
2020-01-07 15:50:21 UTC - Roman Popenov: When you serialize the object from the source and write it to a topic, it looks like the value is `null`
----
2020-01-07 16:31:29 UTC - Jocelyne Bergey: @Jocelyne Bergey has joined the channel
----
2020-01-07 16:39:06 UTC - Lukas Chripko: did anybody succesfully build install/build pulsar-client on alpine? dockerfile example working with alpine?
----
2020-01-07 16:47:42 UTC - juraj: u can create a "Snippet" post here on slack to avoid flooding the channel
----
2020-01-07 16:50:52 UTC - juraj: seach for the keyword `build` on this slack, u will find some stuff
----
2020-01-07 19:55:03 UTC - Martin Garnier: @Martin Garnier has joined the channel
----
2020-01-07 20:19:18 UTC - Mathieu Druart: Hi, we try to use the State storage inside a Pulsar Function but we always have this error : `java.lang.IllegalStateException: State is not enabled.`
----
2020-01-07 20:20:28 UTC - Mathieu Druart: The Function is run in a k8s cluster deployed with the mini Helm configuration
----
2020-01-07 20:22:30 UTC - Mathieu Druart: @Sijie Guo we changed our code to use a JSONSchema instead of the Avro Schema and now it works
----
2020-01-07 20:24:36 UTC - Mathieu Druart: Is there something to do to enable State ?
----
2020-01-07 21:02:42 UTC - Pedro Cardoso: Yes, you need to add
```extraServerComponents: "org.apache.bookkeeper.stream.server.StreamStorageLifecycleComponent"```
Under the bookkeeper.configData section, see <https://github.com/apache/pulsar/issues/5687>
+1 : Mathieu Druart
----
2020-01-07 21:03:51 UTC - Pedro Cardoso: I believe is fixed on the 2.5.0 release candidate
----
2020-01-07 21:36:08 UTC - Martin Garnier: awesome, thanks Pedro :)
----
2020-01-07 22:08:44 UTC - Eron Wright: Would it make sense to have Pulsar SQL listed as a connector in Presto documentation? Just curious.
<https://prestodb.io/docs/current/connector.html>
----
2020-01-07 22:41:10 UTC - Roman Popenov: So the functions are packaged in a `jar`, but the connectors are in a `nar` ?
----
2020-01-07 22:42:03 UTC - Roman Popenov: Can I specify a `--jar` for Pulsar IO?
----
2020-01-07 22:42:49 UTC - Roman Popenov: or point it to a jar with the `-a`?
----
2020-01-07 22:43:27 UTC - Roman Popenov: ```Usage: pulsar-admin sources [options] [command] [command options]
Commands:
create Submit a Pulsar IO source connector to run in a Pulsar cluster
Usage: create [options]
Options:
-a, --archive
The path to the NAR archive for the Source. It also supports
url-path [http/https/file (file protocol assumes that file already exists on
worker host)] from which worker can download the package.```
----
2020-01-07 23:52:54 UTC - Roman Popenov: JAR also seems to be working just fine as well
----
2020-01-08 00:01:24 UTC - Roman Popenov: I keep getting:
```00:00:22.491 [function-timer-thread-5-1] ERROR org.apache.pulsar.functions.runtime.RuntimeSpawner - test-tenant/test-ns/s3-org.apache.avro.AvroRuntimeException: avro.shaded.com.google.common.util.concurrent.UncheckedExecutionException: org.apache.avro.AvroRuntimeException: Can't find element type of Collection Function Container is dead with exception.. restarting```
----
2020-01-08 00:17:50 UTC - Addison Higham: :thinking_face: has anyone tried to write to s3 using the pulsar IO hdfs sink and s3a library? Quick read of the code makes me think it should work is likely to produce duplicates and also would likely cache to local disk... also not sure how in S3 it would chunk files
----
2020-01-08 00:21:13 UTC - Roman Popenov: s3a library works like a charm, I used it in a Pulsar Function
----
2020-01-08 00:52:26 UTC - Roman Popenov: Guess this is related to <https://github.com/streamnative/pulsar/issues/222>
----
2020-01-08 01:31:07 UTC - juraj: how far back does the broker search for a messageid to fulfill a cumulative acknowledgement ? i'm trying after ~25 minutes (~1k msgs), but the ack call is basically ignored (persistent topic, deduplication enabled)
----
2020-01-08 05:53:19 UTC - vikash: Hello ,I have struck in issue where i want to know whats going on from server side
----
2020-01-08 05:53:31 UTC - vikash: how to start pulsar standalone as debug mode
----
2020-01-08 05:53:47 UTC - vikash: bin/pulsar standalone --debug not working
----
2020-01-08 08:01:21 UTC - xiaolong.ran: You can configure it in `conf / log4j.yml`, as follows:
``` - name: org.apache.pulsar.broker.service
level: debug
additivity: false
AppenderRef:
- ref: Console```
----