You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@pulsar.apache.org by Apache Pulsar Slack <ap...@gmail.com> on 2020/04/17 09:11:04 UTC
Slack digest for #general - 2020-04-17
2020-04-16 09:55:02 UTC - Adelina Brask: I believe you are right....but I don't know how to fix it...
----
2020-04-16 09:55:14 UTC - Adelina Brask:
----
2020-04-16 09:55:47 UTC - Adelina Brask: `/opt/apache-pulsar-2.5.0/bin/pulsar-admin sinks create --name elastic --sink-config-file /opt/apache-pulsar-2.5.0/conf/elastic_sink.yml --inputs public/default/test --archive /opt/apache-pulsar-2.5.0/connectors/pulsar-io-elastic-search-2.5.0.nar`
----
2020-04-16 09:56:01 UTC - Adelina Brask: In the logs sink starts with schema "BYTES"
----
2020-04-16 09:56:23 UTC - Adelina Brask: then I create the sources
----
2020-04-16 09:56:28 UTC - Adelina Brask: `/opt/apache-pulsar-2.5.0/bin/pulsar-admin sources create --name netty --source-config-file /opt/apache-pulsar-2.5.0/conf/netty-source-config.yml --destination-topic-name public/default/test --archive /opt/apache-pulsar-2.5.0/connectors/pulsar-io-netty-2.5.0.nar`
----
2020-04-16 09:57:12 UTC - Adelina Brask: in the logs, it inherits schema type from Elastic (BYTES) and theoretically starts, but 10999 is not bind
----
2020-04-16 09:57:29 UTC - Adelina Brask: I can't find any schema related logs ....
----
2020-04-16 09:59:11 UTC - Adelina Brask: If I start them in the opposite way 1. SOURCE starts with no schema, port is bind correctly. 2. SINK fails to start - maybe cause it has schema BYTES and
----
2020-04-16 09:59:38 UTC - Adelina Brask: what I don't understand is, why both works in `localrun` but not with `create`
----
2020-04-16 10:00:07 UTC - Adelina Brask: isn't the same schemas policy they are using ?
----
2020-04-16 10:01:19 UTC - Adelina Brask: So Presto is started in cluster mode
----
2020-04-16 10:01:24 UTC - Adelina Brask: `[root@clstpulsar01 apache-pulsar-2.5.0]# bin/pulsar sql`
`presto> SELECT * FROM system.runtime.nodes;`
`node_id | http_uri | node_version | coordinator | state`
`--------------------------------------+---------------------------+--------------+-------------+--------`
`ffffffff-gggg-gggg-gggg-ffffffffffff | <http://10.220.37.193:8081> | testversion | false | active`
`ffffffff-ffff-ffff-ffff-ffffffffffff | <http://10.220.37.191:8081> | testversion | true | active`
`ffffffff-eeee-eeee-eeee-ffffffffffff | <http://10.220.37.192:8081> | testversion | false | active`
`(3 rows)`
`Query 20200416_100101_00007_fye7x, FINISHED, 2 nodes`
`Splits: 17 total, 17 done (100.00%)`
`0:00 [3 rows, 237B] [12 rows/s, 989B/s]`
----
2020-04-16 10:01:37 UTC - Adelina Brask: according to this: <http://pulsar.apache.org/docs/en/sql-deployment-configurations/>
----
2020-04-16 10:01:46 UTC - Adelina Brask: But when I query a table I get:
----
2020-04-16 10:02:08 UTC - Adelina Brask: `Query 20200416_100158_00008_fye7x failed: <http://java.io|java.io>.IOException: Failed to initialize ledger manager factory`
----
2020-04-16 10:04:10 UTC - Adelina Brask:
----
2020-04-16 10:09:26 UTC - Adelina Brask: I feel that I don't get enough logs, or am I missing something?
----
2020-04-16 10:10:09 UTC - Adelina Brask: I feel that I don't get enough logs, or am I missing something? Is there a configuration for Presto logging ?
----
2020-04-16 10:12:38 UTC - Adelina Brask: ok..I found the --debug tag :slightly_smiling_face:
----
2020-04-16 10:13:25 UTC - Adelina Brask: `Query 20200416_101214_00000_czgjw failed: <http://java.io|java.io>.IOException: Failed to initialize ledger manager factory`
`java.lang.RuntimeException: <http://java.io|java.io>.IOException: Failed to initialize ledger manager factory`
`at org.apache.pulsar.sql.presto.PulsarSplitManager.getSplits(PulsarSplitManager.java:134)`
`at com.facebook.presto.split.SplitManager.getSplits(SplitManager.java:64)`
`at com.facebook.presto.sql.planner.DistributedExecutionPlanner$Visitor.visitTableScan(DistributedExecutionPlanner.java:146)`
`at com.facebook.presto.sql.planner.DistributedExecutionPlanner$Visitor.visitTableScan(DistributedExecutionPlanner.java:122)`
`at com.facebook.presto.sql.planner.plan.TableScanNode.accept(TableScanNode.java:136)`
`at com.facebook.presto.sql.planner.DistributedExecutionPlanner.doPlan(DistributedExecutionPlanner.java:108)`
`at com.facebook.presto.sql.planner.DistributedExecutionPlanner.doPlan(DistributedExecutionPlanner.java:113)`
`at com.facebook.presto.sql.planner.DistributedExecutionPlanner.plan(DistributedExecutionPlanner.java:85)`
`at com.facebook.presto.execution.SqlQueryExecution.planDistribution(SqlQueryExecution.java:385)`
`at com.facebook.presto.execution.SqlQueryExecution.start(SqlQueryExecution.java:287)`
`at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)`
`at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)`
`at java.lang.Thread.run(Thread.java:748)`
`Caused by: <http://java.io|java.io>.IOException: Failed to initialize ledger manager factory`
`at org.apache.bookkeeper.client.BookKeeper.<init>(BookKeeper.java:520)`
`at org.apache.bookkeeper.client.BookKeeper.<init>(BookKeeper.java:368)`
`at org.apache.bookkeeper.mledger.impl.ManagedLedgerFactoryImpl$DefaultBkFactory.<init>(ManagedLedgerFactoryImpl.java:183)`
`at org.apache.bookkeeper.mledger.impl.ManagedLedgerFactoryImpl.<init>(ManagedLedgerFactoryImpl.java:122)`
`at org.apache.bookkeeper.mledger.impl.ManagedLedgerFactoryImpl.<init>(ManagedLedgerFactoryImpl.java:114)`
`at org.apache.pulsar.sql.presto.PulsarConnectorCache.initManagedLedgerFactory(PulsarConnectorCache.java:108)`
`at org.apache.pulsar.sql.presto.PulsarConnectorCache.<init>(PulsarConnectorCache.java:66)`
`at org.apache.pulsar.sql.presto.PulsarConnectorCache.getConnectorCache(PulsarConnectorCache.java:83)`
`at org.apache.pulsar.sql.presto.PulsarSplitManager.getSplitsNonPartitionedTopic(PulsarSplitManager.java:224)`
`at org.apache.pulsar.sql.presto.PulsarSplitManager.getSplits(PulsarSplitManager.java:126)`
`... 12 more`
`Caused by: org.apache.bookkeeper.meta.exceptions.MetadataException: Failed to initialized ledger manager factory`
`at org.apache.bookkeeper.meta.zk.ZKMetadataDriverBase.getLedgerManagerFactory(ZKMetadataDriverBase.java:243)`
`at org.apache.bookkeeper.client.BookKeeper.<init>(BookKeeper.java:518)`
`... 21 more`
`Caused by: <http://java.io|java.io>.IOException: Empty Ledger Root Path.`
`at org.apache.bookkeeper.meta.AbstractZkLedgerManagerFactory.newLedgerManagerFactory(AbstractZkLedgerManagerFactory.java:158)`
`at org.apache.bookkeeper.meta.zk.ZKMetadataDriverBase.getLedgerManagerFactory(ZKMetadataDriverBase.java:239)`
`... 22 more`
----
2020-04-16 10:41:56 UTC - Adelina Brask: This is SOLVED :slightly_smiling_face: it was the wrong broker service URI in the config.
+1 : Ebere Abanonu, Sijie Guo, David Kjerrumgaard
----
2020-04-16 12:06:54 UTC - Ebere Abanonu: Thanks!
----
2020-04-16 12:09:03 UTC - Ebere Abanonu: I guess byte[] is also not fully supported too? Cause I got an error about converting to byte
----
2020-04-16 12:13:31 UTC - Ebere Abanonu: You could use pulsar sql, it is a connector for presto
----
2020-04-16 12:14:23 UTC - Damien: @Sijie Guo hi, I’m also very interested in a possible answer. Could you please point on Flink-Pulsar/Spark-Pulsar or even Pulsar-SQL the part of the code which is responsible to call the segment-reader or state-store ?
----
2020-04-16 13:04:02 UTC - Penghui Li: The byte[] in pulsar convert to presto `VarbinaryType.VARBINARY`
----
2020-04-16 13:57:19 UTC - charlie: @charlie has joined the channel
----
2020-04-16 14:45:29 UTC - Subash Kunjupillai: Hi,
I'm trying to understand the need to <http://pulsar.apache.org/docs/en/deploy-bare-metal/#initialize-cluster-metadata|"Initialize Cluster Metadata">.
The reason for the query is, we are not planning to have a DNS server, in that case we need to mention each host information during this initialization process. I see it is explicitly mentioned in the doc that "You only need to write *once*", so I believe this need not be required when we are adding additional brokers to the cluster. So I'm trying to understand how this initialization is used by Broker. Can someone please help me to understand?
----
2020-04-16 15:20:11 UTC - Sijie Guo: @Adelina Brask thank you for sharing the logs. I will check the provided logs.
----
2020-04-16 15:20:43 UTC - Sijie Guo: Glad to know that it is solved.
+1 : Adelina Brask
----
2020-04-16 15:53:03 UTC - Vladimir Shchur: Hi! I'm testing schema, accoring to <http://pulsar.apache.org/docs/en/schema-evolution-compatibility/> there should be a setting `schemaRegistryCompatibilityCheckers` somewhere, but I've no idea where this setting resides. Can't find any place where to configure schema compatibility check strategy either
----
2020-04-16 16:52:59 UTC - Ebere Abanonu: I need further clarification. If the topic has message already will it be deleted too?
----
2020-04-16 16:55:46 UTC - Ebere Abanonu: The topic got delete even with messages produced but no consumer. Just want to produce and not consume. Going to use presto to query the data @Sijie Guo
----
2020-04-16 17:06:27 UTC - Adelina Brask: <http://pulsar.apache.org/docs/en/schema-manage/#adjust-compatibility>
`schemaRegistryCompatibilityCheckers` is not a setting you/we have access to. It's an automatic check that the producer/consumer uses. Using this link to can check and change the compatibility strategy.
To check the compatibility policy of your namespace use `bin/pulsar-admin namespaces policies tenant/namespace`
----
2020-04-16 17:12:24 UTC - Adelina Brask: Hi Subash, If you don't initialize the cluster meta data , zookeeper won't know which bookies it is responsable for. Because on the other configuration files (broker or presto) we only give the zookeeper server names and not the bookies. The communication between brokers and bookies happens only through zookeeper. I understand it's a pain to do it again if you add another broker , but it's necessary for using all your ressources. If you had a DNS you only need to do it once.<http://broker.co| >
----
2020-04-16 17:13:39 UTC - David Kjerrumgaard: @Vladimir Shchur You can configure that at the namespace level. <https://pulsar.apache.org/docs/en/pulsar-admin/#set-schema-autoupdate-strategy>
----
2020-04-16 17:13:53 UTC - Adelina Brask: @Sijie Guo I appreciate having your eyes and help on this. I am getting desperate :slightly_smiling_face: I am failing to understand why sink + sources won't work together in cluster mode (only separately or only in locarun)
----
2020-04-16 17:15:48 UTC - David Kjerrumgaard: @Adelina Brask I saw your earlier post about having issues running a Source and Sink concurrently. I believe that using the above command to disable compatibility checks at the NS level should resolve that issue. Although longer term you probably need to get them both using the same schema
----
2020-04-16 17:17:08 UTC - Adelina Brask: It's a little missleading...as the name of the parameter is pulsar.broker-service-url (we used to <pulsar://whatever:6650> from other configs) but here is expects the web url :slightly_smiling_face: (<http://whatever:8080>)
----
2020-04-16 17:19:18 UTC - Vladimir Shchur: Thank you! Missed that documentation paragraph
+1 : Adelina Brask
----
2020-04-16 17:21:12 UTC - Adelina Brask: Hi David , I got them to use the same schema type , but with no help. My namespace compatibility strategy is now set to
`"schema_auto_update_compatibility_strategy" : "Full",`
`"schema_compatibility_strategy" : "UNDEFINED",`
----
2020-04-16 17:21:24 UTC - Adelina Brask: if try to play with those settings then :slightly_smiling_face: and turn off the compatibility checks
----
2020-04-16 17:23:27 UTC - David Kjerrumgaard: `ALWAYS_COMPATIBLE` might be the right option.
----
2020-04-16 17:24:47 UTC - Adelina Brask: I just tried it, delete the topics & sources/sink and created them again - the problem persists
----
2020-04-16 17:25:17 UTC - Adelina Brask: But everything works in locarun...localrun uses the same settings/policies right?
----
2020-04-16 17:39:07 UTC - David Kjerrumgaard: It should.
----
2020-04-16 17:45:53 UTC - Sijie Guo: yeah. that is a good point. I am creating a github issue to improve it.
----
2020-04-16 17:49:38 UTC - Sijie Guo: <https://github.com/apache/pulsar/issues/6748>
----
2020-04-16 17:49:49 UTC - Sijie Guo: Contributions are welcome! :slightly_smiling_face:
+1 : Adelina Brask
----
2020-04-16 18:08:40 UTC - Adelina Brask: Nice. I feel like I am part of the family now :sunglasses:
wave : Sijie Guo
----
2020-04-16 18:08:54 UTC - Matteo Merli: So, for non-persistent topics, the messages are anyway not retained, so the ack path is not really needed
----
2020-04-16 18:09:38 UTC - Matteo Merli: since messages are not persisted, there are also no message ids (I agree it would be better to have them, for debug purpose though)
----
2020-04-16 18:14:31 UTC - Sijie Guo: @Adelina Brask there two separate issues here:
1. netty source can’t bound to a port. I think this is irrelative to the schema issue. It is just netty source wasn’t designed to run in a non-containerized environment. <http://pulsar.apache.org/docs/en/io-netty-source/#configuration> (check this documentation. It clarifies at the top). The issue is if you have multiple netty source instances running in the same machine, you will be encountering port conflicts issue.
2. schema issue. Currently elastic sink isn’t designed for schema awareness. It just takes raw bytes. So if you are using the elastic sink with our sources, make sure the input topic exists first before using elastic sink. Then it will work as well.
with that being said, if you are using datagen and elastic together, make sure starting datagen first. So datagen will create a topic with avro schema. Elastic sink will be able to run but it probably doesn’t work well since the bytes are avro encoded binary data. I don’t think Elastic is able to index it.
If you are using netty source with elastic sink, these should work in general. I think the problem is more about the port conflict.
----
2020-04-16 18:27:43 UTC - Adelina Brask: I will try it out tomorrow. However, port is binding fine if I start by creating the netty source first (however then elastic sink fails). If I create the elastic sink first, it starts perfectly , however the netty won't start up. I fail to understand the logic and why everything works perfectly in localrun but we have issues in clustermode?
----
2020-04-16 18:27:59 UTC - Adelina Brask: @Sijie Guo thanks for your time.
----
2020-04-16 18:32:35 UTC - Rahul: @Matteo Merli As per the Release Plan documents I see the following dates are provided.
```December 2019 - March 2020
March 2020 - June 2020
June 2020 - September 2020
September 2020 - December 2020```
Currently I see there are some release candite tags (`2.5.1`) which is for bug fixes. When can we expect feature releases (`2.6.0`)?
----
2020-04-16 18:51:16 UTC - Sijie Guo: okay. I will give it a try as well.
----
2020-04-16 18:54:42 UTC - Ebere Abanonu: ```java.lang.ClassCastException: java.nio.HeapByteBuffer cannot be cast to [B
2020-04-16T18:53:25.559818923Z at org.apache.pulsar.sql.presto.PulsarRecordCursor.getSlice(PulsarRecordCursor.java:496)
2020-04-16T18:53:25.559823223Z at com.facebook.presto.$gen.CursorProcessor_20200416_185325_28.project_0(Unknown Source)
2020-04-16T18:53:25.559827723Z at com.facebook.presto.$gen.CursorProcessor_20200416_185325_28.process(Unknown Source)
2020-04-16T18:53:25.559832623Z at com.facebook.presto.operator.ScanFilterAndProjectOperator.processColumnSource(ScanFilterAndProjectOperator.java:237)
2020-04-16T18:53:25.559836923Z at com.facebook.presto.operator.ScanFilterAndProjectOperator.getOutput(ScanFilterAndProjectOperator.java:229)
2020-04-16T18:53:25.559848923Z at com.facebook.presto.operator.Driver.processInternal(Driver.java:373)
2020-04-16T18:53:25.559852723Z at com.facebook.presto.operator.Driver.lambda$processFor$8(Driver.java:282)
2020-04-16T18:53:25.559856623Z at com.facebook.presto.operator.Driver.tryWithLock(Driver.java:672)
2020-04-16T18:53:25.559860423Z at com.facebook.presto.operator.Driver.processFor(Driver.java:276)
2020-04-16T18:53:25.559864323Z at com.facebook.presto.execution.SqlTaskExecution$DriverSplitRunner.processFor(SqlTaskExecution.java:973)
2020-04-16T18:53:25.559868223Z at com.facebook.presto.execution.executor.PrioritizedSplitRunner.process(PrioritizedSplitRunner.java:162)
2020-04-16T18:53:25.559872323Z at com.facebook.presto.execution.executor.TaskExecutor$TaskRunner.run(TaskExecutor.java:477)
2020-04-16T18:53:25.559876423Z at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
2020-04-16T18:53:25.559880323Z at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
2020-04-16T18:53:25.559884223Z at java.lang.Thread.run(Thread.java:748)```
----
2020-04-16 18:55:17 UTC - Ebere Abanonu: byte[] issues in presto
----
2020-04-16 19:21:26 UTC - Sijie Guo: Interesting. Can you create an issue for that?
----
2020-04-16 19:26:05 UTC - Rahul: Found it. Its due by `May 15`. Means next release cycle.
----
2020-04-16 19:34:32 UTC - Ebere Abanonu: Issue created!
----
2020-04-16 20:01:42 UTC - Jared Mackey: I see, so acking is really pointless in non-persistent topics. What does it do when I ask to redeliver messages?
----
2020-04-16 21:28:43 UTC - Matteo Merli: Nothing really, because brokers are not buffering up :slightly_smiling_face:
----
2020-04-16 21:35:25 UTC - Frederic COLLIN: @Frederic COLLIN has joined the channel
----
2020-04-17 03:00:32 UTC - Tymm: Hi guys, i'm new with Pulsar and need some advice on designing the topic structure. I have couple of iot devices: temperature sensors, infrared sensors, etc, they each provides data in different structures with its own unique ID, and imagine a company has 10 sites and each site has 5 temperature sensor, and I want to generate hourly, weekly, daily reports using aggregation in pulsar function with enrichment from external MSSQL db (time start and time stop for the data which different across site).
Should I create topics that are module-based (temperature-sensor, infrared-sensor, ...) or device-based (device-unique-id-temperature, device-unique-id-infrared, ... ) ?
Is Pulsar SQL a better choice in term of aggregation of data?
----
2020-04-17 07:13:11 UTC - Adelina Brask: @Sijie Guo I fixed the issue. The incompatibility was somehow due to the S3 offloader. I think it was acting like a 'consumer' . The moment I turned it off and let some message in the topic, both the source and sink started normally. :slightly_smiling_face:
----
2020-04-17 07:22:04 UTC - Adelina Brask: If I were you I would create 1 namepace for each site, then 1 topic pr. data structure, module based. Each topic will have a device ID parameter you can make agreggations based on. But you are free to do whatever you wish..I mean if you need each sensor to have different output, it may be easier to make 1 topic pr module pr device .
----
2020-04-17 07:32:25 UTC - Adelina Brask: Hi guys. If I need data from a topic to both be consumed by Elastic sink and offloaded to s3 at the same time, what constellation/settings will you recommend? Should I duplicate the topic to another namespace that is exclusively made for offloading?
----
2020-04-17 07:34:36 UTC - Sijie Guo: No you don’t need to duplicate the topic. You can configure to offload a topic and the same topic can be consumed by any consumers.
----
2020-04-17 07:36:07 UTC - Sijie Guo: hmm interesting …
----
2020-04-17 07:36:44 UTC - Sijie Guo: ideally s3 offloader shouldn’t change any schema information.
----
2020-04-17 07:36:54 UTC - Adelina Brask: But woudn't the topic be empty after consumed ack?
----
2020-04-17 07:37:00 UTC - Sijie Guo: Can you constantly reproduce this issue?
----
2020-04-17 07:39:50 UTC - Sijie Guo: Ah, the messages will be kept if there are retention configured after they are consumed. So I took what I said above. You need to enable retention for this to work.
----
2020-04-17 07:40:29 UTC - Adelina Brask: ah , makes sense :slightly_smiling_face: Thanks a lot
----
2020-04-17 07:44:38 UTC - JG: yes, I will use the book keeper as event storage and going to replay events to create again the read part
----
2020-04-17 08:21:17 UTC - Adelina Brask: No I can't ... it still fails trying to reproduce the setup without the offloaders. I need to spend some more time researching it . :slightly_smiling_face:
----
2020-04-17 08:59:01 UTC - Subash Kunjupillai: Hi Adelina,
Thanks for your response. During initialization we are providing only zookeeper and broker information, in that case how does Broker identifies the Bookie information? Is it hard coded somewhere in Broker to look for specific Znode for Bookie information? Also if broker is able to identify the Bookie through zookeeper, I don't think we have to run this initialization again while adding more Brokers later. Please share your thoughts.
----