You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by "nareshv (via GitHub)" <gi...@apache.org> on 2024/04/15 05:00:02 UTC

[D] is bookkeeper stream storage, pulsar standalone and zookeeper [pulsar]

GitHub user nareshv created a discussion: is bookkeeper stream storage,  pulsar standalone and zookeeper

Pulsar standalone has flags like `--no-stream-storage`. Is it ok to skip using stream storage for production use ?

Also, does disabling stream storage avoid bookkeeper from using zookeeper ?

GitHub link: https://github.com/apache/pulsar/discussions/22503

----
This is an automatically sent email for commits@pulsar.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@pulsar.apache.org


Re: [D] is bookkeeper stream storage, pulsar standalone and zookeeper [pulsar]

Posted by "lhotari (via GitHub)" <gi...@apache.org>.
GitHub user lhotari added a comment to the discussion: is bookkeeper stream storage,  pulsar standalone and zookeeper

> Pulsar standalone has flags like `--no-stream-storage`. Is it ok to skip using stream storage for production use ?

Yes, it's not needed for Pulsar's core functionality. Stream storage is used for [stateful functions](https://pulsar.apache.org/docs/3.2.x/functions-worker-stateful/).
In general, Pulsar standalone isn't designed for "production use", but everyone can decide on their own what they use it for.

> Also, does disabling stream storage avoid bookkeeper from using zookeeper ?

No, stream storage requires using Bookkeeper. The parameter `--no-stream-storage` is unrelated to configuring the metadata store type and whether Zookeeper is used. Pulsar standalone will not use zookeeper by default. 
To use Zookeeper in Pulsar standalone, `PULSAR_STANDALONE_USE_ZOOKEEPER=1` environment variable should be set. 
There has been some instability in using the default Rocksdb based metadata store in Pulsar standalone and that's why the docs now recommend using Zookeeper based storage. That document change was made recently with PR https://github.com/apache/pulsar-site/pull/871 due to issues described in https://github.com/apache/pulsar/discussions/20985 .
For actual production deployments, [StreamNative Oxia](https://github.com/streamnative/oxia) is becoming a stable alternative for Zookeeper. 


GitHub link: https://github.com/apache/pulsar/discussions/22503#discussioncomment-9113895

----
This is an automatically sent email for commits@pulsar.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@pulsar.apache.org


Re: [D] is bookkeeper stream storage, pulsar standalone and zookeeper [pulsar]

Posted by "nareshv (via GitHub)" <gi...@apache.org>.
GitHub user nareshv added a comment to the discussion: is bookkeeper stream storage,  pulsar standalone and zookeeper

> In general, Pulsar standalone isn't designed for "production use", but everyone can decide on their own what they use it for.

Pulsar standalone seem to be a bit closer to realizing https://github.com/apache/pulsar/wiki/PIP-21:-Pulsar-Edge-Component 

> No, stream storage requires using Bookkeeper.

Noted

> The parameter --no-stream-storage is unrelated to configuring the metadata store type and whether Zookeeper is used. Pulsar standalone will not use zookeeper by default.

We are using something like `/bin/env PULSAR_STANDALONE_USE_ZOOKEEPER=1 $PULSAR_DIR/bin/pulsar-daemon start standalone --no-functions-worker` and wondering if we can disable the stream storage. Looks like we can disable it if we don't use stateful functions

> For actual production deployments, [StreamNative Oxia](https://github.com/streamnative/oxia) is becoming a stable alternative for Zookeeper.

Noted.

GitHub link: https://github.com/apache/pulsar/discussions/22503#discussioncomment-9114866

----
This is an automatically sent email for commits@pulsar.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@pulsar.apache.org