You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by "thetumbled (via GitHub)" <gi...@apache.org> on 2024/03/15 02:23:21 UTC
[PR] [improve][conf] PIP-346: Add a simplified configuration file for Pulsar [pulsar]
thetumbled opened a new pull request, #22274:
URL: https://github.com/apache/pulsar/pull/22274
### Motivation
PIP-346
### Verifying this change
- [ ] Make sure that the change passes the CI checks.
*(Please pick either of the following options)*
This change is a trivial rework / code cleanup without any test coverage.
*(or)*
This change is already covered by existing tests, such as *(please describe tests)*.
*(or)*
This change added tests and can be verified as follows:
*(example:)*
- *Added integration tests for end-to-end deployment with large payloads (10MB)*
- *Extended integration test for recovery after broker failure*
### Does this pull request potentially affect one of the following parts:
<!-- DO NOT REMOVE THIS SECTION. CHECK THE PROPER BOX ONLY. -->
*If the box was checked, please highlight the changes*
- [ ] Dependencies (add or upgrade a dependency)
- [ ] The public API
- [ ] The schema
- [ ] The default values of configurations
- [ ] The threading model
- [ ] The binary protocol
- [ ] The REST endpoints
- [ ] The admin CLI options
- [ ] The metrics
- [ ] Anything that affects deployment
### Documentation
<!-- DO NOT REMOVE THIS SECTION. CHECK THE PROPER BOX ONLY. -->
- [ ] `doc` <!-- Your PR contains doc changes. -->
- [ ] `doc-required` <!-- Your PR changes impact docs and you will update later -->
- [ ] `doc-not-needed` <!-- Your PR changes do not impact docs -->
- [ ] `doc-complete` <!-- Docs have been already added -->
### Matching PR in forked repository
PR in forked repository: <!-- ENTER URL HERE -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [improve][pip] PIP-346: Add a simplified configuration file for Pulsar [pulsar]
Posted by "dao-jun (via GitHub)" <gi...@apache.org>.
dao-jun commented on PR #22274:
URL: https://github.com/apache/pulsar/pull/22274#issuecomment-1998895272
I support
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [improve][pip] PIP-346: Add a simplified configuration file for Pulsar [pulsar]
Posted by "BewareMyPower (via GitHub)" <gi...@apache.org>.
BewareMyPower commented on code in PR #22274:
URL: https://github.com/apache/pulsar/pull/22274#discussion_r1525693235
##########
pip/pip-346.md:
##########
@@ -0,0 +1,39 @@
+
+# PIP-346: Add a simplified configuration file for Pulsar
+
+# Background knowledge
+
+Nowadays, `broker.conf` contains almost all the configurations of the broker. It's hard to manage the configurations in a single file, which reach almost **500** configurations.
+
+However, we will only modify a few configurations in the daily work. It's horrible for a beginner to start to learn pulsar, and it's also a potentially error-prone process for advanced users to maintain such a large configuration file.
+
+We can learn from zookeeper that it uses a simplified configuration file, which only contains the configurations that are frequently modified and hides all the configurations that are rarely modified and advanced.
+
+# Goals
+
+## In Scope
+
+Add a simplified configuration file for Pulsar, which only contains the configurations that are frequently modified.
+
+
+# Detailed Design
+
+## Design & Implementation Details
+
+- rename current `broker.conf` to `complete_broker.conf`
+- add a new `broker.conf` which only contains the configurations that are frequently modified
Review Comment:
I think it depends on the experience.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [improve][pip] PIP-346: Add a simplified configuration file for Pulsar [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on PR #22274:
URL: https://github.com/apache/pulsar/pull/22274#issuecomment-2071609387
> Hi, could we move forward with this PIP? It seems that it is not rely on another PIP `Proposal to Replace Pulsar's Homegrown Configuration Framework with Gestalt Config`. @merlimat @BewareMyPower @crossoverJie @codelipenghui @dao-jun @lhotari
The current Pulsar configuration solution is currently a mess (example: `apply-config-from-env.py` is needed to override configuration from environment) and instead of just doing changes that don't really address the problems, I think that it's a distraction.
That's why I think that we shouldn't proceed on PIP-346. We need a design for revisiting the configuration solution for Pulsar 4.0 and beyond.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [improve][pip] PIP-346: Add a simplified configuration file for Pulsar [pulsar]
Posted by "thetumbled (via GitHub)" <gi...@apache.org>.
thetumbled commented on PR #22274:
URL: https://github.com/apache/pulsar/pull/22274#issuecomment-1999094628
Hi, i have added the candidate conf list, looking forward to your feedback. @merlimat @BewareMyPower @crossoverJie @dao-jun
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [improve][pip] PIP-346: Add a simplified configuration file for Pulsar [pulsar]
Posted by "merlimat (via GitHub)" <gi...@apache.org>.
merlimat closed pull request #22274: [improve][pip] PIP-346: Add a simplified configuration file for Pulsar
URL: https://github.com/apache/pulsar/pull/22274
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [improve][pip] PIP-346: Add a simplified configuration file for Pulsar [pulsar]
Posted by "crossoverJie (via GitHub)" <gi...@apache.org>.
crossoverJie commented on code in PR #22274:
URL: https://github.com/apache/pulsar/pull/22274#discussion_r1525689903
##########
pip/pip-346.md:
##########
@@ -0,0 +1,39 @@
+
+# PIP-346: Add a simplified configuration file for Pulsar
+
+# Background knowledge
+
+Nowadays, `broker.conf` contains almost all the configurations of the broker. It's hard to manage the configurations in a single file, which reach almost **500** configurations.
+
+However, we will only modify a few configurations in the daily work. It's horrible for a beginner to start to learn pulsar, and it's also a potentially error-prone process for advanced users to maintain such a large configuration file.
+
+We can learn from zookeeper that it uses a simplified configuration file, which only contains the configurations that are frequently modified and hides all the configurations that are rarely modified and advanced.
+
+# Goals
+
+## In Scope
+
+Add a simplified configuration file for Pulsar, which only contains the configurations that are frequently modified.
+
+
+# Detailed Design
+
+## Design & Implementation Details
+
+- rename current `broker.conf` to `complete_broker.conf`
+- add a new `broker.conf` which only contains the configurations that are frequently modified
Review Comment:
How do we confirm which configurations need to be modified frequently?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [improve][pip] PIP-346: Add a simplified configuration file for Pulsar [pulsar]
Posted by "merlimat (via GitHub)" <gi...@apache.org>.
merlimat commented on code in PR #22274:
URL: https://github.com/apache/pulsar/pull/22274#discussion_r1525809596
##########
pip/pip-346.md:
##########
@@ -0,0 +1,39 @@
+
+# PIP-346: Add a simplified configuration file for Pulsar
+
+# Background knowledge
+
+Nowadays, `broker.conf` contains almost all the configurations of the broker. It's hard to manage the configurations in a single file, which reach almost **500** configurations.
+
+However, we will only modify a few configurations in the daily work. It's horrible for a beginner to start to learn pulsar, and it's also a potentially error-prone process for advanced users to maintain such a large configuration file.
+
+We can learn from zookeeper that it uses a simplified configuration file, which only contains the configurations that are frequently modified and hides all the configurations that are rarely modified and advanced.
+
+# Goals
+
+## In Scope
+
+Add a simplified configuration file for Pulsar, which only contains the configurations that are frequently modified.
+
+
+# Detailed Design
+
+## Design & Implementation Details
+
+- rename current `broker.conf` to `complete_broker.conf`
+- add a new `broker.conf` which only contains the configurations that are frequently modified
Review Comment:
I think this is a good discussion starting point, though the precise list and modality should be eventually part of the PIP that is voted on.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [improve][pip] PIP-346: Add a simplified configuration file for Pulsar [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on PR #22274:
URL: https://github.com/apache/pulsar/pull/22274#issuecomment-1999891418
I provided my feedback on the mailing list https://lists.apache.org/thread/xdq62jytxxj421vly6qzn9dn8vvjqyv1
> Before making this change, I hope that there could be discussion about
> improving the configuration solution in Pulsar.
> The mailing list thread is
> https://lists.apache.org/thread/8splwyrn25pt5rb6ph0yz0tfgknn3hqh .
>
> With a proper configuration library such as Gestalt Config [1], it's
> possible to have a solution that is expressed in this PIP-346.
> In Gestalt Config, the feature "Merge Multiple Sources: Seamlessly
> merge configurations from different sources to create comprehensive
> settings." support this.
>
> This is the reason why I don't think that we should move forward with
> PIP-346 without addressing the proposal in the other mailing list
> thread
> "[DISCUSSION] Proposal to Replace Pulsar's Homegrown Configuration
> Framework with Gestalt Config" [2].
>
> -Lari
>
> 1 - https://github.com/gestalt-config/gestalt#features
> 2 - https://lists.apache.org/thread/8splwyrn25pt5rb6ph0yz0tfgknn3hqh
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [improve][pip] PIP-346: Add a simplified configuration file for Pulsar [pulsar]
Posted by "thetumbled (via GitHub)" <gi...@apache.org>.
thetumbled commented on PR #22274:
URL: https://github.com/apache/pulsar/pull/22274#issuecomment-2684165113
I wonder whether we can push this PIP? Half year past, but nothing improvement for the configuration.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [improve][pip] PIP-346: Add a simplified configuration file for Pulsar [pulsar]
Posted by "thetumbled (via GitHub)" <gi...@apache.org>.
thetumbled commented on PR #22274:
URL: https://github.com/apache/pulsar/pull/22274#issuecomment-2041734373
Hi, could we move forward with this PIP? It seems that it is not rely on another PIP `Proposal to Replace Pulsar's Homegrown Configuration Framework with Gestalt Config`.
@merlimat @BewareMyPower @crossoverJie @codelipenghui @dao-jun @lhotari
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [improve][pip] PIP-346: Add a simplified configuration file for Pulsar [pulsar]
Posted by "thetumbled (via GitHub)" <gi...@apache.org>.
thetumbled commented on code in PR #22274:
URL: https://github.com/apache/pulsar/pull/22274#discussion_r1525697240
##########
pip/pip-346.md:
##########
@@ -0,0 +1,39 @@
+
+# PIP-346: Add a simplified configuration file for Pulsar
+
+# Background knowledge
+
+Nowadays, `broker.conf` contains almost all the configurations of the broker. It's hard to manage the configurations in a single file, which reach almost **500** configurations.
+
+However, we will only modify a few configurations in the daily work. It's horrible for a beginner to start to learn pulsar, and it's also a potentially error-prone process for advanced users to maintain such a large configuration file.
+
+We can learn from zookeeper that it uses a simplified configuration file, which only contains the configurations that are frequently modified and hides all the configurations that are rarely modified and advanced.
+
+# Goals
+
+## In Scope
+
+Add a simplified configuration file for Pulsar, which only contains the configurations that are frequently modified.
+
+
+# Detailed Design
+
+## Design & Implementation Details
+
+- rename current `broker.conf` to `complete_broker.conf`
+- add a new `broker.conf` which only contains the configurations that are frequently modified
Review Comment:
We can vote the configurations in the implementation pr.
It is a process of compression of operation and maintenance experience, which is valuable in my view.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [improve][pip] PIP-346: Add a simplified configuration file for Pulsar [pulsar]
Posted by "thetumbled (via GitHub)" <gi...@apache.org>.
thetumbled commented on PR #22274:
URL: https://github.com/apache/pulsar/pull/22274#issuecomment-2002778266
> I provided my feedback on the mailing list https://lists.apache.org/thread/xdq62jytxxj421vly6qzn9dn8vvjqyv1
>
> > Before making this change, I hope that there could be discussion about
> > improving the configuration solution in Pulsar.
> > The mailing list thread is
> > https://lists.apache.org/thread/8splwyrn25pt5rb6ph0yz0tfgknn3hqh .
> > With a proper configuration library such as Gestalt Config [1], it's
> > possible to have a solution that is expressed in this PIP-346.
> > In Gestalt Config, the feature "Merge Multiple Sources: Seamlessly
> > merge configurations from different sources to create comprehensive
> > settings." support this.
> > This is the reason why I don't think that we should move forward with
> > PIP-346 without addressing the proposal in the other mailing list
> > thread
> > "[DISCUSSION] Proposal to Replace Pulsar's Homegrown Configuration
> > Framework with Gestalt Config" [2].
> > -Lari
> > 1 - https://github.com/gestalt-config/gestalt#features
> > 2 - https://lists.apache.org/thread/8splwyrn25pt5rb6ph0yz0tfgknn3hqh
Based on my understanding, this new lib can merge multiple configuration file with some file having higher priority.
> I provided my feedback on the mailing list https://lists.apache.org/thread/xdq62jytxxj421vly6qzn9dn8vvjqyv1
>
> > Before making this change, I hope that there could be discussion about
> > improving the configuration solution in Pulsar.
> > The mailing list thread is
> > https://lists.apache.org/thread/8splwyrn25pt5rb6ph0yz0tfgknn3hqh .
> > With a proper configuration library such as Gestalt Config [1], it's
> > possible to have a solution that is expressed in this PIP-346.
> > In Gestalt Config, the feature "Merge Multiple Sources: Seamlessly
> > merge configurations from different sources to create comprehensive
> > settings." support this.
> > This is the reason why I don't think that we should move forward with
> > PIP-346 without addressing the proposal in the other mailing list
> > thread
> > "[DISCUSSION] Proposal to Replace Pulsar's Homegrown Configuration
> > Framework with Gestalt Config" [2].
> > -Lari
> > 1 - https://github.com/gestalt-config/gestalt#features
> > 2 - https://lists.apache.org/thread/8splwyrn25pt5rb6ph0yz0tfgknn3hqh
Not mastering to use mail list to reply, i have better reply you here.😂
Hi Lari,
Based on my understanding, this new lib can merge multiple configuration file with some file having higher priority, which is a nice feature as this pip try to introduce another simplified configuration file.
But i wonder why we can't move forward with PIP-346 without introducing Gestalt. Whether introducing Gestalt or not, we will introduce a new simplified configuration file. I don't get why we should block this PIP?
Regards,
thetumbled.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [improve][pip] PIP-346: Add a simplified configuration file for Pulsar [pulsar]
Posted by "merlimat (via GitHub)" <gi...@apache.org>.
merlimat commented on PR #22274:
URL: https://github.com/apache/pulsar/pull/22274#issuecomment-6179357114
Thanks for this proposal. A PIP is accepted through a vote on dev@pulsar.apache.org, and this one was discussed there but never went to a vote. It has been open since March 2024. As part of the open-PR cleanup discussed on dev@, I'm closing PIP proposals that are more than six months old without a passed vote. If you'd like to move it forward, please reopen this PR and start a [DISCUSS] or [VOTE] thread on dev@.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org