You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2022/02/24 10:31:34 UTC
[GitHub] [pulsar] gaozhangmin opened a new pull request #14450: integrate package management service to pulsar functions service
gaozhangmin opened a new pull request #14450:
URL: https://github.com/apache/pulsar/pull/14450
Fixes #14224
### Motivation
The current function creation will directly store the jar file into DistributedLog by calling WorkerUtils.uploadFileToBookkeeper no matter we create function by jar file or functionPkgUrl. It doesn't store the jar file by the unified package service interface
### Modifications
upload package to packageManagement service.
### Documentation
Check the box below or label this PR directly (if you have committer privilege).
Need to update docs?
- [ ] `doc-required`
(If you need help on updating docs, create a doc issue)
- [ ] `no-need-doc`
(Please explain why)
- [ ] `doc`
(If this PR contains doc changes)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] gaozhangmin removed a comment on pull request #14450: [Function] Integrate package management service to pulsar functionWorker
Posted by GitBox <gi...@apache.org>.
gaozhangmin removed a comment on pull request #14450:
URL: https://github.com/apache/pulsar/pull/14450#issuecomment-1082796627
/pulsarbot run-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] gaozhangmin commented on pull request #14450: [Function] Integrate package management service to pulsar functionWorker
Posted by GitBox <gi...@apache.org>.
gaozhangmin commented on pull request #14450:
URL: https://github.com/apache/pulsar/pull/14450#issuecomment-1083016312
/pulsarbot run-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] gaozhangmin commented on pull request #14450: [Function] Integrate package management service to pulsar functionWorker
Posted by GitBox <gi...@apache.org>.
gaozhangmin commented on pull request #14450:
URL: https://github.com/apache/pulsar/pull/14450#issuecomment-1062607554
/pulsarbot run-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] gaozhangmin commented on a change in pull request #14450: [Function] Integrate package management service to pulsar functionWorker
Posted by GitBox <gi...@apache.org>.
gaozhangmin commented on a change in pull request #14450:
URL: https://github.com/apache/pulsar/pull/14450#discussion_r829699012
##########
File path: conf/functions_worker.yml
##########
@@ -26,6 +26,13 @@ workerHostname: localhost
workerPort: 6750
workerPortTls: 6751
+# The metadata store URL
+# Examples:
+# * zk:my-zk-1:2181,my-zk-2:2181,my-zk-3:2181
+# * my-zk-1:2181,my-zk-2:2181,my-zk-3:2181 (will default to ZooKeeper when the schema is not specified)
+# * zk:my-zk-1:2181,my-zk-2:2181,my-zk-3:2181/my-chroot-path (to add a ZK chroot path)
+metadataStoreUrl: zk:localhost:2181
Review comment:
metadataStoreUrl is used for BookKeeperPackagesStorage#initializeDlogNamespace。If we want to start BookKeeperPackagesStorage storage service in a separately running workers. we must support this configuration.
##########
File path: conf/functions_worker.yml
##########
@@ -45,6 +52,10 @@ metadataStoreCacheExpirySeconds: 300
################################
# Function package management
################################
+packagesManagementStorageProvider:
+ org.apache.pulsar.packages.management.storage.bookkeeper.BookKeeperPackagesStorageProvider
+packagesManagementLedgerRootPath: /ledgers
+packagesReplicas: 1
Review comment:
Run Functions-worker separately, independent with broker, in this situation, if we want to start packageManagement service in worker, We should have this configurations.
##########
File path: pulsar-broker/src/main/java/org/apache/pulsar/broker/PulsarService.java
##########
@@ -780,14 +780,14 @@ public void start() throws PulsarServerException {
acquireSLANamespace();
- // start function worker service if necessary
- this.startWorkerService(brokerService.getAuthenticationService(), brokerService.getAuthorizationService());
-
// start packages management service if necessary
if (config.isEnablePackagesManagement()) {
this.startPackagesManagementService();
}
Review comment:
There are two situations:
1、Run Functions-worker with brokers. If broker enables function worker without enabling packageManagement, packageManagement service is enabled in worker by default. Else, worker will use packageManagement service enabled by broker.
2、Run Functions-worker separately. Without broker, We enable packageManagement service in worker by default
##########
File path: conf/functions_worker.yml
##########
@@ -26,6 +26,13 @@ workerHostname: localhost
workerPort: 6750
workerPortTls: 6751
+# The metadata store URL
+# Examples:
+# * zk:my-zk-1:2181,my-zk-2:2181,my-zk-3:2181
+# * my-zk-1:2181,my-zk-2:2181,my-zk-3:2181 (will default to ZooKeeper when the schema is not specified)
+# * zk:my-zk-1:2181,my-zk-2:2181,my-zk-3:2181/my-chroot-path (to add a ZK chroot path)
+metadataStoreUrl: zk:localhost:2181
Review comment:
We can be back to use configurationMetadataStoreUrl if metadataStoreUrl is not set.
##########
File path: conf/functions_worker.yml
##########
@@ -26,6 +26,13 @@ workerHostname: localhost
workerPort: 6750
workerPortTls: 6751
+# The metadata store URL
+# Examples:
+# * zk:my-zk-1:2181,my-zk-2:2181,my-zk-3:2181
+# * my-zk-1:2181,my-zk-2:2181,my-zk-3:2181 (will default to ZooKeeper when the schema is not specified)
+# * zk:my-zk-1:2181,my-zk-2:2181,my-zk-3:2181/my-chroot-path (to add a ZK chroot path)
+metadataStoreUrl: zk:localhost:2181
Review comment:
metadataStoreUrl is used for BookKeeperPackagesStorageConfiguration getMetadataStoreUrl。
##########
File path: conf/functions_worker.yml
##########
@@ -26,6 +26,13 @@ workerHostname: localhost
workerPort: 6750
workerPortTls: 6751
+# The metadata store URL
+# Examples:
+# * zk:my-zk-1:2181,my-zk-2:2181,my-zk-3:2181
+# * my-zk-1:2181,my-zk-2:2181,my-zk-3:2181 (will default to ZooKeeper when the schema is not specified)
+# * zk:my-zk-1:2181,my-zk-2:2181,my-zk-3:2181/my-chroot-path (to add a ZK chroot path)
+metadataStoreUrl: zk:localhost:2181
Review comment:
metadataStoreUrl is used for BookKeeperPackagesStorageConfiguration `getMetadataStoreUrl`。
##########
File path: conf/functions_worker.yml
##########
@@ -26,6 +26,13 @@ workerHostname: localhost
workerPort: 6750
workerPortTls: 6751
+# The metadata store URL
+# Examples:
+# * zk:my-zk-1:2181,my-zk-2:2181,my-zk-3:2181
+# * my-zk-1:2181,my-zk-2:2181,my-zk-3:2181 (will default to ZooKeeper when the schema is not specified)
+# * zk:my-zk-1:2181,my-zk-2:2181,my-zk-3:2181/my-chroot-path (to add a ZK chroot path)
+metadataStoreUrl: zk:localhost:2181
Review comment:
We can be back to use configurationMetadataStoreUrl if metadataStoreUrl is not set.
##########
File path: conf/functions_worker.yml
##########
@@ -26,6 +26,13 @@ workerHostname: localhost
workerPort: 6750
workerPortTls: 6751
+# The metadata store URL
+# Examples:
+# * zk:my-zk-1:2181,my-zk-2:2181,my-zk-3:2181
+# * my-zk-1:2181,my-zk-2:2181,my-zk-3:2181 (will default to ZooKeeper when the schema is not specified)
+# * zk:my-zk-1:2181,my-zk-2:2181,my-zk-3:2181/my-chroot-path (to add a ZK chroot path)
+metadataStoreUrl: zk:localhost:2181
Review comment:
@nlu90
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] nlu90 commented on a change in pull request #14450: [Function] Integrate package management service to pulsar functionWorker
Posted by GitBox <gi...@apache.org>.
nlu90 commented on a change in pull request #14450:
URL: https://github.com/apache/pulsar/pull/14450#discussion_r829289750
##########
File path: conf/functions_worker.yml
##########
@@ -26,6 +26,13 @@ workerHostname: localhost
workerPort: 6750
workerPortTls: 6751
+# The metadata store URL
+# Examples:
+# * zk:my-zk-1:2181,my-zk-2:2181,my-zk-3:2181
+# * my-zk-1:2181,my-zk-2:2181,my-zk-3:2181 (will default to ZooKeeper when the schema is not specified)
+# * zk:my-zk-1:2181,my-zk-2:2181,my-zk-3:2181/my-chroot-path (to add a ZK chroot path)
+metadataStoreUrl: zk:localhost:2181
Review comment:
what's the difference with the following `configurationMetadataStoreUrl`?
##########
File path: conf/functions_worker.yml
##########
@@ -45,6 +52,10 @@ metadataStoreCacheExpirySeconds: 300
################################
# Function package management
################################
+packagesManagementStorageProvider:
+ org.apache.pulsar.packages.management.storage.bookkeeper.BookKeeperPackagesStorageProvider
+packagesManagementLedgerRootPath: /ledgers
+packagesReplicas: 1
Review comment:
`broker.conf` and `standalone.conf` also have these same configurations.
If you added them here, what's the relationship between these duplicated configuration?
Do you want to remove them from other conf files or they will have some override relationship?
##########
File path: pulsar-broker/src/main/java/org/apache/pulsar/broker/PulsarService.java
##########
@@ -780,14 +780,14 @@ public void start() throws PulsarServerException {
acquireSLANamespace();
- // start function worker service if necessary
- this.startWorkerService(brokerService.getAuthenticationService(), brokerService.getAuthorizationService());
-
// start packages management service if necessary
if (config.isEnablePackagesManagement()) {
this.startPackagesManagementService();
}
Review comment:
If you make function worker depend on package management service, then shouldn't the package management service be always enabled when function worker is enabled?
Currently it seems they are separately configured, which may result the case that function worker is enabled but package management service is not running.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] gaozhangmin edited a comment on pull request #14450: [Function] Integrate package management service to pulsar functionWorker
Posted by GitBox <gi...@apache.org>.
gaozhangmin edited a comment on pull request #14450:
URL: https://github.com/apache/pulsar/pull/14450#issuecomment-1077554524
@freeznet I agree with you this pr brings breaking changes.
But for task one, My idea is, now user can create or update a function/sink/source by:
1、local path to the JAR file for the function (if the function is written in Java).
2、`functionPkgUrl` which starts with http/https/file/package URL from packages management service
To use the unified package service api to read and write jar files. We should upload package to packageManagement service either we provide a local path or `functionPkgUrl with http/file/https`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] gaozhangmin commented on pull request #14450: [Function] Integrate package management service to pulsar functionWorker
Posted by GitBox <gi...@apache.org>.
gaozhangmin commented on pull request #14450:
URL: https://github.com/apache/pulsar/pull/14450#issuecomment-1082796627
/pulsarbot run-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] gaozhangmin commented on pull request #14450: [Function] Integrate package management service to pulsar functionWorker
Posted by GitBox <gi...@apache.org>.
gaozhangmin commented on pull request #14450:
URL: https://github.com/apache/pulsar/pull/14450#issuecomment-1084098790
/pulsarbot run-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] gaozhangmin commented on a change in pull request #14450: [Function] Integrate package management service to pulsar functionWorker
Posted by GitBox <gi...@apache.org>.
gaozhangmin commented on a change in pull request #14450:
URL: https://github.com/apache/pulsar/pull/14450#discussion_r829703439
##########
File path: pulsar-broker/src/main/java/org/apache/pulsar/broker/PulsarService.java
##########
@@ -780,14 +780,14 @@ public void start() throws PulsarServerException {
acquireSLANamespace();
- // start function worker service if necessary
- this.startWorkerService(brokerService.getAuthenticationService(), brokerService.getAuthorizationService());
-
// start packages management service if necessary
if (config.isEnablePackagesManagement()) {
this.startPackagesManagementService();
}
Review comment:
There are two situations:
1、Run Functions-worker with brokers. If broker enables function worker without enabling packageManagement, packageManagement service is enabled in worker by default. Else, worker will use packageManagement service enabled by broker.
2、Run Functions-worker separately. Without broker, We enable packageManagement service in worker by default
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] gaozhangmin commented on a change in pull request #14450: [Function] Integrate package management service to pulsar functionWorker
Posted by GitBox <gi...@apache.org>.
gaozhangmin commented on a change in pull request #14450:
URL: https://github.com/apache/pulsar/pull/14450#discussion_r829709921
##########
File path: conf/functions_worker.yml
##########
@@ -26,6 +26,13 @@ workerHostname: localhost
workerPort: 6750
workerPortTls: 6751
+# The metadata store URL
+# Examples:
+# * zk:my-zk-1:2181,my-zk-2:2181,my-zk-3:2181
+# * my-zk-1:2181,my-zk-2:2181,my-zk-3:2181 (will default to ZooKeeper when the schema is not specified)
+# * zk:my-zk-1:2181,my-zk-2:2181,my-zk-3:2181/my-chroot-path (to add a ZK chroot path)
+metadataStoreUrl: zk:localhost:2181
Review comment:
We can be back to use configurationMetadataStoreUrl if metadataStoreUrl is not set.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] gaozhangmin commented on a change in pull request #14450: [Function] Integrate package management service to pulsar functionWorker
Posted by GitBox <gi...@apache.org>.
gaozhangmin commented on a change in pull request #14450:
URL: https://github.com/apache/pulsar/pull/14450#discussion_r829700363
##########
File path: conf/functions_worker.yml
##########
@@ -45,6 +52,10 @@ metadataStoreCacheExpirySeconds: 300
################################
# Function package management
################################
+packagesManagementStorageProvider:
+ org.apache.pulsar.packages.management.storage.bookkeeper.BookKeeperPackagesStorageProvider
+packagesManagementLedgerRootPath: /ledgers
+packagesReplicas: 1
Review comment:
Run Functions-worker separately, independent with broker, in this situation, if we want to start packageManagement service in worker, We should have this configurations.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] gaozhangmin edited a comment on pull request #14450: [Function] Integrate package management service to pulsar functionWorker
Posted by GitBox <gi...@apache.org>.
gaozhangmin edited a comment on pull request #14450:
URL: https://github.com/apache/pulsar/pull/14450#issuecomment-1077554524
@freeznet I agree with you this pr brings breaking changes.
But for task one, Now, user can create or update a function/sink/source by:
1、local path to the JAR file for the function (if the function is written in Java).
2、`functionPkgUrl` which starts with http/https/file/package URL from packages management service
I think if We config function worker to use packageManagement service, We should upload package to it either we create functions by local path or `functionPkgUrl with http/file/https`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] gaozhangmin edited a comment on pull request #14450: [Function] Integrate package management service to pulsar functionWorker
Posted by GitBox <gi...@apache.org>.
gaozhangmin edited a comment on pull request #14450:
URL: https://github.com/apache/pulsar/pull/14450#issuecomment-1077554524
@freeznet I agree with you this pr brings breaking changes.
But for task one, My idea is, now user can create or update a function/sink/source by:
1、local path to the JAR file for the function (if the function is written in Java).
2、`functionPkgUrl` which starts with http/https/file/package URL from packages management service
We should upload package to packageManagement service either we provide a local path or `functionPkgUrl with http/file/https`.
If we still `uploadFileToBookkeeper` when we enable packageManagement. function worker still relays on bookkeeper if we use `FileSystemPackagesStorageProvider`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] gaozhangmin edited a comment on pull request #14450: [Function] Integrate package management service to pulsar functionWorker
Posted by GitBox <gi...@apache.org>.
gaozhangmin edited a comment on pull request #14450:
URL: https://github.com/apache/pulsar/pull/14450#issuecomment-1077554524
@freeznet I agree with you this pr brings breaking changes.
But for task one, Now, user can create or update a function/sink/source by:
1、local path to the JAR file for the function (if the function is written in Java).
2、`functionPkgUrl` which starts with http/https/file/package URL from packages management service
I think We should upload package to packageManagement service either we create functions by local path or `functionPkgUrl with http/file/https` if We config function worker to use packageManagement service,
If we still `uploadFileToBookkeeper` when we enable packageManagement. function worker still relays on bookkeeper.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] github-actions[bot] commented on pull request #14450: Integrate package management service to pulsar functions service
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #14450:
URL: https://github.com/apache/pulsar/pull/14450#issuecomment-1049722193
@gaozhangmin:Thanks for your contribution. For this PR, do we need to update docs?
(The [PR template contains info about doc](https://github.com/apache/pulsar/blob/master/.github/PULL_REQUEST_TEMPLATE.md#documentation), which helps others know more about the changes. Can you provide doc-related info in this and future PR descriptions? Thanks)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] github-actions[bot] commented on pull request #14450: Integrate package management service to pulsar functions service
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #14450:
URL: https://github.com/apache/pulsar/pull/14450#issuecomment-1049724946
@gaozhangmin:Thanks for providing doc info!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] gaozhangmin commented on pull request #14450: [Function] Integrate package management service to pulsar functionWorker
Posted by GitBox <gi...@apache.org>.
gaozhangmin commented on pull request #14450:
URL: https://github.com/apache/pulsar/pull/14450#issuecomment-1085335940
/pulsarbot run-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] gaozhangmin commented on a change in pull request #14450: [Function] Integrate package management service to pulsar functionWorker
Posted by GitBox <gi...@apache.org>.
gaozhangmin commented on a change in pull request #14450:
URL: https://github.com/apache/pulsar/pull/14450#discussion_r829699012
##########
File path: conf/functions_worker.yml
##########
@@ -26,6 +26,13 @@ workerHostname: localhost
workerPort: 6750
workerPortTls: 6751
+# The metadata store URL
+# Examples:
+# * zk:my-zk-1:2181,my-zk-2:2181,my-zk-3:2181
+# * my-zk-1:2181,my-zk-2:2181,my-zk-3:2181 (will default to ZooKeeper when the schema is not specified)
+# * zk:my-zk-1:2181,my-zk-2:2181,my-zk-3:2181/my-chroot-path (to add a ZK chroot path)
+metadataStoreUrl: zk:localhost:2181
Review comment:
metadataStoreUrl is used for BookKeeperPackagesStorage#initializeDlogNamespace。If we want to start BookKeeperPackagesStorage storage service in a separately running workers. we must support this configuration.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] gaozhangmin commented on pull request #14450: [Function] Integrate package management service to pulsar functionWorker
Posted by GitBox <gi...@apache.org>.
gaozhangmin commented on pull request #14450:
URL: https://github.com/apache/pulsar/pull/14450#issuecomment-1066080165
/pulsarbot run-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] gaozhangmin edited a comment on pull request #14450: [Function] Integrate package management service to pulsar functionWorker
Posted by GitBox <gi...@apache.org>.
gaozhangmin edited a comment on pull request #14450:
URL: https://github.com/apache/pulsar/pull/14450#issuecomment-1077554524
@freeznet I agree with you this pr brings breaking changes.
But for task one, Now, user can create or update a function/sink/source by:
1、local path to the JAR file for the function (if the function is written in Java).
2、`functionPkgUrl` which starts with http/https/file/package URL from packages management service
I think We should upload package to packageManagement service either we create functions by local path or `functionPkgUrl with http/file/https` if We config function worker to use packageManagement service,
Late, download package directly from packageManagement service.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] gaozhangmin removed a comment on pull request #14450: [Function] Integrate package management service to pulsar functionWorker
Posted by GitBox <gi...@apache.org>.
gaozhangmin removed a comment on pull request #14450:
URL: https://github.com/apache/pulsar/pull/14450#issuecomment-1065162951
/pulsarbot run-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] gaozhangmin removed a comment on pull request #14450: [Function] Integrate package management service to pulsar functionWorker
Posted by GitBox <gi...@apache.org>.
gaozhangmin removed a comment on pull request #14450:
URL: https://github.com/apache/pulsar/pull/14450#issuecomment-1066080165
/pulsarbot run-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] freeznet commented on a change in pull request #14450: [Function] Integrate package management service to pulsar functionWorker
Posted by GitBox <gi...@apache.org>.
freeznet commented on a change in pull request #14450:
URL: https://github.com/apache/pulsar/pull/14450#discussion_r830710026
##########
File path: pulsar-functions/worker/src/main/java/org/apache/pulsar/functions/worker/rest/api/ComponentImpl.java
##########
@@ -1403,7 +1406,16 @@ private StreamingOutput getStreamingOutput(String pkgPath) {
output.flush();
}
} else {
- WorkerUtils.downloadFromBookkeeper(worker().getDlogNamespace(), output, pkgPath);
Review comment:
why removed the `WorkerUtils.downloadFromBookkeeper` here and changed to `downloadPackageFile`? the `pkgPath` may not valid `packageURL` for the package management service. if you remove the support of bookkeeper as the function package store, it will bring breaking changes and no longer have backward compatibilities.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] gaozhangmin edited a comment on pull request #14450: [Function] Integrate package management service to pulsar functionWorker
Posted by GitBox <gi...@apache.org>.
gaozhangmin edited a comment on pull request #14450:
URL: https://github.com/apache/pulsar/pull/14450#issuecomment-1077554524
@freeznet I agree with you this pr brings breaking changes.
But for task one, Now, user can create or update a function/sink/source by:
1、local path to the JAR file for the function (if the function is written in Java).
2、`functionPkgUrl` which starts with http/https/file/package URL from packages management service
I think We should upload package to packageManagement service either we create functions by local path or `functionPkgUrl with http/file/https` if We config function worker to use packageManagement service,
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] gaozhangmin commented on pull request #14450: [Function] Integrate package management service to pulsar functionWorker
Posted by GitBox <gi...@apache.org>.
gaozhangmin commented on pull request #14450:
URL: https://github.com/apache/pulsar/pull/14450#issuecomment-1077554524
@freeznet For task one, Now, user can create or update a function/sink/source by:
1、local path to the JAR file for the function (if the function is written in Java).
2、`functionPkgUrl` which starts with http/https/file/package URL from packages management service
I think if We config function worker to use packageManagement service, We should upload package to it either we create functions by local path or `functionPkgUrl with http/file/https`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] gaozhangmin edited a comment on pull request #14450: [Function] Integrate package management service to pulsar functionWorker
Posted by GitBox <gi...@apache.org>.
gaozhangmin edited a comment on pull request #14450:
URL: https://github.com/apache/pulsar/pull/14450#issuecomment-1077554524
@freeznet I agree with you this pr brings breaking changes.
But for task one, My idea is, now user can create or update a function/sink/source by:
1、local path to the JAR file for the function (if the function is written in Java).
2、`functionPkgUrl` which starts with http/https/file/package URL from packages management service
We should upload package to packageManagement service either we provide a local path or `functionPkgUrl with http/file/https`. To use the unified package service api to read and write jar files
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] gaozhangmin edited a comment on pull request #14450: [Function] Integrate package management service to pulsar functionWorker
Posted by GitBox <gi...@apache.org>.
gaozhangmin edited a comment on pull request #14450:
URL: https://github.com/apache/pulsar/pull/14450#issuecomment-1077554524
@freeznet I agree with you this pr brings breaking changes.
But for task one, My idea is, now user can create or update a function/sink/source by:
1、local path to the JAR file for the function (if the function is written in Java).
2、`functionPkgUrl` which starts with http/https/file/package URL from packages management service
We should upload package to packageManagement service either we provide a local path or `functionPkgUrl with http/file/https`. If we still `uploadFileToBookkeeper` when we enable packageManagement. function worker still relays on bookkeeper.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] gaozhangmin removed a comment on pull request #14450: [Function] Integrate package management service to pulsar functionWorker
Posted by GitBox <gi...@apache.org>.
gaozhangmin removed a comment on pull request #14450:
URL: https://github.com/apache/pulsar/pull/14450#issuecomment-1083996403
/pulsarbot run-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] gaozhangmin commented on a change in pull request #14450: [Function] Integrate package management service to pulsar functionWorker
Posted by GitBox <gi...@apache.org>.
gaozhangmin commented on a change in pull request #14450:
URL: https://github.com/apache/pulsar/pull/14450#discussion_r829709921
##########
File path: conf/functions_worker.yml
##########
@@ -26,6 +26,13 @@ workerHostname: localhost
workerPort: 6750
workerPortTls: 6751
+# The metadata store URL
+# Examples:
+# * zk:my-zk-1:2181,my-zk-2:2181,my-zk-3:2181
+# * my-zk-1:2181,my-zk-2:2181,my-zk-3:2181 (will default to ZooKeeper when the schema is not specified)
+# * zk:my-zk-1:2181,my-zk-2:2181,my-zk-3:2181/my-chroot-path (to add a ZK chroot path)
+metadataStoreUrl: zk:localhost:2181
Review comment:
We can be back to use configurationMetadataStoreUrl if metadataStoreUrl is not set.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] gaozhangmin commented on pull request #14450: [Function] Integrate package management service to pulsar functionWorker
Posted by GitBox <gi...@apache.org>.
gaozhangmin commented on pull request #14450:
URL: https://github.com/apache/pulsar/pull/14450#issuecomment-1060291385
/pulsarbot run-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] gaozhangmin commented on pull request #14450: [Function] Integrate package management service to pulsar functionWorker
Posted by GitBox <gi...@apache.org>.
gaozhangmin commented on pull request #14450:
URL: https://github.com/apache/pulsar/pull/14450#issuecomment-1083996403
/pulsarbot run-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] gaozhangmin commented on pull request #14450: [Function] Integrate package management service to pulsar functionWorker
Posted by GitBox <gi...@apache.org>.
gaozhangmin commented on pull request #14450:
URL: https://github.com/apache/pulsar/pull/14450#issuecomment-1082563881
@freeznet PTAL. Unit test will be added lately.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] gaozhangmin commented on pull request #14450: [Function] Integrate package management service to pulsar functionWorker
Posted by GitBox <gi...@apache.org>.
gaozhangmin commented on pull request #14450:
URL: https://github.com/apache/pulsar/pull/14450#issuecomment-1072081144
/pulsarbot run-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] gaozhangmin commented on pull request #14450: [Function] Integrate package management service to pulsar functionWorker
Posted by GitBox <gi...@apache.org>.
gaozhangmin commented on pull request #14450:
URL: https://github.com/apache/pulsar/pull/14450#issuecomment-1072081144
/pulsarbot run-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] gaozhangmin commented on a change in pull request #14450: [Function] Integrate package management service to pulsar functionWorker
Posted by GitBox <gi...@apache.org>.
gaozhangmin commented on a change in pull request #14450:
URL: https://github.com/apache/pulsar/pull/14450#discussion_r829699012
##########
File path: conf/functions_worker.yml
##########
@@ -26,6 +26,13 @@ workerHostname: localhost
workerPort: 6750
workerPortTls: 6751
+# The metadata store URL
+# Examples:
+# * zk:my-zk-1:2181,my-zk-2:2181,my-zk-3:2181
+# * my-zk-1:2181,my-zk-2:2181,my-zk-3:2181 (will default to ZooKeeper when the schema is not specified)
+# * zk:my-zk-1:2181,my-zk-2:2181,my-zk-3:2181/my-chroot-path (to add a ZK chroot path)
+metadataStoreUrl: zk:localhost:2181
Review comment:
metadataStoreUrl is used for BookKeeperPackagesStorageConfiguration getMetadataStoreUrl。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] freeznet commented on pull request #14450: [Function] Integrate package management service to pulsar functionWorker
Posted by GitBox <gi...@apache.org>.
freeznet commented on pull request #14450:
URL: https://github.com/apache/pulsar/pull/14450#issuecomment-1080074152
@gaozhangmin
> 1、We only upload and download package from packageManagement service If We set `enablePackagesManagement=true` in broker and add a new configuration to function worker to set if we use packageManagement service to host package or not.
> 2、Remove the internal packageManagement service in function worker.
Thanks, lgtm
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] gaozhangmin removed a comment on pull request #14450: [Function] Integrate package management service to pulsar functionWorker
Posted by GitBox <gi...@apache.org>.
gaozhangmin removed a comment on pull request #14450:
URL: https://github.com/apache/pulsar/pull/14450#issuecomment-1065806087
/pulsarbot run-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] gaozhangmin removed a comment on pull request #14450: [Function] Integrate package management service to pulsar functionWorker
Posted by GitBox <gi...@apache.org>.
gaozhangmin removed a comment on pull request #14450:
URL: https://github.com/apache/pulsar/pull/14450#issuecomment-1060291385
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] gaozhangmin commented on pull request #14450: [Function] Integrate package management service to pulsar functionWorker
Posted by GitBox <gi...@apache.org>.
gaozhangmin commented on pull request #14450:
URL: https://github.com/apache/pulsar/pull/14450#issuecomment-1066128280
/pulsarbot run-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] gaozhangmin removed a comment on pull request #14450: [Function] Integrate package management service to pulsar functionWorker
Posted by GitBox <gi...@apache.org>.
gaozhangmin removed a comment on pull request #14450:
URL: https://github.com/apache/pulsar/pull/14450#issuecomment-1083016312
/pulsarbot run-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] gaozhangmin commented on a change in pull request #14450: [Function] Integrate package management service to pulsar functionWorker
Posted by GitBox <gi...@apache.org>.
gaozhangmin commented on a change in pull request #14450:
URL: https://github.com/apache/pulsar/pull/14450#discussion_r829699012
##########
File path: conf/functions_worker.yml
##########
@@ -26,6 +26,13 @@ workerHostname: localhost
workerPort: 6750
workerPortTls: 6751
+# The metadata store URL
+# Examples:
+# * zk:my-zk-1:2181,my-zk-2:2181,my-zk-3:2181
+# * my-zk-1:2181,my-zk-2:2181,my-zk-3:2181 (will default to ZooKeeper when the schema is not specified)
+# * zk:my-zk-1:2181,my-zk-2:2181,my-zk-3:2181/my-chroot-path (to add a ZK chroot path)
+metadataStoreUrl: zk:localhost:2181
Review comment:
metadataStoreUrl is used for BookKeeperPackagesStorageConfiguration `getMetadataStoreUrl`。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] gaozhangmin commented on pull request #14450: [Function] Integrate package management service to pulsar functionWorker
Posted by GitBox <gi...@apache.org>.
gaozhangmin commented on pull request #14450:
URL: https://github.com/apache/pulsar/pull/14450#issuecomment-1070506036
/pulsarbot run-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] nlu90 commented on a change in pull request #14450: [Function] Integrate package management service to pulsar functionWorker
Posted by GitBox <gi...@apache.org>.
nlu90 commented on a change in pull request #14450:
URL: https://github.com/apache/pulsar/pull/14450#discussion_r829289750
##########
File path: conf/functions_worker.yml
##########
@@ -26,6 +26,13 @@ workerHostname: localhost
workerPort: 6750
workerPortTls: 6751
+# The metadata store URL
+# Examples:
+# * zk:my-zk-1:2181,my-zk-2:2181,my-zk-3:2181
+# * my-zk-1:2181,my-zk-2:2181,my-zk-3:2181 (will default to ZooKeeper when the schema is not specified)
+# * zk:my-zk-1:2181,my-zk-2:2181,my-zk-3:2181/my-chroot-path (to add a ZK chroot path)
+metadataStoreUrl: zk:localhost:2181
Review comment:
what's the difference with the following `configurationMetadataStoreUrl`?
##########
File path: conf/functions_worker.yml
##########
@@ -45,6 +52,10 @@ metadataStoreCacheExpirySeconds: 300
################################
# Function package management
################################
+packagesManagementStorageProvider:
+ org.apache.pulsar.packages.management.storage.bookkeeper.BookKeeperPackagesStorageProvider
+packagesManagementLedgerRootPath: /ledgers
+packagesReplicas: 1
Review comment:
`broker.conf` and `standalone.conf` also have these same configurations.
If you added them here, what's the relationship between these duplicated configuration?
Do you want to remove them from other conf files or they will have some override relationship?
##########
File path: pulsar-broker/src/main/java/org/apache/pulsar/broker/PulsarService.java
##########
@@ -780,14 +780,14 @@ public void start() throws PulsarServerException {
acquireSLANamespace();
- // start function worker service if necessary
- this.startWorkerService(brokerService.getAuthenticationService(), brokerService.getAuthorizationService());
-
// start packages management service if necessary
if (config.isEnablePackagesManagement()) {
this.startPackagesManagementService();
}
Review comment:
If you make function worker depend on package management service, then shouldn't the package management service be always enabled when function worker is enabled?
Currently it seems they are separately configured, which may result the case that function worker is enabled but package management service is not running.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] gaozhangmin commented on a change in pull request #14450: [Function] Integrate package management service to pulsar functionWorker
Posted by GitBox <gi...@apache.org>.
gaozhangmin commented on a change in pull request #14450:
URL: https://github.com/apache/pulsar/pull/14450#discussion_r829971726
##########
File path: conf/functions_worker.yml
##########
@@ -26,6 +26,13 @@ workerHostname: localhost
workerPort: 6750
workerPortTls: 6751
+# The metadata store URL
+# Examples:
+# * zk:my-zk-1:2181,my-zk-2:2181,my-zk-3:2181
+# * my-zk-1:2181,my-zk-2:2181,my-zk-3:2181 (will default to ZooKeeper when the schema is not specified)
+# * zk:my-zk-1:2181,my-zk-2:2181,my-zk-3:2181/my-chroot-path (to add a ZK chroot path)
+metadataStoreUrl: zk:localhost:2181
Review comment:
@nlu90
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] gaozhangmin commented on pull request #14450: [Function] Integrate package management service to pulsar functionWorker
Posted by GitBox <gi...@apache.org>.
gaozhangmin commented on pull request #14450:
URL: https://github.com/apache/pulsar/pull/14450#issuecomment-1065162951
/pulsarbot run-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] gaozhangmin commented on pull request #14450: [Function] Integrate package management service to pulsar functionWorker
Posted by GitBox <gi...@apache.org>.
gaozhangmin commented on pull request #14450:
URL: https://github.com/apache/pulsar/pull/14450#issuecomment-1065806087
/pulsarbot run-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] gaozhangmin commented on pull request #14450: [Function] Integrate package management service to pulsar functionWorker
Posted by GitBox <gi...@apache.org>.
gaozhangmin commented on pull request #14450:
URL: https://github.com/apache/pulsar/pull/14450#issuecomment-1077584542
For backward compatibility, I want to make this changes. @freeznet
1、We only upload and download package from packageManagement service If We set `enablePackagesManagement=true` in broker and add a new configuration to function worker to set if we use packageManagement service to host package or not.
2、Remove the internal packageManagement service in function worker.
Thanks again for your reviews.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org