You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by "dragon-zhang (via GitHub)" <gi...@apache.org> on 2024/01/26 15:17:16 UTC
[PR] [improve][fn] support WASM functions which can support many languages [pulsar]
dragon-zhang opened a new pull request, #21975:
URL: https://github.com/apache/pulsar/pull/21975
### Motivation
The wasm bytecode is designed to be encoded in a size- and load-time-efficient binary format. WebAssembly aims to execute at native speed by taking advantage of common hardware capabilities available on a wide range of platforms.
This PR support run WASM functions
### Verifying this change
see `pulsar-functions/java-examples/src/main/rust-function/README.md` and `org.apache.pulsar.functions.api.examples.WasmFunction`
- [ ] Make sure that the change passes the CI checks.
*(Please pick either of the following options)*
This change is a trivial rework / code cleanup without any test coverage.
*(or)*
This change is already covered by existing tests, such as *(please describe tests)*.
*(or)*
This change added tests and can be verified as follows:
*(example:)*
- *Added integration tests for end-to-end deployment with large payloads (10MB)*
- *Extended integration test for recovery after broker failure*
### Does this pull request potentially affect one of the following parts:
<!-- DO NOT REMOVE THIS SECTION. CHECK THE PROPER BOX ONLY. -->
*If the box was checked, please highlight the changes*
- [ ] Dependencies (add or upgrade a dependency)
- [x] The public API
- [ ] The schema
- [ ] The default values of configurations
- [ ] The threading model
- [ ] The binary protocol
- [ ] The REST endpoints
- [ ] The admin CLI options
- [ ] The metrics
- [ ] Anything that affects deployment
### Documentation
<!-- DO NOT REMOVE THIS SECTION. CHECK THE PROPER BOX ONLY. -->
- [ ] `doc` <!-- Your PR contains doc changes. -->
- [x] `doc-required` <!-- Your PR changes impact docs and you will update later -->
- [ ] `doc-not-needed` <!-- Your PR changes do not impact docs -->
- [ ] `doc-complete` <!-- Docs have been already added -->
### Matching PR in forked repository
PR in forked repository: <!-- ENTER URL HERE -->
<!--
After opening this PR, the build in apache/pulsar will fail and instructions will
be provided for opening a PR in the PR author's forked repository.
apache/pulsar pull requests should be first tested in your own fork since the
apache/pulsar CI based on GitHub Actions has constrained resources and quota.
GitHub Actions provides separate quota for pull requests that are executed in
a forked repository.
The tests will be run in the forked repository until all PR review comments have
been handled, the tests pass and the PR is approved by a reviewer.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [improve][fn] support WASM functions which can support many languages [pulsar]
Posted by "merlimat (via GitHub)" <gi...@apache.org>.
merlimat closed pull request #21975: [improve][fn] support WASM functions which can support many languages
URL: https://github.com/apache/pulsar/pull/21975
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [improve][fn] support WASM functions which can support many languages [pulsar]
Posted by "dragon-zhang (via GitHub)" <gi...@apache.org>.
dragon-zhang commented on PR #21975:
URL: https://github.com/apache/pulsar/pull/21975#issuecomment-1925187863
> Just for info, a pure wasm java runtime is now in development https://github.com/dylibso/chicory
>
> I did use it successfully in some POC:
>
> * https://lburgazzoli.github.io/posts/2024-02-01_apache_kafka_connect_meets_wasm_part_1/
> * https://lburgazzoli.github.io/posts/2024-01-14_apache_camel_meets_wasm_part_1/
It looks very good, what about the performance ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [improve][fn] support WASM functions which can support many languages [pulsar]
Posted by "lburgazzoli (via GitHub)" <gi...@apache.org>.
lburgazzoli commented on PR #21975:
URL: https://github.com/apache/pulsar/pull/21975#issuecomment-1928993883
> I would like to provide an SPI where submodules can be implemented using [wasmtime-java](https://github.com/kawamuray/wasmtime-java) and [chicory](https://github.com/dylibso/chicory) respectively :)
>
I'm not sure hot that would really be feasible given the low level APIs the various runtime expose, but if you come up with something I can definitively try it out and help with the chicory implementation.
> About pulsar-function-wasm-sdk, do we need implement similar SDKs for other languages? Of course, except for the existing SDK in pulsar functions.
>
> Then when a language is not supported by pulsar functions, we can quickly provide a lightweight implementation of pulsar functions sdk. When there is more time, we can continue to support the complete SDK.
>
> I think this is a good idea, do you have any other suggestions? @lburgazzoli
Given the current state of Wasm, an SDK is likely very needed otherwise each function would be required to re-implement the ABI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [improve][fn] support WASM functions which can support many languages [pulsar]
Posted by "dragon-zhang (via GitHub)" <gi...@apache.org>.
dragon-zhang commented on PR #21975:
URL: https://github.com/apache/pulsar/pull/21975#issuecomment-1925530828
Personally, I prefer [wasmtime-java](https://github.com/kawamuray/wasmtime-java), there's also no need to write native for bridge, and the low-level [wasmtime](https://github.com/bytecodealliance/wasmtime) is mature enough.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [improve][fn] support WASM functions which can support many languages [pulsar]
Posted by "dragon-zhang (via GitHub)" <gi...@apache.org>.
dragon-zhang commented on PR #21975:
URL: https://github.com/apache/pulsar/pull/21975#issuecomment-1933278909
I will start abstracting SPI interfaces as soon as possible after the Chinese New Year.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [improve][fn] support WASM functions which can support many languages [pulsar]
Posted by "dragon-zhang (via GitHub)" <gi...@apache.org>.
dragon-zhang commented on PR #21975:
URL: https://github.com/apache/pulsar/pull/21975#issuecomment-1926441328
Welcome, the cross platform capability of [chicory](https://github.com/dylibso/chicory) is very good due to java, as for the [wasmtime-java](https://github.com/kawamuray/wasmtime-java), it's only support linux-x86_64, windows-x86_64, macos-x86_64 and macos-aarch64. But now, almost all of servers are x86_64, so I overlooked this. @lburgazzoli
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [improve][fn] support WASM functions which can support many languages [pulsar]
Posted by "lburgazzoli (via GitHub)" <gi...@apache.org>.
lburgazzoli commented on PR #21975:
URL: https://github.com/apache/pulsar/pull/21975#issuecomment-1926972624
Here a very simple POC: https://github.com/lburgazzoli/pulsar-function-wasm
NOTES:
1. I'm by no mean suggesting that chicory is the runtime that should be adopted
2. since I'm involved in the chicory project, I'm doing a series of POC mainly to explore how to improve it so take it only as an experiment :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [improve][fn] support WASM functions which can support many languages [pulsar]
Posted by "lburgazzoli (via GitHub)" <gi...@apache.org>.
lburgazzoli commented on PR #21975:
URL: https://github.com/apache/pulsar/pull/21975#issuecomment-1925246410
At this stage I haven't run any performance measurement as both the runtime and the integrations are at at very early stage.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [improve][fn] support WASM functions which can support many languages [pulsar]
Posted by "lburgazzoli (via GitHub)" <gi...@apache.org>.
lburgazzoli commented on PR #21975:
URL: https://github.com/apache/pulsar/pull/21975#issuecomment-1924110739
Just for info, a pure wasm java runtime is now in development https://github.com/dylibso/chicory
I did use it successfully in some POC:
- https://lburgazzoli.github.io/posts/2024-02-01_apache_kafka_connect_meets_wasm_part_1/
- https://lburgazzoli.github.io/posts/2024-01-14_apache_camel_meets_wasm_part_1/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [improve][fn] support WASM functions which can support many languages [pulsar]
Posted by "dragon-zhang (via GitHub)" <gi...@apache.org>.
dragon-zhang commented on PR #21975:
URL: https://github.com/apache/pulsar/pull/21975#issuecomment-1916628718
see https://github.com/apache/pulsar/pull/21992 for doc
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [improve][fn] support WASM functions which can support many languages [pulsar]
Posted by "merlimat (via GitHub)" <gi...@apache.org>.
merlimat commented on PR #21975:
URL: https://github.com/apache/pulsar/pull/21975#issuecomment-6182263606
Hi @loongs-zhang, thanks for the WASM functions work and the discussion it started. As RobertIndie noted, a new function runtime needs a PIP or at least a discussion on dev@ first, and that hasn't happened yet. The build has also moved from Maven to Gradle since then, so the new module would need to be set up again. We're going through the backlog of open PRs (https://lists.apache.org/thread/cgy7w0qgnq8s7j0ykyzgrn3hj4nb67wf), so I'm closing this one. If you'd like to pursue it, a [DISCUSS] thread on dev@ is the way forward.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [improve][fn] support WASM functions which can support many languages [pulsar]
Posted by "dragon-zhang (via GitHub)" <gi...@apache.org>.
dragon-zhang commented on PR #21975:
URL: https://github.com/apache/pulsar/pull/21975#issuecomment-1928662335
I would like to provide an SPI where submodules can be implemented using [wasmtime-java](https://github.com/kawamuray/wasmtime-java) and [chicory](https://github.com/dylibso/chicory) respectively.
About pulsar-function-wasm-sdk, do we need implement similar SDKs for other languages? Of course, except for the existing SDK in pulsar functions.
When a language is not supported by pulsar functions, we can quickly provide a lightweight implementation of pulsar functions. When there is more time, we can continue to support the complete SDK.
I think this is a good idea, do you have any other suggestions? @lburgazzoli
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [improve][fn] support WASM functions which can support many languages [pulsar]
Posted by "RobertIndie (via GitHub)" <gi...@apache.org>.
RobertIndie commented on PR #21975:
URL: https://github.com/apache/pulsar/pull/21975#issuecomment-1913630739
This is a new feature for the Pulsar Function. Please create a PIP for this or start a discussion on the dev mailing list first. You could refer to [here](https://github.com/apache/pulsar/tree/master/pip) for how to submit a PIP.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [improve][fn] support WASM functions which can support many languages [pulsar]
Posted by "lburgazzoli (via GitHub)" <gi...@apache.org>.
lburgazzoli commented on PR #21975:
URL: https://github.com/apache/pulsar/pull/21975#issuecomment-1926410244
I think the biggest advantage of [chicory](https://github.com/dylibso/chicory) over [wasmtime-java](https://github.com/kawamuray/wasmtime-java) + [wasmtime](https://github.com/bytecodealliance/wasmtime) is the fact that the first is a pure meant to be a pure java implementation so it does not require to have any jni/ffi binding but yeah, in therm of maturity at this stage wasmtime is quite more mature.
For example purpose, I'll try to write a small POC for chicory + pulsar to see how it would looks like
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org