You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@samza.apache.org by "Cameron Lee (Confluence)" <no...@apache.org> on 2019/12/02 20:20:00 UTC
[CONF] Apache Samza > SEP-24: Cluster-based Job Coordinator
Dependency Isolation
There's **1 new comment** and **1 new edit** on this page
---
|
---
| | [](https://cwiki.apache.org/confluence/display/SAMZA/SEP-24%3A+Cluster-
based+Job+Coordinator+Dependency+Isolation?src=mail&src.mail.product=confluence-
server&src.mail.timestamp=1575318000544&src.mail.notification=com.atlassian.confluence.plugins.confluence-
notifications-batch-plugin%3Abatching-
notification&src.mail.recipient=8aa980875bf24635015c82a7d09902ac&src.mail.action=view
"page icon")
---
[SEP-24: Cluster-based Job Coordinator Dependency
Isolation](https://cwiki.apache.org/confluence/display/SAMZA/SEP-24%3A+Cluster-
based+Job+Coordinator+Dependency+Isolation?src=mail&src.mail.product=confluence-
server&src.mail.timestamp=1575318000544&src.mail.notification=com.atlassian.confluence.plugins.confluence-
notifications-batch-plugin%3Abatching-
notification&src.mail.recipient=8aa980875bf24635015c82a7d09902ac&src.mail.action=view
"SEP-24: Cluster-based Job Coordinator Dependency Isolation")
| 1 new comment
---
| | | |  | | Daniel
Nishimura
---
| | |
Any test plan guidance for this SEP?
---
|
| | |  | | Cameron Lee
---
| | |
Good catch. Added a new section.
---
| | | [View
comment](https://cwiki.apache.org/confluence/display/SAMZA/SEP-24%3A+Cluster-
based+Job+Coordinator+Dependency+Isolation?focusedCommentId=135863470&src=mail&src.mail.product=confluence-
server&src.mail.timestamp=1575318000544&src.mail.notification=com.atlassian.confluence.plugins.confluence-
notifications-batch-plugin%3Abatching-
notification&src.mail.recipient=8aa980875bf24635015c82a7d09902ac&src.mail.action=view#comment-135863470
"View comment") | •
---|---
[Reply](https://cwiki.apache.org/confluence/display/SAMZA/SEP-24%3A+Cluster-
based+Job+Coordinator+Dependency+Isolation?replyToComment=135863470&src=mail&src.mail.product=confluence-
server&src.mail.timestamp=1575318000544&src.mail.notification=com.atlassian.confluence.plugins.confluence-
notifications-batch-plugin%3Abatching-
notification&src.mail.recipient=8aa980875bf24635015c82a7d09902ac&src.mail.action=reply#comment-135863470
"Reply") | •
---|---
[Like](https://cwiki.apache.org/confluence/plugins/likes/like.action?contentId=135863470&src=mail&src.mail.product=confluence-
server&src.mail.timestamp=1575318000544&src.mail.notification=com.atlassian.confluence.plugins.confluence-
notifications-batch-plugin%3Abatching-
notification&src.mail.recipient=8aa980875bf24635015c82a7d09902ac&src.mail.action=like&jwt=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJzdWIiOiJ4c3JmOjhhYTk4MDg3NWJmMjQ2MzUwMTVjODJhN2QwOTkwMmFjIiwicXNoIjoiZGFjMzcyOWZjMDA4ZDIzNWFiMmNkZDEwNWUzYWUxNTY3NDVhNmYyOTM0MTc0YzQwMWUwYjQ4MTAwZDE1MDcxYiIsImlzcyI6ImNvbmZsdWVuY2Vfbm90aWZpY2F0aW9uc0FSRUgtWFVEMS1QT1FHLUNTQU8iLCJleHAiOjE1NzU5MjI4MDAsImlhdCI6MTU3NTMxODAwMH0.NLouy8qbiwKPHyuAEe8xBLOSPmTQSCHsJC4YB6tj4eo
"Like")
---
---
1 new edit
---
| | | |  | | Cameron Lee
edited this page
---
|
| | Here's what changed:
---
|
...
#### Generating the Samza API whitelist
In order to load the Samza API classes from the API classloader, we need to
tell _cytodynamics_ what those classes are. We can do this by providing a
whitelist of packages/classes when building the _cytodynamics_ classloader.
All public interfaces/classes inside of samza-api should be considered an API
class. One way to generate this whitelist is to use a Gradle task to find all
the classes from samza-api and put that list in a file. Then, that file can be
read by Samza when constructing the _cytodynamics_ classloader. The Gradle
task should also include classes from samza-kv.
...
Once we have general split deployment, this will no longer be a problem,
because the version of Samza used across all parts of the application will be
consistent.
# Testing
## Local testing
We can use samza-hello-samza to test this locally. It has scripts to set up
Zookeeper, Kafka, and YARN locally. The local YARN deployment will give the
process isolation necessary to test the AM.
1. Locally build the framework tarballs for API and infrastructure. It would be useful to put an example somewhere for how to build those tarballs.
2. Deploy Zookeeper, Kafka, and YARN locally ([https://samza.apache.org/startup/hello-samza/latest/](https://samza.apache.org/startup/hello-samza/latest/)).
3. Fill in certain configs (see [New configs](/confluence/pages/viewpage.action?pageId=135861548) above). These will go into the properties file passed to the run-app.sh script.
4. Create the tarball for the application ([https://samza.apache.org/startup/hello-samza/latest/](https://samza.apache.org/startup/hello-samza/latest/)). For testing local changes, remember to run the "publishToMavenLocal" command.
## Automated integration test
* Build API and infrastructure framework artifacts
* Build a simple test job with dependency isolation enabled
* This will require multiple configs, including the location of the framework artifacts for YARN resources (see [New configs](/confluence/pages/viewpage.action?pageId=135861548) above).
* Use the integration test framework (which uses real YARN) to check that the job runs successfully
# Alternative solutions
[Alternative solutions for
SEP-24](/confluence/pages/viewpage.action?pageId=135861951)
---
| | | [Go to page
history](https://cwiki.apache.org/confluence/pages/viewpreviousversions.action?pageId=135861548&src=mail&src.mail.product=confluence-
server&src.mail.timestamp=1575318000544&src.mail.notification=com.atlassian.confluence.plugins.confluence-
notifications-batch-plugin%3Abatching-
notification&src.mail.recipient=8aa980875bf24635015c82a7d09902ac "Go to page
history")
---
---
| [View
page](https://cwiki.apache.org/confluence/display/SAMZA/SEP-24%3A+Cluster-
based+Job+Coordinator+Dependency+Isolation?src=mail&src.mail.product=confluence-
server&src.mail.timestamp=1575318000544&src.mail.notification=com.atlassian.confluence.plugins.confluence-
notifications-batch-plugin%3Abatching-
notification&src.mail.recipient=8aa980875bf24635015c82a7d09902ac&src.mail.action=view)
---
| | [Stop watching
space](https://cwiki.apache.org/confluence/users/removespacenotification.action?spaceKey=SAMZA&src=mail&src.mail.product=confluence-
server&src.mail.timestamp=1575318000544&src.mail.notification=com.atlassian.confluence.plugins.confluence-
notifications-batch-plugin%3Abatching-
notification&src.mail.recipient=8aa980875bf24635015c82a7d09902ac&src.mail.action=stop-
watching&jwt=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJzdWIiOiJ4c3JmOjhhYTk4MDg3NWJmMjQ2MzUwMTVjODJhN2QwOTkwMmFjIiwicXNoIjoiZDY4YmUyN2FkZTFjZWI3YmVlNDM2NTViNWRhY2E5OWQ3M2FkYTAwOGQ1ZTY5ZWE3YmQ5YmE0NTdmYmVlNTkwMiIsImlzcyI6ImNvbmZsdWVuY2Vfbm90aWZpY2F0aW9uc0FSRUgtWFVEMS1QT1FHLUNTQU8iLCJleHAiOjE1NzU5MjI4MDAsImlhdCI6MTU3NTMxODAwMH0.kehLtSzE4FNheEEgSMakZamVdiiQiOCMjST7P8QLeOM)
| •
---|---
[Manage
notifications](https://cwiki.apache.org/confluence/users/editmyemailsettings.action?src=mail&src.mail.product=confluence-
server&src.mail.timestamp=1575318000544&src.mail.notification=com.atlassian.confluence.plugins.confluence-
notifications-batch-plugin%3Abatching-
notification&src.mail.recipient=8aa980875bf24635015c82a7d09902ac&src.mail.action=manage)
---
| 
---
This message was sent by Atlassian Confluence 6.15.8
---