You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@submarine.apache.org by GitBox <gi...@apache.org> on 2020/02/19 10:21:31 UTC
[GitHub] [submarine] yuanzac opened a new pull request #187: Add yarn
submitter with classloader.
yuanzac opened a new pull request #187: Add yarn submitter with classloader.
URL: https://github.com/apache/submarine/pull/187
### What is this PR for?
Submitter dependencies are excluded from submarine server. Submarine rpc server should initialize yarn submitter through a classloader.
### What type of PR is it?
Improvement
### What is the Jira issue?
https://issues.apache.org/jira/browse/SUBMARINE-390
### How should this be tested?
https://travis-ci.org/yuanzac/hadoop-submarine/builds/652407783?utm_source=github_status&utm_medium=notification
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@submarine.apache.org
For additional commands, e-mail: dev-help@submarine.apache.org
[GitHub] [submarine] liuxunorg commented on a change in pull request #187:
Add yarn submitter with classloader.
Posted by GitBox <gi...@apache.org>.
liuxunorg commented on a change in pull request #187: Add yarn submitter with classloader.
URL: https://github.com/apache/submarine/pull/187#discussion_r381634873
##########
File path: bin/common.sh
##########
@@ -88,7 +88,7 @@ function download_mysql_jdbc_jar(){
echo "Mysql jdbc jar is downloaded and put in the path of submarine/lib."
}
-JAVA_OPTS+=" ${SUBMARINE_SERVER_JAVA_OPTS} -Dfile.encoding=UTF-8 ${SUBMARINE_SERVER_MEM}"
+JAVA_OPTS+=" -Dfile.encoding=UTF-8"
Review comment:
It's because `${SUBMARINE_SERVER_JAVA_OPTS}` and `${SUBMARINE_SERVER_MEM}"` are duplicated, so they need to be deleted, right?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@submarine.apache.org
For additional commands, e-mail: dev-help@submarine.apache.org
[GitHub] [submarine] yuanzac commented on a change in pull request #187:
SUBMARINE-390. Add yarn submitter with classloader
Posted by GitBox <gi...@apache.org>.
yuanzac commented on a change in pull request #187: SUBMARINE-390. Add yarn submitter with classloader
URL: https://github.com/apache/submarine/pull/187#discussion_r382468317
##########
File path: conf/submarine-site.xml
##########
@@ -143,6 +143,12 @@
<description>RuntimeFactory for Submarine jobs</description>
</property>
+ <property>
+ <name>submarine.server.remote.execution.enabled</name>
Review comment:
Done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@submarine.apache.org
For additional commands, e-mail: dev-help@submarine.apache.org
[GitHub] [submarine] liuxunorg commented on a change in pull request #187:
SUBMARINE-390. Add yarn submitter with classloader
Posted by GitBox <gi...@apache.org>.
liuxunorg commented on a change in pull request #187: SUBMARINE-390. Add yarn submitter with classloader
URL: https://github.com/apache/submarine/pull/187#discussion_r382368332
##########
File path: conf/submarine-site.xml
##########
@@ -143,6 +143,12 @@
<description>RuntimeFactory for Submarine jobs</description>
</property>
+ <property>
+ <name>submarine.server.remote.execution.enabled</name>
Review comment:
Amended to `submarine.server.prc.enabled`, will it be clearer?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@submarine.apache.org
For additional commands, e-mail: dev-help@submarine.apache.org
[GitHub] [submarine] yuanzac commented on a change in pull request #187: Add
yarn submitter with classloader.
Posted by GitBox <gi...@apache.org>.
yuanzac commented on a change in pull request #187: Add yarn submitter with classloader.
URL: https://github.com/apache/submarine/pull/187#discussion_r381730145
##########
File path: submarine-server/server-rpc/src/main/java/org/apache/submarine/server/rpc/SubmarineRpcServer.java
##########
@@ -109,11 +116,38 @@ private static ClientContext getClientContext(RpcContext rpcContext) {
ClientContext clientContext = new ClientContext();
clientContext.setYarnConfig(conf);
mergeSubmarineConfiguration(clientContext.getSubmarineConfig(), rpcContext);
+ ClassLoader classLoader = new URLClassLoader(constructUrlsFromClasspath("../lib/submitter"));
Review comment:
Yeah, agree with you. The code just adds jars in the submitter, not including jars in the submitter/submitter-k8s
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@submarine.apache.org
For additional commands, e-mail: dev-help@submarine.apache.org
[GitHub] [submarine] yuanzac commented on a change in pull request #187:
SUBMARINE-390. Add yarn submitter with classloader
Posted by GitBox <gi...@apache.org>.
yuanzac commented on a change in pull request #187: SUBMARINE-390. Add yarn submitter with classloader
URL: https://github.com/apache/submarine/pull/187#discussion_r382468196
##########
File path: submarine-server/server-rpc/src/main/java/org/apache/submarine/server/rpc/SubmarineRpcServer.java
##########
@@ -109,11 +118,45 @@ private static ClientContext getClientContext(RpcContext rpcContext) {
ClientContext clientContext = new ClientContext();
clientContext.setYarnConfig(conf);
mergeSubmarineConfiguration(clientContext.getSubmarineConfig(), rpcContext);
+ String runtimeclass =
Review comment:
Done.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@submarine.apache.org
For additional commands, e-mail: dev-help@submarine.apache.org
[GitHub] [submarine] yuanzac commented on a change in pull request #187:
SUBMARINE-390. Add yarn submitter with classloader
Posted by GitBox <gi...@apache.org>.
yuanzac commented on a change in pull request #187: SUBMARINE-390. Add yarn submitter with classloader
URL: https://github.com/apache/submarine/pull/187#discussion_r383068475
##########
File path: submarine-commons/commons-utils/src/main/java/org/apache/submarine/commons/utils/SubmarineConfVars.java
##########
@@ -44,10 +44,10 @@
SUBMARINE_SERVER_SSL_TRUSTSTORE_TYPE("submarine.server.ssl.truststore.type", null),
SUBMARINE_SERVER_SSL_TRUSTSTORE_PASSWORD("submarine.server.ssl.truststore.password", null),
SUBMARINE_CLUSTER_ADDR("submarine.cluster.addr", ""),
- SUBMARINE_SERVER_REMOTE_EXECUTION_ENABLED(
- "submarine.server.remote.execution.enabled", false),
- SUBMARINE_SERVER_REMOTE_EXECUTION_PORT(
- "submarine.server.remote.execution.port", 8980),
+ SUBMARINE_SERVER_RPC_ENABLED(
+ "submarine.server.rpc.enabled", false),
+ SUBMARINE_SERVER_RPC_PORT(
Review comment:
Done.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@submarine.apache.org
For additional commands, e-mail: dev-help@submarine.apache.org
[GitHub] [submarine] yuanzac commented on a change in pull request #187: Add
yarn submitter with classloader.
Posted by GitBox <gi...@apache.org>.
yuanzac commented on a change in pull request #187: Add yarn submitter with classloader.
URL: https://github.com/apache/submarine/pull/187#discussion_r381729370
##########
File path: submarine-server/server-rpc/src/main/java/org/apache/submarine/server/rpc/SubmarineRpcServer.java
##########
@@ -109,11 +116,38 @@ private static ClientContext getClientContext(RpcContext rpcContext) {
ClientContext clientContext = new ClientContext();
clientContext.setYarnConfig(conf);
mergeSubmarineConfiguration(clientContext.getSubmarineConfig(), rpcContext);
+ ClassLoader classLoader = new URLClassLoader(constructUrlsFromClasspath("../lib/submitter"));
+
RuntimeFactory runtimeFactory = RuntimeFactory.getRuntimeFactory(
- clientContext);
+ clientContext, classLoader);
clientContext.setRuntimeFactory(runtimeFactory);
return clientContext;
}
+ private static URL[] constructUrlsFromClasspath(String classpath) {
+ List<URL> urls = new ArrayList<>();
+ for (String path : classpath.split(File.pathSeparator)) {
+ if (path.endsWith("/*")) {
Review comment:
It is not a recurisive loading jar. Only the jars in the folder will be loaded. The jars in the subfolder will not.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@submarine.apache.org
For additional commands, e-mail: dev-help@submarine.apache.org
[GitHub] [submarine] asfgit closed pull request #187: SUBMARINE-390. Add
yarn submitter with classloader
Posted by GitBox <gi...@apache.org>.
asfgit closed pull request #187: SUBMARINE-390. Add yarn submitter with classloader
URL: https://github.com/apache/submarine/pull/187
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@submarine.apache.org
For additional commands, e-mail: dev-help@submarine.apache.org
[GitHub] [submarine] yuanzac commented on issue #187: Add yarn submitter
with classloader.
Posted by GitBox <gi...@apache.org>.
yuanzac commented on issue #187: Add yarn submitter with classloader.
URL: https://github.com/apache/submarine/pull/187#issuecomment-588600953
@jiwq,
Yeah, finally we will replace this logic with JobManager. But if we do this. we need to refactor the module structures to avoid dependency circles. After syncing with @liuxunorg and @tangzhankun last week, we decided to focus on HTTP API related development. For RPC part, we just make sure it works for what we have. When JobManager is done, it'll be used.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@submarine.apache.org
For additional commands, e-mail: dev-help@submarine.apache.org
[GitHub] [submarine] liuxunorg commented on a change in pull request #187:
Add yarn submitter with classloader.
Posted by GitBox <gi...@apache.org>.
liuxunorg commented on a change in pull request #187: Add yarn submitter with classloader.
URL: https://github.com/apache/submarine/pull/187#discussion_r381637005
##########
File path: submarine-server/server-rpc/src/main/java/org/apache/submarine/server/rpc/SubmarineRpcServer.java
##########
@@ -109,11 +116,38 @@ private static ClientContext getClientContext(RpcContext rpcContext) {
ClientContext clientContext = new ClientContext();
clientContext.setYarnConfig(conf);
mergeSubmarineConfiguration(clientContext.getSubmarineConfig(), rpcContext);
+ ClassLoader classLoader = new URLClassLoader(constructUrlsFromClasspath("../lib/submitter"));
Review comment:
The `../lib/submitter` path will store `submitter-yarn` and `submitter-k8s` separately,
If we load all jars in the `../lib/submitter` path at the same time, there may be conflicts.
I think the RpcServer only needs to load the jar in the `submitter-yarn` or `submitter-k8s` path.
What do you think?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@submarine.apache.org
For additional commands, e-mail: dev-help@submarine.apache.org
[GitHub] [submarine] liuxunorg commented on a change in pull request #187:
SUBMARINE-390. Add yarn submitter with classloader
Posted by GitBox <gi...@apache.org>.
liuxunorg commented on a change in pull request #187: SUBMARINE-390. Add yarn submitter with classloader
URL: https://github.com/apache/submarine/pull/187#discussion_r382368668
##########
File path: submarine-server/server-rpc/src/main/java/org/apache/submarine/server/rpc/SubmarineRpcServer.java
##########
@@ -109,11 +118,45 @@ private static ClientContext getClientContext(RpcContext rpcContext) {
ClientContext clientContext = new ClientContext();
clientContext.setYarnConfig(conf);
mergeSubmarineConfiguration(clientContext.getSubmarineConfig(), rpcContext);
+ String runtimeclass =
Review comment:
Change `runtimeclass` to `runtimeClass`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@submarine.apache.org
For additional commands, e-mail: dev-help@submarine.apache.org
[GitHub] [submarine] yuanzac commented on a change in pull request #187: Add
yarn submitter with classloader.
Posted by GitBox <gi...@apache.org>.
yuanzac commented on a change in pull request #187: Add yarn submitter with classloader.
URL: https://github.com/apache/submarine/pull/187#discussion_r381728290
##########
File path: bin/common.sh
##########
@@ -88,7 +88,7 @@ function download_mysql_jdbc_jar(){
echo "Mysql jdbc jar is downloaded and put in the path of submarine/lib."
}
-JAVA_OPTS+=" ${SUBMARINE_SERVER_JAVA_OPTS} -Dfile.encoding=UTF-8 ${SUBMARINE_SERVER_MEM}"
+JAVA_OPTS+=" -Dfile.encoding=UTF-8"
Review comment:
Because they are added in submarine-daemon.sh and submarine.sh. So we need to delete them to avoid deplications.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@submarine.apache.org
For additional commands, e-mail: dev-help@submarine.apache.org
[GitHub] [submarine] liuxunorg commented on a change in pull request #187:
Add yarn submitter with classloader.
Posted by GitBox <gi...@apache.org>.
liuxunorg commented on a change in pull request #187: Add yarn submitter with classloader.
URL: https://github.com/apache/submarine/pull/187#discussion_r381637314
##########
File path: submarine-server/server-rpc/src/main/java/org/apache/submarine/server/rpc/SubmarineRpcServer.java
##########
@@ -109,11 +116,38 @@ private static ClientContext getClientContext(RpcContext rpcContext) {
ClientContext clientContext = new ClientContext();
clientContext.setYarnConfig(conf);
mergeSubmarineConfiguration(clientContext.getSubmarineConfig(), rpcContext);
+ ClassLoader classLoader = new URLClassLoader(constructUrlsFromClasspath("../lib/submitter"));
+
RuntimeFactory runtimeFactory = RuntimeFactory.getRuntimeFactory(
- clientContext);
+ clientContext, classLoader);
clientContext.setRuntimeFactory(runtimeFactory);
return clientContext;
}
+ private static URL[] constructUrlsFromClasspath(String classpath) {
+ List<URL> urls = new ArrayList<>();
+ for (String path : classpath.split(File.pathSeparator)) {
+ if (path.endsWith("/*")) {
Review comment:
Maybe no recursive loading jar?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@submarine.apache.org
For additional commands, e-mail: dev-help@submarine.apache.org
[GitHub] [submarine] liuxunorg commented on a change in pull request #187:
SUBMARINE-390. Add yarn submitter with classloader
Posted by GitBox <gi...@apache.org>.
liuxunorg commented on a change in pull request #187: SUBMARINE-390. Add yarn submitter with classloader
URL: https://github.com/apache/submarine/pull/187#discussion_r382997260
##########
File path: submarine-commons/commons-utils/src/main/java/org/apache/submarine/commons/utils/SubmarineConfVars.java
##########
@@ -44,10 +44,10 @@
SUBMARINE_SERVER_SSL_TRUSTSTORE_TYPE("submarine.server.ssl.truststore.type", null),
SUBMARINE_SERVER_SSL_TRUSTSTORE_PASSWORD("submarine.server.ssl.truststore.password", null),
SUBMARINE_CLUSTER_ADDR("submarine.cluster.addr", ""),
- SUBMARINE_SERVER_REMOTE_EXECUTION_ENABLED(
- "submarine.server.remote.execution.enabled", false),
- SUBMARINE_SERVER_REMOTE_EXECUTION_PORT(
- "submarine.server.remote.execution.port", 8980),
+ SUBMARINE_SERVER_RPC_ENABLED(
+ "submarine.server.rpc.enabled", false),
+ SUBMARINE_SERVER_RPC_PORT(
Review comment:
@yuanzac I think need add config entity `submarine.server.rpc.port` to `conf/submarine-site.xml` and `conf/submarine-site.xml.template`
like.
```
<property>
<name>submarine.server.rpc.port</name>
<value>8980</value>
<description>Run jobs using rpc server pord.</description>
</property>
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@submarine.apache.org
For additional commands, e-mail: dev-help@submarine.apache.org
[GitHub] [submarine] liuxunorg commented on issue #187: Add yarn submitter
with classloader.
Posted by GitBox <gi...@apache.org>.
liuxunorg commented on issue #187: Add yarn submitter with classloader.
URL: https://github.com/apache/submarine/pull/187#issuecomment-588572683
@yuanzac
Can you add `SUBMARINE-390.` to PR title?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@submarine.apache.org
For additional commands, e-mail: dev-help@submarine.apache.org
[GitHub] [submarine] liuxunorg commented on issue #187: SUBMARINE-390. Add
yarn submitter with classloader
Posted by GitBox <gi...@apache.org>.
liuxunorg commented on issue #187: SUBMARINE-390. Add yarn submitter with classloader
URL: https://github.com/apache/submarine/pull/187#issuecomment-588992837
Yes, we prefer to provide external REST interfaces. The PRC interface will be perfected later.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@submarine.apache.org
For additional commands, e-mail: dev-help@submarine.apache.org