You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dolphinscheduler.apache.org by GitBox <gi...@apache.org> on 2020/11/04 14:21:47 UTC
[GitHub] [incubator-dolphinscheduler] gabrywu opened a new issue #4025: [Improvement][common] Improvement title
gabrywu opened a new issue #4025:
URL: https://github.com/apache/incubator-dolphinscheduler/issues/4025
**Describe the question**
For now, if we execute a yarn job in a SHELL script, we find the application IDs in the logs by regex 'application_\d+_\d+'.
I think it's so ugly and has performance issues. So I suggest that we register an aspect when executing 'yarn jar' command,
we can weave a join point to org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.submitApplication, where we can get the submitted application id and the tracking URL, and output them into one local file
**Which version of DolphinScheduler:**
- all version
**Describe alternatives you've considered**
add the following two env to global envs
`export YARN_CLIENT_OPTS="-javaagent:/pathto/aspectjweaver-1.9.6.jar"`
`export YARN_USER_CLASSPATH=/pathto/Aop2YarnClient-1.0-SNAPSHOT.jar`
Then when submitting applications to the yarn cluster, the aspect in Aop2YarnClient-1.0-SNAPSHOT.jar will be registered, and we can get the submitted application id and the tracking URL
This is an example, I just output the application id to console
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-dolphinscheduler] CalvinKirs commented on issue #4025: [Improvement][common] Improvement title
Posted by GitBox <gi...@apache.org>.
CalvinKirs commented on issue #4025:
URL: https://github.com/apache/incubator-dolphinscheduler/issues/4025#issuecomment-721775392
I think this is a good idea
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-dolphinscheduler] gabrywu commented on issue #4025: [Improvement][common] get application id in SHELL scripts
Posted by GitBox <gi...@apache.org>.
gabrywu commented on issue #4025:
URL: https://github.com/apache/incubator-dolphinscheduler/issues/4025#issuecomment-735339964
> > > it will not be able to fetch the applicationId in the case of use HiveServer2 submitting the SQL, should we consider storing the appId information in public storage? @gabrywu
> >
> >
> > Do you have any good ideas to resolve it? @xiejiajun
>
> I thought about writing the appId to a public storage such as Mysql, but it will introduce additional third-party service configuration such as JdbcUrl , so we still need to think about it carefully.
Yes, so the example project just put it to local file
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-dolphinscheduler] gabrywu edited a comment on issue #4025: [Improvement][common] get application id in SHELL scripts
Posted by GitBox <gi...@apache.org>.
gabrywu edited a comment on issue #4025:
URL: https://github.com/apache/incubator-dolphinscheduler/issues/4025#issuecomment-729464760
> it will not be able to fetch the applicationId in the case of use HiveServer2 submitting the SQL, should we consider storing the appId information in public storage? @gabrywu
Do you have any good ideas to resolve it? @xiejiajun
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-dolphinscheduler] gabrywu edited a comment on issue #4025: [Improvement][common] get application id in SHELL scripts
Posted by GitBox <gi...@apache.org>.
gabrywu edited a comment on issue #4025:
URL: https://github.com/apache/incubator-dolphinscheduler/issues/4025#issuecomment-735339964
> > > it will not be able to fetch the applicationId in the case of use HiveServer2 submitting the SQL, should we consider storing the appId information in public storage? @gabrywu
> >
> >
> > Do you have any good ideas to resolve it? @xiejiajun
>
> I thought about writing the appId to a public storage such as Mysql, but it will introduce additional third-party service configuration such as JdbcUrl , so we still need to think about it carefully.
Yes, so the example project just put it to a local file
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-dolphinscheduler] gabrywu commented on issue #4025: [Improvement][common] get application id in SHELL scripts
Posted by GitBox <gi...@apache.org>.
gabrywu commented on issue #4025:
URL: https://github.com/apache/incubator-dolphinscheduler/issues/4025#issuecomment-723387813
This is a public repo which can achieve this function, https://github.com/gabrywu/Aop2YarnClient
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-dolphinscheduler] xiejiajun commented on issue #4025: [Improvement][common] get application id in SHELL scripts
Posted by GitBox <gi...@apache.org>.
xiejiajun commented on issue #4025:
URL: https://github.com/apache/incubator-dolphinscheduler/issues/4025#issuecomment-730193031
> > it will not be able to fetch the applicationId in the case of use HiveServer2 submitting the SQL, should we consider storing the appId information in public storage? @gabrywu
>
> Do you have any good ideas to resolve it? @xiejiajun
I thought about writing the appId to a public storage such as Mysql, but it will introduce additional third-party service configuration such as JdbcUrl , so we still need to think about it carefully.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-dolphinscheduler] gabrywu commented on issue #4025: [Improvement][common] get application id in SHELL scripts
Posted by GitBox <gi...@apache.org>.
gabrywu commented on issue #4025:
URL: https://github.com/apache/incubator-dolphinscheduler/issues/4025#issuecomment-729464760
> it will not be able to fetch the applicationId in the case of use HiveServer2 submitting the SQL, should we consider storing the appId information in public storage? @gabrywu
Do you have any good ideas to resolve it?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-dolphinscheduler] xiejiajun commented on issue #4025: [Improvement][common] get application id in SHELL scripts
Posted by GitBox <gi...@apache.org>.
xiejiajun commented on issue #4025:
URL: https://github.com/apache/incubator-dolphinscheduler/issues/4025#issuecomment-723774758
it will not be able to fetch the applicationId in the case of use HiveServer2 submitting the SQL, should we consider storing the appId information in public storage? @gabrywu
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org