You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@zeppelin.apache.org by GitBox <gi...@apache.org> on 2022/09/07 13:44:59 UTC
[GitHub] [zeppelin] huage1994 opened a new pull request, #4459: [ZEPPELIN-5815] Frontend CI jobs improvement
huage1994 opened a new pull request, #4459:
URL: https://github.com/apache/zeppelin/pull/4459
### What is this PR for?
This PR is to improve a some step in Frontend CI jobs.
There are two command in this step in file `frontend.yml` as follow:
https://github.com/apache/zeppelin/blob/f42943c6d5bc04498b7fc9ca518dc433091a9c9c/.github/workflows/frontend.yml#L123-L125
The first command would `mvn install` many modules including the submodule of `zeppelin-plugins`.
The second command would `mvn package` `zeppelin-plugins`.
Actually, we only need to `mvn package` module `zeppelin-plugins` one time instead of install then package it, which would save running time of CI.
And there is no need to install module `zeppelin-plugins` , which would bring something unexpected related to dependency when I upgrade junit 4 to junit 5.
### What type of PR is it?
Improvement
### Todos
* [ ] - Task
### What is the Jira issue?
* Open an issue on Jira [ZEPPELIN-5815](https://issues.apache.org/jira/browse/ZEPPELIN-5815)
### How should this be tested?
CI passed
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need to update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [zeppelin] Reamer commented on pull request #4459: [ZEPPELIN-5815] Frontend CI jobs improvement
Posted by GitBox <gi...@apache.org>.
Reamer commented on PR #4459:
URL: https://github.com/apache/zeppelin/pull/4459#issuecomment-1248158107
Currently, the compilation of the plugins is quite fast. Where did you notice that it takes a long time?
Source: https://github.com/apache/zeppelin/actions/runs/3059081927/jobs/4936042114
```
[INFO] Zeppelin: Plugins Parent ........................... SUCCESS [ 4.729 s]
[INFO] Zeppelin: Plugin S3NotebookRepo .................... SUCCESS [ 5.785 s]
[INFO] Zeppelin: Plugin GitHubNotebookRepo ................ SUCCESS [ 5.088 s]
[INFO] Zeppelin: Plugin AzureNotebookRepo ................. SUCCESS [ 4.792 s]
[INFO] Zeppelin: Plugin GCSNotebookRepo ................... SUCCESS [ 5.270 s]
[INFO] Zeppelin: Plugin FileSystemNotebookRepo ............ SUCCESS [ 5.207 s]
[INFO] Zeppelin: Plugin MongoNotebookRepo ................. SUCCESS [ 5.177 s]
[INFO] Zeppelin: Plugin OSSNotebookRepo ................... SUCCESS [ 5.037 s]
[INFO] Zeppelin: Plugin Kubernetes StandardLauncher ....... SUCCESS [ 9.391 s]
[INFO] Zeppelin: Plugin Flink Launcher .................... SUCCESS [ 6.069 s]
[INFO] Zeppelin: Plugin Docker Launcher ................... SUCCESS [ 5.728 s]
[INFO] Zeppelin: Plugin Cluster Launcher .................. SUCCESS [ 5.420 s]
[INFO] Zeppelin: Plugin Yarn Launcher ..................... SUCCESS [ 5.052 s]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [zeppelin] Reamer commented on pull request #4459: [ZEPPELIN-5815] Frontend CI jobs improvement
Posted by GitBox <gi...@apache.org>.
Reamer commented on PR #4459:
URL: https://github.com/apache/zeppelin/pull/4459#issuecomment-1274532079
Sorry if I was vague here. Let me describe it in a little more detail.
During the tests the project `zeppelin-integration` is executed. In my opinion this sub-module has to define all dependencies in the `pom.xml` so that the tests can be executed successfully, without install the whole zeppelin project via `mvn install` before.
Unfortunately this is not yet the case, because only a few dependencies are needed during the installation.
```
./mvnw package -pl zeppelin-integration -Pintegration -am -DskipTests
[INFO] Scanning for projects...
[WARNING] The project org.apache.zeppelin:zeppelin-integration:jar:0.11.0-SNAPSHOT uses prerequisites which is only intended for maven-plugin projects but not for non maven-plugin projects. For such purposes you should use the maven-enforcer-plugin. See https://maven.apache.org/enforcer/enforcer-rules/requireMavenVersion.html
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Build Order:
[INFO]
[INFO] Zeppelin [pom]
[INFO] Zeppelin: Common [jar]
[INFO] Zeppelin: Interpreter [jar]
[INFO] Zeppelin: Jupyter Support [jar]
[INFO] Zeppelin: Zengine [jar]
[INFO] Zeppelin: Integration Test [jar]
```
My point is that it may not be necessary to install the Zeppelin plugins at all.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [zeppelin] zjffdu commented on pull request #4459: [ZEPPELIN-5815] Frontend CI jobs improvement
Posted by GitBox <gi...@apache.org>.
zjffdu commented on PR #4459:
URL: https://github.com/apache/zeppelin/pull/4459#issuecomment-1270992699
Frontend CI is passed，Thanks @huage1994 LGTM
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [zeppelin] huage1994 commented on pull request #4459: [ZEPPELIN-5815] Frontend CI jobs improvement
Posted by GitBox <gi...@apache.org>.
huage1994 commented on PR #4459:
URL: https://github.com/apache/zeppelin/pull/4459#issuecomment-1272967709
> Zeppelin uses maven 3.8.1 . mvn -pl parent-module would ignores any child modules.
`/mvnw clean package -pl zeppelin-plugins -amd -B`
`-amd` is a trick to solve this problem.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [zeppelin] huage1994 commented on pull request #4459: [ZEPPELIN-5815] Frontend CI jobs improvement
Posted by GitBox <gi...@apache.org>.
huage1994 commented on PR #4459:
URL: https://github.com/apache/zeppelin/pull/4459#issuecomment-1291453956
Hi @Reamer @jongyoul ,
I'm sorry I deleted this branch by mistake，I would submit a new PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [zeppelin] huage1994 closed pull request #4459: [ZEPPELIN-5815] Frontend CI jobs improvement
Posted by GitBox <gi...@apache.org>.
huage1994 closed pull request #4459: [ZEPPELIN-5815] Frontend CI jobs improvement
URL: https://github.com/apache/zeppelin/pull/4459
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [zeppelin] huage1994 commented on pull request #4459: [ZEPPELIN-5815] Frontend CI jobs improvement
Posted by GitBox <gi...@apache.org>.
huage1994 commented on PR #4459:
URL: https://github.com/apache/zeppelin/pull/4459#issuecomment-1249597925
> Currently, the compilation of the plugins is quite fast. Where did you notice that it takes a long time? Source:
Hi @Reamer, I never thought `mvn package zeppelin-plugins` would take much time.
Currently, front end CI would firstly `mvn install zeppelin-plugins` and then `mvn package zeppelin-plugins`. This is not necessary. This PR is to remove `mvn install zeppelin-plugins`.
> Currently there are two command in this step in file `frontend.yml` as follow:
> https://github.com/apache/zeppelin/blob/f42943c6d5bc04498b7fc9ca518dc433091a9c9c/.github/workflows/frontend.yml#L123-L125
> The first command would `mvn install` many modules including the submodule of `zeppelin-plugins`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [zeppelin] huage1994 commented on pull request #4459: [ZEPPELIN-5815] Frontend CI jobs improvement
Posted by GitBox <gi...@apache.org>.
huage1994 commented on PR #4459:
URL: https://github.com/apache/zeppelin/pull/4459#issuecomment-1249612266
Maybe I didn't write clear about the following description, which may mislead you.
Actually PR #4279 is not related to this PR in terms of content. But this PR would help to solve the blocking problem mentioned in following description in PR #4279 and I don't know why, so I put the following description here to ask for help.
> By the way when I upgrade junit 4 to junit 5 https://github.com/apache/zeppelin/pull/4279 , Frontend CI job would block at downloading dependency when mvn install the module k8s-standard, but it is ok to mvn package this module.
> It 's really weird that both mvn install and mvn package is OK in my local environment, this problem only happens in Github CI jobs .
> It has costed me very much time, but hard to find the real reason.I would really appreciate if someone tell me the reason.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [zeppelin] huage1994 commented on pull request #4459: [ZEPPELIN-5815] Frontend CI jobs improvement
Posted by GitBox <gi...@apache.org>.
huage1994 commented on PR #4459:
URL: https://github.com/apache/zeppelin/pull/4459#issuecomment-1272962124
> In general, I am against additional Maven calls. Maybe it makes sense to work with the Maven parameters "-am" and "-pl" in the first call. This should build only the most necessary artefacts, which have been specified with "pl". See: https://maven.apache.org/guides/mini/guide-multiple-modules-4.html#command-line-options
Hi @Reamer , I don't know if I get your point.
Zeppelin uses maven 3.8.1
If we only use "-pl" instead of `-DskipZeppelinPlugins` here, we have to specify all 11 children modules of `zeppelin-plugins` like this:
`!zeppelin-plugins/launcher/cluster//,!zeppelin-plugins/launer/docker,!zeppelin-plugins/notebookrepo/azure,...`
That's the reason I add `-DskipZeppelinPlugins` here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [zeppelin] Reamer commented on pull request #4459: [ZEPPELIN-5815] Frontend CI jobs improvement
Posted by GitBox <gi...@apache.org>.
Reamer commented on PR #4459:
URL: https://github.com/apache/zeppelin/pull/4459#issuecomment-1272467375
In general, I am against additional Maven calls. Maybe it makes sense to work with the Maven parameters "-am" and "-pl" in the first call.
This should build only the most necessary artefacts, which have been specified with "pl".
See: https://maven.apache.org/guides/mini/guide-multiple-modules-4.html#command-line-options
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org