You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@zeppelin.apache.org by GitBox <gi...@apache.org> on 2021/10/05 11:02:58 UTC
[GitHub] [zeppelin] martin-g opened a new pull request #4243: ZEPPELIN-5543 Use TravisCI for testing on Linux ARM64
martin-g opened a new pull request #4243:
URL: https://github.com/apache/zeppelin/pull/4243
### What is this PR for?
Build and test on Linux ARM64 at TravisCI.
It uses [Partner Queue Solution](https://docs.travis-ci.com/user/billing-overview/#partner-queue-solution) at Equinix Metal ARM64 nodes, so the jobs do **not** consume build credits!
The PR is based on the latest version of `.travis.yml` from `branch-0.9` and the Miniconda3 with Python 3.8 with the packages listed in `testing/env_python_3.8_with_R.yml`
### What type of PR is it?
Improvement
### Todos
* [ ] - Confirm that the build passes after the merge of https://github.com/apache/zeppelin/pull/4237. Currently most of the jobs fail due to the usage of PhantomJS
* [ ] - Debug an issue in the PySpark jobs :
```
testSharedInterpreter(org.apache.zeppelin.livy.LivyInterpreterIT) Time elapsed: 26.146 sec <<< FAILURE!
java.lang.AssertionError: %text No module named matplotlib.pyplot
Traceback (most recent call last):
ImportError: No module named matplotlib.pyplot
expected:<SUCCESS> but was:<ERROR>
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:834)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.apache.zeppelin.livy.LivyInterpreterIT.testSharedInterpreter(LivyInterpreterIT.java:807)
INFO [2021-10-05 09:37:15,548] ({Thread-0} Logging.scala[info]:39) - Shutting down cluster pool.
Results :
Failed tests:
LivyInterpreterIT.testSharedInterpreter:807 %text No module named matplotlib.pyplot
Traceback (most recent call last):
ImportError: No module named matplotlib.pyplot
expected:<SUCCESS> but was:<ERROR>
```
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-5543
### How should this be tested?
* Make sure that the build at TravisCI passes!
### Questions:
* Does the licenses files need update? - NO
* Is there breaking changes for older versions? - NO
* Does this needs documentation? - Update the `How to contribute documents` to mention that Linux ARM64 is being tested
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [zeppelin] martin-g commented on pull request #4243: ZEPPELIN-5543 Use TravisCI for testing on Linux ARM64
Posted by GitBox <gi...@apache.org>.
martin-g commented on pull request #4243:
URL: https://github.com/apache/zeppelin/pull/4243#issuecomment-968622440
I've started working few times on this PR but the build is very unstable on Travis.
Part of the reason is also that I don't quite get the build setup. As a Maven project I expect that `mvn clean install` should do everything that is needed. But it fails because some Maven modules cannot find other modules. One has to `mvn install -DskipTests` first and then execute `mvn test` in a second pass.
Last Thursday there was a [meeting](https://cwiki.apache.org/confluence/x/FJBnCw) with Apache Infra where we discussed how we could use Linux ARM64 build nodes for Github Actions CI. The idea is to use self-hosted Linux ARM64 runner + custom Github Action runner with security enhancements to prevent abuses from contributors. Also see https://cwiki.apache.org/confluence/display/INFRA/GitHub+-+self-hosted+runners
This way Zeppelin will continue using just Github Actions as a CI.
I'll let you know once I have a working POC!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [zeppelin] martin-g commented on pull request #4243: ZEPPELIN-5543 Use TravisCI for testing on Linux ARM64
Posted by GitBox <gi...@apache.org>.
martin-g commented on pull request #4243:
URL: https://github.com/apache/zeppelin/pull/4243#issuecomment-943033007
Yes! It works fine on my fork!
Let me investigate!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [zeppelin] martin-g commented on pull request #4243: ZEPPELIN-5543 Use TravisCI for testing on Linux ARM64
Posted by GitBox <gi...@apache.org>.
martin-g commented on pull request #4243:
URL: https://github.com/apache/zeppelin/pull/4243#issuecomment-989819219
@zjffdu Did you have a chance to take a look at the GitHub Actions self-hosted runners on Oracle Cloud approach ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [zeppelin] zjffdu commented on pull request #4243: ZEPPELIN-5543 Use TravisCI for testing on Linux ARM64
Posted by GitBox <gi...@apache.org>.
zjffdu commented on pull request #4243:
URL: https://github.com/apache/zeppelin/pull/4243#issuecomment-942379372
#4237 is merged, @martin-g Could you rebase this PR ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [zeppelin] martin-g commented on pull request #4243: ZEPPELIN-5543 Use TravisCI for testing on Linux ARM64
Posted by GitBox <gi...@apache.org>.
martin-g commented on pull request #4243:
URL: https://github.com/apache/zeppelin/pull/4243#issuecomment-1048633032
I was waiting for https://github.com/apache/zeppelin/commit/d8d18f59b2926c90b34b5cfb27a3c2c3174f62a5 to be merged.
I will resume/restart my work on this in the coming days!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [zeppelin] martin-g commented on pull request #4243: ZEPPELIN-5543 Use TravisCI for testing on Linux ARM64
Posted by GitBox <gi...@apache.org>.
martin-g commented on pull request #4243:
URL: https://github.com/apache/zeppelin/pull/4243#issuecomment-992525985
I've setup 4 self-hosted runners on my ARM64 VM at Oracle Cloud.
With https://github.com/martin-g/zeppelin/pull/1/files I've started using them.
But there is a problem with Miniconda: https://github.com/martin-g/zeppelin/runs/4507098156?check_suite_focus=true
```
Error: No installed conda 'base' enviroment found at
```
I've asked for help at https://github.com/conda-incubator/setup-miniconda/issues/203#issuecomment-992520979
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [zeppelin] martin-g closed pull request #4243: ZEPPELIN-5543 Use TravisCI for testing on Linux ARM64
Posted by GitBox <gi...@apache.org>.
martin-g closed pull request #4243:
URL: https://github.com/apache/zeppelin/pull/4243
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [zeppelin] zjffdu commented on pull request #4243: ZEPPELIN-5543 Use TravisCI for testing on Linux ARM64
Posted by GitBox <gi...@apache.org>.
zjffdu commented on pull request #4243:
URL: https://github.com/apache/zeppelin/pull/4243#issuecomment-992065653
Sorry for late response @martin-g , this approach LGTM, have you tried to build Zeppelin in oracle cloud?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [zeppelin] martin-g commented on pull request #4243: ZEPPELIN-5543 Use TravisCI for testing on Linux ARM64
Posted by GitBox <gi...@apache.org>.
martin-g commented on pull request #4243:
URL: https://github.com/apache/zeppelin/pull/4243#issuecomment-943350256
Apache Infra team enabled it: https://issues.apache.org/jira/browse/INFRA-22411
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [zeppelin] zjffdu commented on pull request #4243: ZEPPELIN-5543 Use TravisCI for testing on Linux ARM64
Posted by GitBox <gi...@apache.org>.
zjffdu commented on pull request #4243:
URL: https://github.com/apache/zeppelin/pull/4243#issuecomment-1048625766
@martin-g Are you still working on it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [zeppelin] martin-g commented on pull request #4243: ZEPPELIN-5543 Use TravisCI for testing on Linux ARM64
Posted by GitBox <gi...@apache.org>.
martin-g commented on pull request #4243:
URL: https://github.com/apache/zeppelin/pull/4243#issuecomment-970135140
@zjffdu I've documented the approach at https://martin-grigorov.medium.com/github-actions-arm64-runner-on-oracle-cloud-a77cdf7a325a
If you are interested in this solution then you will need to contact Apache Infra for the token needed by `./config.sh` and to setup the security/approvals. Everything else could be done by any member of the Zeppelin team!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [zeppelin] zjffdu edited a comment on pull request #4243: ZEPPELIN-5543 Use TravisCI for testing on Linux ARM64
Posted by GitBox <gi...@apache.org>.
zjffdu edited a comment on pull request #4243:
URL: https://github.com/apache/zeppelin/pull/4243#issuecomment-943025657
@martin-g Have you tried use travis CI for your forked zeppelin repo first ?
https://docs.travis-ci.com/user/tutorial/#to-get-started-with-travis-ci-using-github
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [zeppelin] zjffdu commented on pull request #4243: ZEPPELIN-5543 Use TravisCI for testing on Linux ARM64
Posted by GitBox <gi...@apache.org>.
zjffdu commented on pull request #4243:
URL: https://github.com/apache/zeppelin/pull/4243#issuecomment-968630934
Thanks @martin-g
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [zeppelin] martin-g commented on pull request #4243: ZEPPELIN-5543 Use TravisCI for testing on Linux ARM64
Posted by GitBox <gi...@apache.org>.
martin-g commented on pull request #4243:
URL: https://github.com/apache/zeppelin/pull/4243#issuecomment-943008904
@zjffdu I've rebased the branch but I don't see the TravisCI check. Should it be enabled manually via GitHub UI ?!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [zeppelin] martin-g commented on pull request #4243: ZEPPELIN-5543 Use TravisCI for testing on Linux ARM64
Posted by GitBox <gi...@apache.org>.
martin-g commented on pull request #4243:
URL: https://github.com/apache/zeppelin/pull/4243#issuecomment-1048633761
Closing this PR because TravisCI won't be used.
I will work on self-hosted runner for Github Actions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [zeppelin] zjffdu commented on pull request #4243: ZEPPELIN-5543 Use TravisCI for testing on Linux ARM64
Posted by GitBox <gi...@apache.org>.
zjffdu commented on pull request #4243:
URL: https://github.com/apache/zeppelin/pull/4243#issuecomment-942379372
#4237 is merged, @martin-g Could you rebase this PR ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [zeppelin] martin-g commented on pull request #4243: ZEPPELIN-5543 Use TravisCI for testing on Linux ARM64
Posted by GitBox <gi...@apache.org>.
martin-g commented on pull request #4243:
URL: https://github.com/apache/zeppelin/pull/4243#issuecomment-992212839
I haven't tried yet!
Before I do it I want to explain something that was a stopper for another project:
When creating an account at Oracle Cloud (OCI) they will ask you for a credit card details. It won't be charged if you use the Free Tier (and this is enough for testing on ARM64)! But if you want to share this OCI account with the rest of the Zeppelin team then this might be problematic, because someone may add a paid resource to the account and OCI will start charging your card!
If the above is a problem for Zeppelin too then I could offer you a VM donated by my employer (OpenLab Testing) that could be used the same way!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [zeppelin] martin-g commented on pull request #4243: ZEPPELIN-5543 Use TravisCI for testing on Linux ARM64
Posted by GitBox <gi...@apache.org>.
martin-g commented on pull request #4243:
URL: https://github.com/apache/zeppelin/pull/4243#issuecomment-997649375
OK! I will request a VM for Zeppelin project!
In the meantime I have started testing at OracleCloud with Github Actions self-hosted runner and I've found some issues with setup-miniconda action but I was able to work them around!
The latest state could be seen at https://github.com/martin-g/zeppelin/runs/4560861885?check_suite_focus=true:
```
Tests run: 2, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 0.452 sec <<< FAILURE! - in org.apache.zeppelin.notebook.repo.MongoNotebookRepoTest
testGetNotePath(org.apache.zeppelin.notebook.repo.MongoNotebookRepoTest) Time elapsed: 0.198 sec <<< ERROR!
java.lang.IllegalArgumentException: this version does not support 32Bit: PRODUCTION:Linux:B32
```
Embedded Mongo supports Linux ARM64 since recently: https://github.com/flapdoodle-oss/de.flapdoodle.embed.mongo/pull/343
I will open new PRs for the new improvements!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [zeppelin] zjffdu commented on pull request #4243: ZEPPELIN-5543 Use TravisCI for testing on Linux ARM64
Posted by GitBox <gi...@apache.org>.
zjffdu commented on pull request #4243:
URL: https://github.com/apache/zeppelin/pull/4243#issuecomment-997367883
> I haven't tried yet! Before I do it I want to explain something that was a stopper for another project:
>
> When creating an account at Oracle Cloud (OCI) they will ask you for a credit card details. It won't be charged if you use the Free Tier (and this is enough for testing on ARM64)! But if you want to share this OCI account with the rest of the Zeppelin team then this might be problematic, because someone may add a paid resource to the account and OCI will start charging your card!
>
> If the above is a problem for Zeppelin too then I could offer you a VM donated by my employer (OpenLab Testing) that could be used the same way!
This is indeed a problem, it would be great if your employer can help with that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [zeppelin] zjffdu commented on pull request #4243: ZEPPELIN-5543 Use TravisCI for testing on Linux ARM64
Posted by GitBox <gi...@apache.org>.
zjffdu commented on pull request #4243:
URL: https://github.com/apache/zeppelin/pull/4243#issuecomment-997367752
Thanks for your update @martin-g
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [zeppelin] zjffdu commented on pull request #4243: ZEPPELIN-5543 Use TravisCI for testing on Linux ARM64
Posted by GitBox <gi...@apache.org>.
zjffdu commented on pull request #4243:
URL: https://github.com/apache/zeppelin/pull/4243#issuecomment-943025657
@martin-g Have you tried use travis CI for your own repo first ?
https://docs.travis-ci.com/user/tutorial/#to-get-started-with-travis-ci-using-github
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org