You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@storm.apache.org by Rahul Aggarwal <ra...@puresoftware.com> on 2020/06/22 09:26:09 UTC
Adding arm64 support to travis-ci in storm
Hi Team,
I am adding arm64 support to storm's travis-ci with jdk 8 and 11.
During build, I am getting an error related to mock as below :
ImportError: No module named mock
I have manually installed mock via pip install mock and pip3 install mock.
But the error is still the same.
Can you please suggest a resolution.
Thanks & Regards
Rahul Aggarwal
--
*Disclaimer* -The information transmitted is intended solely for the
individual
or entity to which it is addressed and may contain confidential
and/or
privileged material. Any review, re-transmission, dissemination or
other use of
or taking action in reliance upon this information by persons
or entities other
than the intended recipient is prohibited. If you have
received this email in
error please contact the sender and delete the
material from any computer. In
such instances you are further prohibited
from reproducing, disclosing,
distributing or taking any action in reliance
on it.As a recipient of this email,
you are responsible for screening its
contents and the contents of any
attachments for the presence of viruses.
No liability is accepted for any
damages caused by any virus transmitted by
this email.
Re: Adding arm64 support to travis-ci in storm
Posted by Sheng Liu <li...@gmail.com>.
Hi Rahul,
Yes, the issues #1, #2 and #3 Yikun and I have also met and mentioned
previously, we have tried adding some workarounds to fix, but also met the
problems as your Travis CI logs show. according to the logs, we guess it is
related to Travis ARM64 support itself, as mentioned in last mail.
Best Regards.
Rahul Aggarwal <ra...@puresoftware.com> 于2020年6月27日周六 下午9:38写道：
> Hi there,
>
> Thank you so much for this detailed information. I am glad to have a quick
> response from you.
> I got busy with my routine office stuff, so could not focus much on this
> package this whole week.
> Actually I am more focused towards adding arm64 support to travis-ci only,
> because of my professional commitments.
> I have tried something like this :
> 1. As pip and pip3 were causing issues, hence I changed the python version
> from 2.7 to 3. Now mock is getting installed successfully. Alternatively,
> we can also install mock using pip2 on python 2.7 .
> 2. Also, JAVA_HOME was getting set with jre, not jdk. So, I manually
> installed openjdk-11-jdk and then set JAVA_HOME according to it. That issue
> has been cleared now.
> 3. After that, I am getting issues related to swapon file permissions, and
> an unexpected interrupt in 3 out of 6 arm64 tests on travis-ci. I am
> linking my logs below. Plz have a look at those test cases with jdk 11 on
> arm64.
> TRAVIS-CI logs : https://travis-ci.com/github/rahulgit-ps/storm
>
> I will keep sharing my findings. Hoping the same :)
>
> Thanks and Regards
> Rahul Aggarwal
>
> On Tue, Jun 23, 2020 at 7:05 AM Yikun Jiang <yi...@gmail.com> wrote:
>
> > Hi,
> >
> > We have some invistatgation on ARM CI support of STORM.
> > https://github.com/Yikun/storm/pull/2/files
> >
> > Some introduction of our invesitagation, FYI:
> > *1. We should set the JAVA_HOME, because the travis issue see more in
> [2]*
> >
> > if [ "$(uname -m)" == aarch64 ]; then
> > sudo apt-get -y install openjdk-8-jdk && export
> > JAVA_HOME=/usr/lib/jvm/java-8-openjdk-arm64;
> > fi
> >
> > Try to do something as above.
> >
> > *2. travis-script.sh failed due to swapon failed in [4]*
> > After some investigation, I found that the filesystem of aarch64 in
> travis
> > is “shiftfs”, but the x86 is “ext4”.
> > So, I raise a problem in [3].
> >
> > 3. Rocksdb version problem is fixed in [5]
> >
> > 4. testLargeTopologiesOnLargeClustersGras is failed due to timeout error
> > [6]
> >
> > [1] https://issues.apache.org/jira/browse/STORM-3401
> > [2]
> >
> >
> https://travis-ci.community/t/java-home-is-not-exported-for-arm64-architecture/6993/2
> > [3]
> >
> >
> https://travis-ci.community/t/about-the-multi-cpu-architecture-category/5336/21?u=yikun
> > [4]
> >
> >
> https://github.com/apache/storm/blob/master/integration-test/run-it.sh#L40-L42
> > [5] https://issues.apache.org/jira/browse/STORM-3599
> > [6]
> >
> >
> https://github.com/Yikun/storm/blob/master/storm-server/src/test/java/org/apache/storm/scheduler/resource/TestResourceAwareScheduler.java#L1358
> >
> > Regards,
> > Yikun
> > ----------------------------------------
> > Jiang Yikun（Kero）
> > Mail: yikunkero@gmail.com
> >
>
> --
>
>
>
>
> *Disclaimer* -The information transmitted is intended solely for the
> individual
> or entity to which it is addressed and may contain confidential
> and/or
> privileged material. Any review, re-transmission, dissemination or
> other use of
> or taking action in reliance upon this information by persons
> or entities other
> than the intended recipient is prohibited. If you have
> received this email in
> error please contact the sender and delete the
> material from any computer. In
> such instances you are further prohibited
> from reproducing, disclosing,
> distributing or taking any action in reliance
> on it.As a recipient of this email,
> you are responsible for screening its
> contents and the contents of any
> attachments for the presence of viruses.
> No liability is accepted for any
> damages caused by any virus transmitted by
> this email.
>
Re: Adding arm64 support to travis-ci in storm
Posted by Rahul Aggarwal <ra...@puresoftware.com>.
Hi there,
Thank you so much for this detailed information. I am glad to have a quick
response from you.
I got busy with my routine office stuff, so could not focus much on this
package this whole week.
Actually I am more focused towards adding arm64 support to travis-ci only,
because of my professional commitments.
I have tried something like this :
1. As pip and pip3 were causing issues, hence I changed the python version
from 2.7 to 3. Now mock is getting installed successfully. Alternatively,
we can also install mock using pip2 on python 2.7 .
2. Also, JAVA_HOME was getting set with jre, not jdk. So, I manually
installed openjdk-11-jdk and then set JAVA_HOME according to it. That issue
has been cleared now.
3. After that, I am getting issues related to swapon file permissions, and
an unexpected interrupt in 3 out of 6 arm64 tests on travis-ci. I am
linking my logs below. Plz have a look at those test cases with jdk 11 on
arm64.
TRAVIS-CI logs : https://travis-ci.com/github/rahulgit-ps/storm
I will keep sharing my findings. Hoping the same :)
Thanks and Regards
Rahul Aggarwal
On Tue, Jun 23, 2020 at 7:05 AM Yikun Jiang <yi...@gmail.com> wrote:
> Hi,
>
> We have some invistatgation on ARM CI support of STORM.
> https://github.com/Yikun/storm/pull/2/files
>
> Some introduction of our invesitagation, FYI:
> *1. We should set the JAVA_HOME, because the travis issue see more in [2]*
>
> if [ "$(uname -m)" == aarch64 ]; then
> sudo apt-get -y install openjdk-8-jdk && export
> JAVA_HOME=/usr/lib/jvm/java-8-openjdk-arm64;
> fi
>
> Try to do something as above.
>
> *2. travis-script.sh failed due to swapon failed in [4]*
> After some investigation, I found that the filesystem of aarch64 in travis
> is “shiftfs”, but the x86 is “ext4”.
> So, I raise a problem in [3].
>
> 3. Rocksdb version problem is fixed in [5]
>
> 4. testLargeTopologiesOnLargeClustersGras is failed due to timeout error
> [6]
>
> [1] https://issues.apache.org/jira/browse/STORM-3401
> [2]
>
> https://travis-ci.community/t/java-home-is-not-exported-for-arm64-architecture/6993/2
> [3]
>
> https://travis-ci.community/t/about-the-multi-cpu-architecture-category/5336/21?u=yikun
> [4]
>
> https://github.com/apache/storm/blob/master/integration-test/run-it.sh#L40-L42
> [5] https://issues.apache.org/jira/browse/STORM-3599
> [6]
>
> https://github.com/Yikun/storm/blob/master/storm-server/src/test/java/org/apache/storm/scheduler/resource/TestResourceAwareScheduler.java#L1358
>
> Regards,
> Yikun
> ----------------------------------------
> Jiang Yikun（Kero）
> Mail: yikunkero@gmail.com
>
--
*Disclaimer* -The information transmitted is intended solely for the
individual
or entity to which it is addressed and may contain confidential
and/or
privileged material. Any review, re-transmission, dissemination or
other use of
or taking action in reliance upon this information by persons
or entities other
than the intended recipient is prohibited. If you have
received this email in
error please contact the sender and delete the
material from any computer. In
such instances you are further prohibited
from reproducing, disclosing,
distributing or taking any action in reliance
on it.As a recipient of this email,
you are responsible for screening its
contents and the contents of any
attachments for the presence of viruses.
No liability is accepted for any
damages caused by any virus transmitted by
this email.
Re: Adding arm64 support to travis-ci in storm
Posted by Yikun Jiang <yi...@gmail.com>.
Hi,
We have some invistatgation on ARM CI support of STORM.
https://github.com/Yikun/storm/pull/2/files
Some introduction of our invesitagation, FYI:
*1. We should set the JAVA_HOME, because the travis issue see more in [2]*
if [ "$(uname -m)" == aarch64 ]; then
sudo apt-get -y install openjdk-8-jdk && export
JAVA_HOME=/usr/lib/jvm/java-8-openjdk-arm64;
fi
Try to do something as above.
*2. travis-script.sh failed due to swapon failed in [4]*
After some investigation, I found that the filesystem of aarch64 in travis
is “shiftfs”, but the x86 is “ext4”.
So, I raise a problem in [3].
3. Rocksdb version problem is fixed in [5]
4. testLargeTopologiesOnLargeClustersGras is failed due to timeout error [6]
[1] https://issues.apache.org/jira/browse/STORM-3401
[2]
https://travis-ci.community/t/java-home-is-not-exported-for-arm64-architecture/6993/2
[3]
https://travis-ci.community/t/about-the-multi-cpu-architecture-category/5336/21?u=yikun
[4]
https://github.com/apache/storm/blob/master/integration-test/run-it.sh#L40-L42
[5] https://issues.apache.org/jira/browse/STORM-3599
[6]
https://github.com/Yikun/storm/blob/master/storm-server/src/test/java/org/apache/storm/scheduler/resource/TestResourceAwareScheduler.java#L1358
Regards,
Yikun
----------------------------------------
Jiang Yikun（Kero）
Mail: yikunkero@gmail.com
Re: Adding arm64 support to travis-ci in storm
Posted by Govind Menon <go...@gmail.com>.
From the logs it looks like the JAVA_HOME isn't set. And when I looked at
the job config it seems you've commented out that section.
Could you also file a JIRA? - the CLI should fail more gracefully when the
JAVA_HOME isn't set.
On Mon, 22 Jun 2020 at 13:36, Rahul Aggarwal <
rahul.aggarwal@puresoftware.com> wrote:
> Hi sir,
>
> Thank you for the suggestion. Yes, pip2 install mock has solved the mock
> import problem.
> But I am again coming up with some test errors. I am linking my travis-ci
> job logs below.
> https://travis-ci.com/github/rahulgit-ps/storm/jobs/352331819
> I am guessing that the code is making some assumptions that are not working
> for arm64 travis job.
> Can you please review the travis logs?
>
> Regards
> Rahul
>
>
>
>
> On Mon, Jun 22, 2020 at 8:29 PM Govind Menon <go...@gmail.com>
> wrote:
>
> > What is the Python version which is showing the error? It's possible your
> > default Python is also 3 - try pip2 install mock as well.
> >
> > On Mon, 22 Jun 2020 at 04:26, Rahul Aggarwal <
> > rahul.aggarwal@puresoftware.com> wrote:
> >
> > > Hi Team,
> > >
> > > I am adding arm64 support to storm's travis-ci with jdk 8 and 11.
> > > During build, I am getting an error related to mock as below :
> > >
> > > ImportError: No module named mock
> > >
> > > I have manually installed mock via pip install mock and pip3 install
> > mock.
> > > But the error is still the same.
> > > Can you please suggest a resolution.
> > >
> > > Thanks & Regards
> > > Rahul Aggarwal
> > >
> > > --
> > >
> > >
> > >
> > >
> > > *Disclaimer* -The information transmitted is intended solely for the
> > > individual
> > > or entity to which it is addressed and may contain confidential
> > > and/or
> > > privileged material. Any review, re-transmission, dissemination or
> > > other use of
> > > or taking action in reliance upon this information by persons
> > > or entities other
> > > than the intended recipient is prohibited. If you have
> > > received this email in
> > > error please contact the sender and delete the
> > > material from any computer. In
> > > such instances you are further prohibited
> > > from reproducing, disclosing,
> > > distributing or taking any action in reliance
> > > on it.As a recipient of this email,
> > > you are responsible for screening its
> > > contents and the contents of any
> > > attachments for the presence of viruses.
> > > No liability is accepted for any
> > > damages caused by any virus transmitted by
> > > this email.
> > >
> >
>
> --
>
>
>
>
> *Disclaimer* -The information transmitted is intended solely for the
> individual
> or entity to which it is addressed and may contain confidential
> and/or
> privileged material. Any review, re-transmission, dissemination or
> other use of
> or taking action in reliance upon this information by persons
> or entities other
> than the intended recipient is prohibited. If you have
> received this email in
> error please contact the sender and delete the
> material from any computer. In
> such instances you are further prohibited
> from reproducing, disclosing,
> distributing or taking any action in reliance
> on it.As a recipient of this email,
> you are responsible for screening its
> contents and the contents of any
> attachments for the presence of viruses.
> No liability is accepted for any
> damages caused by any virus transmitted by
> this email.
>
Re: Adding arm64 support to travis-ci in storm
Posted by Rahul Aggarwal <ra...@puresoftware.com>.
Hi sir,
Thank you for the suggestion. Yes, pip2 install mock has solved the mock
import problem.
But I am again coming up with some test errors. I am linking my travis-ci
job logs below.
https://travis-ci.com/github/rahulgit-ps/storm/jobs/352331819
I am guessing that the code is making some assumptions that are not working
for arm64 travis job.
Can you please review the travis logs?
Regards
Rahul
On Mon, Jun 22, 2020 at 8:29 PM Govind Menon <go...@gmail.com>
wrote:
> What is the Python version which is showing the error? It's possible your
> default Python is also 3 - try pip2 install mock as well.
>
> On Mon, 22 Jun 2020 at 04:26, Rahul Aggarwal <
> rahul.aggarwal@puresoftware.com> wrote:
>
> > Hi Team,
> >
> > I am adding arm64 support to storm's travis-ci with jdk 8 and 11.
> > During build, I am getting an error related to mock as below :
> >
> > ImportError: No module named mock
> >
> > I have manually installed mock via pip install mock and pip3 install
> mock.
> > But the error is still the same.
> > Can you please suggest a resolution.
> >
> > Thanks & Regards
> > Rahul Aggarwal
> >
> > --
> >
> >
> >
> >
> > *Disclaimer* -The information transmitted is intended solely for the
> > individual
> > or entity to which it is addressed and may contain confidential
> > and/or
> > privileged material. Any review, re-transmission, dissemination or
> > other use of
> > or taking action in reliance upon this information by persons
> > or entities other
> > than the intended recipient is prohibited. If you have
> > received this email in
> > error please contact the sender and delete the
> > material from any computer. In
> > such instances you are further prohibited
> > from reproducing, disclosing,
> > distributing or taking any action in reliance
> > on it.As a recipient of this email,
> > you are responsible for screening its
> > contents and the contents of any
> > attachments for the presence of viruses.
> > No liability is accepted for any
> > damages caused by any virus transmitted by
> > this email.
> >
>
--
*Disclaimer* -The information transmitted is intended solely for the
individual
or entity to which it is addressed and may contain confidential
and/or
privileged material. Any review, re-transmission, dissemination or
other use of
or taking action in reliance upon this information by persons
or entities other
than the intended recipient is prohibited. If you have
received this email in
error please contact the sender and delete the
material from any computer. In
such instances you are further prohibited
from reproducing, disclosing,
distributing or taking any action in reliance
on it.As a recipient of this email,
you are responsible for screening its
contents and the contents of any
attachments for the presence of viruses.
No liability is accepted for any
damages caused by any virus transmitted by
this email.
Re: Adding arm64 support to travis-ci in storm
Posted by Govind Menon <go...@gmail.com>.
What is the Python version which is showing the error? It's possible your
default Python is also 3 - try pip2 install mock as well.
On Mon, 22 Jun 2020 at 04:26, Rahul Aggarwal <
rahul.aggarwal@puresoftware.com> wrote:
> Hi Team,
>
> I am adding arm64 support to storm's travis-ci with jdk 8 and 11.
> During build, I am getting an error related to mock as below :
>
> ImportError: No module named mock
>
> I have manually installed mock via pip install mock and pip3 install mock.
> But the error is still the same.
> Can you please suggest a resolution.
>
> Thanks & Regards
> Rahul Aggarwal
>
> --
>
>
>
>
> *Disclaimer* -The information transmitted is intended solely for the
> individual
> or entity to which it is addressed and may contain confidential
> and/or
> privileged material. Any review, re-transmission, dissemination or
> other use of
> or taking action in reliance upon this information by persons
> or entities other
> than the intended recipient is prohibited. If you have
> received this email in
> error please contact the sender and delete the
> material from any computer. In
> such instances you are further prohibited
> from reproducing, disclosing,
> distributing or taking any action in reliance
> on it.As a recipient of this email,
> you are responsible for screening its
> contents and the contents of any
> attachments for the presence of viruses.
> No liability is accepted for any
> damages caused by any virus transmitted by
> this email.
>