You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@aurora.apache.org by Yiannis Gkoufas <jo...@gmail.com> on 2016/07/14 08:44:27 UTC
Build thermos executor on ARM
Hi guys,
I have been trying to get aurora working on ARM processors.
I had no problem with the scheduler, but I didn't manage to compile the
thermos executor.
When I give:
./pants binary src/main/python/apache/aurora/executor:thermos_executor
I get back:
FAILURE: Update --binaries-path-by-id to find binaries for Linux armv7l
3.10.40
Let me know if you have any hints on how to workaround this.
Thanks!
Re: Build thermos executor on ARM
Posted by Yiannis Gkoufas <jo...@gmail.com>.
Hi guys,
thanks so much for taking the time to assist me during this effort.
Will try to outline the overall steps needed to successfully build aurora
components and deploy the hello_world of docker.
1. Made the configurations needed as outlined in the
*provision-dev-cluster.sh*
- From the *prepare_sources *I just picked up the configuration needed
for the mesos-slaves and add them as flags on my start-mesos-slave.sh
script, because I have built it already from source ( I couldn't use the
mesosphere ppa with the ARM processor)
- In *prepare_extras *I just changed it to point to my directory
- Same for *install_cluster_config*
- Kept *install_ssh_config *as it is
- Same for *start_services*
- Disabled *enable_gradle_daemon configure_netrc sudoless_docker_setup *as
I don't think I need them
2. Changed the configurations for the upstart jobs:
- In *aurora-scheduler.conf *I changed the references to my directory
and IP/Hostname values. Moreover I had to edit as follows:
env JAVA_OPTS='-Djava.library.path=/usr/lib:/usr/local/lib
Because my libmesos.so was created there
- In *aurora-thermos-observer.conf *I had to just change the directory
3. I had to copy all the *.egg files in the third_party directory as
described by Florian and Stephan
4. Modified *aurorabuild.sh *all the pointers to my directory. Moreover I
had to edit as follows:
export LD_LIBRARY_PATH=/usr/lib/jvm/java-7-openjdk-armhf/jre/lib/arm/server
5. Followed the instructions by John to fix the issue with pants:
- ./build-support/thrift/thriftw 0.9.1 --version
- modified pants.ini to add
[binaries]
baseurls: +['http://localhost:9999']
path_by_id: {('linux', 'armv7l'): ['local']}
- mkdir -p build-support/thrift/serve/bin/thrift/local/0.9.1
- ln -s $PWD/build-support/thrift/thrift-0.9.1/compiler/cpp/thrift
build-support/thrift/serve/bin/thrift/local/0.9.1/thrift
- cd build-support/thrift/serve && (nohup python2 -m SimpleHTTPServer
9999 >&/tmp/thrift.serve &) && cd -
6. Executed ./*aurorabuild.sh all* to build the components
7. Created my docker container since the example doesn't work out of the
box as the instructions found here
https://issues.apache.org/jira/browse/AURORA-1487
- cd examples/jobs/docker
- Create Dockerfile as:
FROM armv7/armhf-ubuntu:14.04.3
RUN apt-get update && apt-get -y install \
libapr1-dev \
libcurl4-nss-dev \
libsvn-dev \
python-all-dev
- docker build -t aurora/docker:2.7 .
-
- Edit *hello_docker.aurora *as follows:
with image = 'aurora/docker:2.7'
- aurora job create devcluster/docker-test/devel/hello_docker
./hello_docker.aurora
I have one question regarding the architecture. If I need to add another
node as a "worker" (to contain observer+executor) what are the components I
need to build in that case?
It's just the build_executor and build_observer functions need to be called
from the build script?
Thanks a lot,
Yiannis
On 19 July 2016 at 09:39, Erb, Stephan <St...@blue-yonder.com> wrote:
> Minor addition to the explanation of Florian: the folder in question has
> to be called `third_party’ (mind the underscore).
>
>
>
> *From: *Florian Pfeiffer <fp...@x8s.de>
> *Reply-To: *"user@aurora.apache.org" <us...@aurora.apache.org>
> *Date: *Tuesday 19 July 2016 at 10:19
>
> *To: *"user@aurora.apache.org" <us...@aurora.apache.org>
> *Subject: *Re: Build thermos executor on ARM
>
>
>
> if this hasn't changed somewhen during the last versions, you need to
> create a directory "thirdparty" (written out.. don't mix it up with the
> already existing "3rdparty" directory) and copy the egg there.
>
>
>
> 2016-07-18 18:09 GMT+02:00 Yiannis Gkoufas <jo...@gmail.com>:
>
> great, so if I understood correctly, it just needs the *.egg files right?
>
> since I have built it myself, what is the directory in which I would have
> to copy them?
>
>
>
> On 18 July 2016 at 16:52, Erb, Stephan <St...@blue-yonder.com>
> wrote:
>
> The Thermos executor is bundling the Mesos driver necessary for
> communication with to the agent. To make this work, you have to build
> Python egg files containing the Mesos driver. This is the script we
> normally use to achieve this:
> https://github.com/apache/aurora/blob/master/build-support/python/make-mesos-native-egg
>
>
>
>
>
> *From: *Yiannis Gkoufas <jo...@gmail.com>
> *Reply-To: *"user@aurora.apache.org" <us...@aurora.apache.org>
> *Date: *Monday 18 July 2016 at 17:35
>
>
> *To: *"user@aurora.apache.org" <us...@aurora.apache.org>
> *Subject: *Re: Build thermos executor on ARM
>
>
>
> So the error I am getting now is:
>
>
>
> 15:33:32 00:25 [chroot]INFO] Attempting to fetch thrift binary from:
> https://dl.bintray.com/pantsbuild/bin/build-support/bin/thrift/local/0.9.1/thrift
> ...
>
> INFO] Attempting to fetch thrift binary from:
> http://localhost:9999/bin/thrift/local/0.9.1/thrift ...
>
> 100% .................................................. 20270 KB 0.285s
>
> INFO] Fetched thrift binary from:
> http://localhost:9999/bin/thrift/local/0.9.1/thrift .
>
>
>
> Waiting for background workers to finish.
>
> 15:33:39 00:32 [complete]
>
> FAILURE
>
> Exception caught: (<class 'pex.resolver.Unsatisfiable'>)
>
>
>
> Exception message: Could not satisfy all requirements for
> mesos.native==0.28.2:
>
> mesos.native==0.28.2
>
>
>
> Which is kind of strange, because I have actually already installed mesos
> 0.28.2
>
>
>
> On 18 July 2016 at 16:16, John Sirois <jo...@gmail.com> wrote:
>
> On Jul 18, 2016 9:04 AM, "Yiannis Gkoufas" <jo...@gmail.com> wrote:
> >
> > Hi John,
> >
> > just to confirm: this is how this part of pants.ini should look like:
> >
> > backend_packages: [
> > [binaries]
> > baseurls: +['http://localhost:9999']
> > path_by_id: {('linux', 'armv7l'): ['local']}
> > ]
> >
> > right?
>
> Not quite, kill the surrounding backend_packages, so:
>
> [binaries]
> baseurls: +['http://localhost:9999']
> path_by_id: {('linux', 'armv7l'): ['local']}
>
> >
>
>
> > On 17 July 2016 at 11:58, Yiannis Gkoufas <jo...@gmail.com> wrote:
> >>
> >> Wow, thanks a lot John!
> >> Will give it a shot and get back to you!
> >>
> >> On 15 July 2016 at 18:05, John Sirois <jo...@gmail.com> wrote:
> >>>
> >>> Here's a workaround for you to try. If this works, I can modify the
> Aurora pants build without modifying pants itself to do all this
> automatically.
> >>>
> >>> The basic idea is that pants probes a list of URLs for pre-built
> binaries. You can add a local HTTP server to the list and point that
> server at the thrift binary Aurora builds for gradle via make.
> >>> Below is how you do this:
> >>>
> >>> 0.) Ensure the Aurora gradle thrift is built.
> >>> $ ./build-support/thrift/thriftw 0.9.1 --version
> >>> Thrift version 0.9.1
> >>>
> >>> 1.) Just for reference, the pants options in-play here
> >>> $ ./pants options | grep binaries
> >>> binaries.baseurls = ['
> https://dl.bintray.com/pantsbuild/bin/build-support'] (from HARDCODED)
> >>> binaries.fetch_timeout_secs = 30 (from HARDCODED)
> >>> binaries.path_by_id = {} (from NONE)
> >>> $ ./pants options | grep binary
> >>> thrift-binary.supportdir = bin/thrift (from HARDCODED)
> >>> thrift-binary.version = 0.9.1 (from CONFIG in pants.ini)
> >>>
> >>> 2.) Grab machine name pants will try to find your thrift binary under
> (here is an example from my machine with result 'x86_64' - yours will be
> different and arm-ey):
> >>> $ python2 -c 'import os; sysname, _, _, _, machine = os.uname(); print
> sysname.lower(), machine'
> >>> linux x86_64
> >>>
> >>> 3.) Edit pants.ini to tell pants where to look for binaries for your
> machine:
> >>> $ git diff -U1
> >>> diff --git a/pants.ini b/pants.ini
> >>> index de0ab30..e2cd94e 100644
> >>> --- a/pants.ini
> >>> +++ b/pants.ini
> >>> @@ -32,2 +32,6 @@ backend_packages: [
> >>>
> >>> +[binaries]
> >>> +baseurls: +['http://localhost:9999']
> >>> +path_by_id: {('linux', 'x86_64'): ['local']}
> >>> +
> >>> [thrift-binary]
> >>>
> >>> 4.) Setup the URL you told pants about above:
> >>> $ mkdir -p build-support/thrift/serve/bin/thrift/local/0.9.1
> >>> $ ln -s $PWD/build-support/thrift/thrift-0.9.1/compiler/cpp/thrift
> build-support/thrift/serve/bin/thrift/local/0.9.1/thrift
> >>> $ cd build-support/thrift/serve && (nohup python2 -m SimpleHTTPServer
> 9999 >&/tmp/thrift.serve &) && cd -
> >>>
> >>> 5.) Run pants and confirm download of a thrift binary for your machine:
> >>> $ ./pants binary src/main/python/::
> >>>
> >>> 10:58:44 00:00 [main]
> >>> (To run a reporting server: ./pants server)
> >>> 10:58:44 00:00 [setup]
> >>> 10:58:44 00:00 [parse]
> >>> Executing tasks in goals: binary
> >>> 10:58:44 00:00 [binary]
> >>> 10:58:44 00:00 [binary-prep-command]
> >>> 10:58:44 00:00 [python-binary-create]
> >>> 10:58:44 00:00 [cache]
> >>> No cached artifacts for 8 targets.
> >>> Invalidated 8 targets.
> >>> 10:58:44 00:00 [chroot]INFO] Attempting to fetch thrift binary
> from: http://localhost:9999/bin/thrift/local/0.9.1/thrift ...
> >>> 100% .................................................. 31661 KB 0.364s
> >>> INFO] Fetched thrift binary from:
> http://localhost:9999/bin/thrift/local/0.9.1/thrift .
> >>> ...
> >>>
> >>> Let me know what you get!
> >>>
> >>> On Thu, Jul 14, 2016 at 6:24 AM, Yiannis Gkoufas <jo...@gmail.com>
> wrote:
> >>>>
> >>>> Cool, can you imagine a way to work around this?
> >>>> I have to customize pants itself right?
> >>>>
> >>>> On 14 July 2016 at 11:54, Erb, Stephan <St...@blue-yonder.com>
> wrote:
> >>>>>
> >>>>> Yeah, that sounds like it could be the issue here.
> >>>>>
> >>>>>
> >>>>>
> >>>>> From: Yiannis Gkoufas <jo...@gmail.com>
> >>>>> Reply-To: "user@aurora.apache.org" <us...@aurora.apache.org>
> >>>>> Date: Thursday 14 July 2016 at 12:28
> >>>>> To: "user@aurora.apache.org" <us...@aurora.apache.org>
> >>>>> Subject: Re: Build thermos executor on ARM
> >>>>>
> >>>>>
> >>>>>>
> >>>>>> Hi there,
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> sure:
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> 10:26:45 00:00 [binary]
> >>>>>>
> >>>>>> 10:26:45 00:00 [binary-jvm-prep-command]
> >>>>>>
> >>>>>> 10:26:46 00:01 [jvm_prep_command]
> >>>>>>
> >>>>>> 10:26:46 00:01 [binary-prep-command]
> >>>>>>
> >>>>>> 10:26:46 00:01 [python-binary-create]
> >>>>>>
> >>>>>> Detected interpreter /usr/bin/python2.7:
> CPython-2.7.6
> >>>>>>
> >>>>>> 10:26:46 00:01 [chroot]
> >>>>>>
> >>>>>> Building chroot for
> [PythonBinary(BuildFileAddress(BuildFile(src/main/python/apache/aurora/executor/BUILD,
> FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)),
> thermos_executor))]:
> >>>>>>
> >>>>>> Dumping library:
> PythonLibrary(BuildFileAddress(BuildFile(src/main/python/apache/aurora/executor/BUILD,
> FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)), _executor))
> >>>>>>
> >>>>>> Dumping library:
> PythonLibrary(BuildFileAddress(BuildFile(api/src/main/thrift/org/apache/aurora/gen/BUILD,
> FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)), gen))
> >>>>>>
> >>>>>> Dumping library:
> PythonLibrary(BuildFileAddress(BuildFile(src/main/python/apache/aurora/common/BUILD,
> FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)), common))
> >>>>>>
> >>>>>> Dumping library:
> PythonLibrary(BuildFileAddress(BuildFile(src/main/python/apache/aurora/config/BUILD,
> FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)), config))
> >>>>>>
> >>>>>> Dumping library:
> PythonLibrary(BuildFileAddress(BuildFile(src/main/python/apache/thermos/config/BUILD,
> FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)), config))
> >>>>>>
> >>>>>> Dumping library:
> PythonLibrary(BuildFileAddress(BuildFile(src/main/python/apache/thermos/common/BUILD,
> FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)), common))
> >>>>>>
> >>>>>> Dumping library:
> PythonLibrary(BuildFileAddress(BuildFile(src/main/python/apache/thermos/core/BUILD,
> FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)), core))
> >>>>>>
> >>>>>> Dumping library:
> PythonLibrary(BuildFileAddress(BuildFile(src/main/python/apache/thermos/monitoring/BUILD,
> FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)), monitoring))
> >>>>>>
> >>>>>> Dumping library:
> PythonBinary(BuildFileAddress(BuildFile(src/main/python/apache/aurora/executor/BUILD,
> FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)),
> thermos_executor))
> >>>>>>
> >>>>>> FAILURE: Update --binaries-path-by-id to find binaries for Linux
> armv7l 3.10.40
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> If I am not completely wrong, I think pants is looking one of the
> architectures outlined here:
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> https://github.com/pantsbuild/pants/blob/master/src/python/pants/binaries/binary_util.py#L24
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> and probably that's why it fails
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> On 14 July 2016 at 11:17, Erb, Stephan <St...@blue-yonder.com>
> wrote:
> >>>>>>>
> >>>>>>> Cool, ARM J. Please try to run it like this, so that we get a
> little bit more output:
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> PEX_VERBOSE=1 ./pants -ldebug binary
> src/main/python/apache/aurora/executor:thermos_executor
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> From: Yiannis Gkoufas <jo...@gmail.com>
> >>>>>>> Reply-To: "user@aurora.apache.org" <us...@aurora.apache.org>
> >>>>>>> Date: Thursday 14 July 2016 at 10:44
> >>>>>>> To: "user@aurora.apache.org" <us...@aurora.apache.org>
> >>>>>>> Subject: Build thermos executor on ARM
> >>>>>>>
> >>>>>>>
> >>>>>>>>
> >>>>>>>> Hi guys,
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> I have been trying to get aurora working on ARM processors.
> >>>>>>>>
> >>>>>>>> I had no problem with the scheduler, but I didn't manage to
> compile the thermos executor.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> When I give:
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> ./pants binary
> src/main/python/apache/aurora/executor:thermos_executor
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> I get back:
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> FAILURE: Update --binaries-path-by-id to find binaries for Linux
> armv7l 3.10.40
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> Let me know if you have any hints on how to workaround this.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> Thanks!
> >>>>>>
> >>>>>>
> >>>>
> >>>>
> >>>
> >>
> >
>
>
>
>
>
>
>
>
Re: Build thermos executor on ARM
Posted by "Erb, Stephan" <St...@blue-yonder.com>.
Minor addition to the explanation of Florian: the folder in question has to be called `third_party’ (mind the underscore).
From: Florian Pfeiffer <fp...@x8s.de>
Reply-To: "user@aurora.apache.org" <us...@aurora.apache.org>
Date: Tuesday 19 July 2016 at 10:19
To: "user@aurora.apache.org" <us...@aurora.apache.org>
Subject: Re: Build thermos executor on ARM
if this hasn't changed somewhen during the last versions, you need to create a directory "thirdparty" (written out.. don't mix it up with the already existing "3rdparty" directory) and copy the egg there.
2016-07-18 18:09 GMT+02:00 Yiannis Gkoufas <jo...@gmail.com>>:
great, so if I understood correctly, it just needs the *.egg files right?
since I have built it myself, what is the directory in which I would have to copy them?
On 18 July 2016 at 16:52, Erb, Stephan <St...@blue-yonder.com>> wrote:
The Thermos executor is bundling the Mesos driver necessary for communication with to the agent. To make this work, you have to build Python egg files containing the Mesos driver. This is the script we normally use to achieve this: https://github.com/apache/aurora/blob/master/build-support/python/make-mesos-native-egg
From: Yiannis Gkoufas <jo...@gmail.com>>
Reply-To: "user@aurora.apache.org<ma...@aurora.apache.org>" <us...@aurora.apache.org>>
Date: Monday 18 July 2016 at 17:35
To: "user@aurora.apache.org<ma...@aurora.apache.org>" <us...@aurora.apache.org>>
Subject: Re: Build thermos executor on ARM
So the error I am getting now is:
15:33:32 00:25 [chroot]INFO] Attempting to fetch thrift binary from: https://dl.bintray.com/pantsbuild/bin/build-support/bin/thrift/local/0.9.1/thrift ...
INFO] Attempting to fetch thrift binary from: http://localhost:9999/bin/thrift/local/0.9.1/thrift ...
100% .................................................. 20270 KB 0.285s
INFO] Fetched thrift binary from: http://localhost:9999/bin/thrift/local/0.9.1/thrift .
Waiting for background workers to finish.
15:33:39 00:32 [complete]
FAILURE
Exception caught: (<class 'pex.resolver.Unsatisfiable'>)
Exception message: Could not satisfy all requirements for mesos.native==0.28.2:
mesos.native==0.28.2
Which is kind of strange, because I have actually already installed mesos 0.28.2
On 18 July 2016 at 16:16, John Sirois <jo...@gmail.com>> wrote:
On Jul 18, 2016 9:04 AM, "Yiannis Gkoufas" <jo...@gmail.com>> wrote:
>
> Hi John,
>
> just to confirm: this is how this part of pants.ini should look like:
>
> backend_packages: [
> [binaries]
> baseurls: +['http://localhost:9999']
> path_by_id: {('linux', 'armv7l'): ['local']}
> ]
>
> right?
Not quite, kill the surrounding backend_packages, so:
[binaries]
baseurls: +['http://localhost:9999']
path_by_id: {('linux', 'armv7l'): ['local']}
>
> On 17 July 2016 at 11:58, Yiannis Gkoufas <jo...@gmail.com>> wrote:
>>
>> Wow, thanks a lot John!
>> Will give it a shot and get back to you!
>>
>> On 15 July 2016 at 18:05, John Sirois <jo...@gmail.com>> wrote:
>>>
>>> Here's a workaround for you to try. If this works, I can modify the Aurora pants build without modifying pants itself to do all this automatically.
>>>
>>> The basic idea is that pants probes a list of URLs for pre-built binaries. You can add a local HTTP server to the list and point that server at the thrift binary Aurora builds for gradle via make.
>>> Below is how you do this:
>>>
>>> 0.) Ensure the Aurora gradle thrift is built.
>>> $ ./build-support/thrift/thriftw 0.9.1 --version
>>> Thrift version 0.9.1
>>>
>>> 1.) Just for reference, the pants options in-play here
>>> $ ./pants options | grep binaries
>>> binaries.baseurls = ['https://dl.bintray.com/pantsbuild/bin/build-support'] (from HARDCODED)
>>> binaries.fetch_timeout_secs = 30 (from HARDCODED)
>>> binaries.path_by_id = {} (from NONE)
>>> $ ./pants options | grep binary
>>> thrift-binary.supportdir = bin/thrift (from HARDCODED)
>>> thrift-binary.version = 0.9.1 (from CONFIG in pants.ini)
>>>
>>> 2.) Grab machine name pants will try to find your thrift binary under (here is an example from my machine with result 'x86_64' - yours will be different and arm-ey):
>>> $ python2 -c 'import os; sysname, _, _, _, machine = os.uname(); print sysname.lower(), machine'
>>> linux x86_64
>>>
>>> 3.) Edit pants.ini to tell pants where to look for binaries for your machine:
>>> $ git diff -U1
>>> diff --git a/pants.ini b/pants.ini
>>> index de0ab30..e2cd94e 100644
>>> --- a/pants.ini
>>> +++ b/pants.ini
>>> @@ -32,2 +32,6 @@ backend_packages: [
>>>
>>> +[binaries]
>>> +baseurls: +['http://localhost:9999']
>>> +path_by_id: {('linux', 'x86_64'): ['local']}
>>> +
>>> [thrift-binary]
>>>
>>> 4.) Setup the URL you told pants about above:
>>> $ mkdir -p build-support/thrift/serve/bin/thrift/local/0.9.1
>>> $ ln -s $PWD/build-support/thrift/thrift-0.9.1/compiler/cpp/thrift build-support/thrift/serve/bin/thrift/local/0.9.1/thrift
>>> $ cd build-support/thrift/serve && (nohup python2 -m SimpleHTTPServer 9999 >&/tmp/thrift.serve &) && cd -
>>>
>>> 5.) Run pants and confirm download of a thrift binary for your machine:
>>> $ ./pants binary src/main/python/::
>>>
>>> 10:58:44 00:00 [main]
>>> (To run a reporting server: ./pants server)
>>> 10:58:44 00:00 [setup]
>>> 10:58:44 00:00 [parse]
>>> Executing tasks in goals: binary
>>> 10:58:44 00:00 [binary]
>>> 10:58:44 00:00 [binary-prep-command]
>>> 10:58:44 00:00 [python-binary-create]
>>> 10:58:44 00:00 [cache]
>>> No cached artifacts for 8 targets.
>>> Invalidated 8 targets.
>>> 10:58:44 00:00 [chroot]INFO] Attempting to fetch thrift binary from: http://localhost:9999/bin/thrift/local/0.9.1/thrift ...
>>> 100% .................................................. 31661 KB 0.364s
>>> INFO] Fetched thrift binary from: http://localhost:9999/bin/thrift/local/0.9.1/thrift .
>>> ...
>>>
>>> Let me know what you get!
>>>
>>> On Thu, Jul 14, 2016 at 6:24 AM, Yiannis Gkoufas <jo...@gmail.com>> wrote:
>>>>
>>>> Cool, can you imagine a way to work around this?
>>>> I have to customize pants itself right?
>>>>
>>>> On 14 July 2016 at 11:54, Erb, Stephan <St...@blue-yonder.com>> wrote:
>>>>>
>>>>> Yeah, that sounds like it could be the issue here.
>>>>>
>>>>>
>>>>>
>>>>> From: Yiannis Gkoufas <jo...@gmail.com>>
>>>>> Reply-To: "user@aurora.apache.org<ma...@aurora.apache.org>" <us...@aurora.apache.org>>
>>>>> Date: Thursday 14 July 2016 at 12:28
>>>>> To: "user@aurora.apache.org<ma...@aurora.apache.org>" <us...@aurora.apache.org>>
>>>>> Subject: Re: Build thermos executor on ARM
>>>>>
>>>>>
>>>>>>
>>>>>> Hi there,
>>>>>>
>>>>>>
>>>>>>
>>>>>> sure:
>>>>>>
>>>>>>
>>>>>>
>>>>>> 10:26:45 00:00 [binary]
>>>>>>
>>>>>> 10:26:45 00:00 [binary-jvm-prep-command]
>>>>>>
>>>>>> 10:26:46 00:01 [jvm_prep_command]
>>>>>>
>>>>>> 10:26:46 00:01 [binary-prep-command]
>>>>>>
>>>>>> 10:26:46 00:01 [python-binary-create]
>>>>>>
>>>>>> Detected interpreter /usr/bin/python2.7: CPython-2.7.6
>>>>>>
>>>>>> 10:26:46 00:01 [chroot]
>>>>>>
>>>>>> Building chroot for [PythonBinary(BuildFileAddress(BuildFile(src/main/python/apache/aurora/executor/BUILD, FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)), thermos_executor))]:
>>>>>>
>>>>>> Dumping library: PythonLibrary(BuildFileAddress(BuildFile(src/main/python/apache/aurora/executor/BUILD, FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)), _executor))
>>>>>>
>>>>>> Dumping library: PythonLibrary(BuildFileAddress(BuildFile(api/src/main/thrift/org/apache/aurora/gen/BUILD, FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)), gen))
>>>>>>
>>>>>> Dumping library: PythonLibrary(BuildFileAddress(BuildFile(src/main/python/apache/aurora/common/BUILD, FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)), common))
>>>>>>
>>>>>> Dumping library: PythonLibrary(BuildFileAddress(BuildFile(src/main/python/apache/aurora/config/BUILD, FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)), config))
>>>>>>
>>>>>> Dumping library: PythonLibrary(BuildFileAddress(BuildFile(src/main/python/apache/thermos/config/BUILD, FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)), config))
>>>>>>
>>>>>> Dumping library: PythonLibrary(BuildFileAddress(BuildFile(src/main/python/apache/thermos/common/BUILD, FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)), common))
>>>>>>
>>>>>> Dumping library: PythonLibrary(BuildFileAddress(BuildFile(src/main/python/apache/thermos/core/BUILD, FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)), core))
>>>>>>
>>>>>> Dumping library: PythonLibrary(BuildFileAddress(BuildFile(src/main/python/apache/thermos/monitoring/BUILD, FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)), monitoring))
>>>>>>
>>>>>> Dumping library: PythonBinary(BuildFileAddress(BuildFile(src/main/python/apache/aurora/executor/BUILD, FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)), thermos_executor))
>>>>>>
>>>>>> FAILURE: Update --binaries-path-by-id to find binaries for Linux armv7l 3.10.40
>>>>>>
>>>>>>
>>>>>>
>>>>>> If I am not completely wrong, I think pants is looking one of the architectures outlined here:
>>>>>>
>>>>>>
>>>>>>
>>>>>> https://github.com/pantsbuild/pants/blob/master/src/python/pants/binaries/binary_util.py#L24
>>>>>>
>>>>>>
>>>>>>
>>>>>> and probably that's why it fails
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 14 July 2016 at 11:17, Erb, Stephan <St...@blue-yonder.com>> wrote:
>>>>>>>
>>>>>>> Cool, ARM J. Please try to run it like this, so that we get a little bit more output:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> PEX_VERBOSE=1 ./pants -ldebug binary src/main/python/apache/aurora/executor:thermos_executor
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> From: Yiannis Gkoufas <jo...@gmail.com>>
>>>>>>> Reply-To: "user@aurora.apache.org<ma...@aurora.apache.org>" <us...@aurora.apache.org>>
>>>>>>> Date: Thursday 14 July 2016 at 10:44
>>>>>>> To: "user@aurora.apache.org<ma...@aurora.apache.org>" <us...@aurora.apache.org>>
>>>>>>> Subject: Build thermos executor on ARM
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>> Hi guys,
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> I have been trying to get aurora working on ARM processors.
>>>>>>>>
>>>>>>>> I had no problem with the scheduler, but I didn't manage to compile the thermos executor.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> When I give:
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> ./pants binary src/main/python/apache/aurora/executor:thermos_executor
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> I get back:
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> FAILURE: Update --binaries-path-by-id to find binaries for Linux armv7l 3.10.40
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Let me know if you have any hints on how to workaround this.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Thanks!
>>>>>>
>>>>>>
>>>>
>>>>
>>>
>>
>
Re: Build thermos executor on ARM
Posted by Florian Pfeiffer <fp...@x8s.de>.
if this hasn't changed somewhen during the last versions, you need to
create a directory "thirdparty" (written out.. don't mix it up with the
already existing "3rdparty" directory) and copy the egg there.
2016-07-18 18:09 GMT+02:00 Yiannis Gkoufas <jo...@gmail.com>:
> great, so if I understood correctly, it just needs the *.egg files right?
> since I have built it myself, what is the directory in which I would have
> to copy them?
>
> On 18 July 2016 at 16:52, Erb, Stephan <St...@blue-yonder.com>
> wrote:
>
>> The Thermos executor is bundling the Mesos driver necessary for
>> communication with to the agent. To make this work, you have to build
>> Python egg files containing the Mesos driver. This is the script we
>> normally use to achieve this:
>> https://github.com/apache/aurora/blob/master/build-support/python/make-mesos-native-egg
>>
>>
>>
>>
>>
>> *From: *Yiannis Gkoufas <jo...@gmail.com>
>> *Reply-To: *"user@aurora.apache.org" <us...@aurora.apache.org>
>> *Date: *Monday 18 July 2016 at 17:35
>>
>> *To: *"user@aurora.apache.org" <us...@aurora.apache.org>
>> *Subject: *Re: Build thermos executor on ARM
>>
>>
>>
>> So the error I am getting now is:
>>
>>
>>
>> 15:33:32 00:25 [chroot]INFO] Attempting to fetch thrift binary
>> from:
>> https://dl.bintray.com/pantsbuild/bin/build-support/bin/thrift/local/0.9.1/thrift
>> ...
>>
>> INFO] Attempting to fetch thrift binary from:
>> http://localhost:9999/bin/thrift/local/0.9.1/thrift ...
>>
>> 100% .................................................. 20270 KB 0.285s
>>
>> INFO] Fetched thrift binary from:
>> http://localhost:9999/bin/thrift/local/0.9.1/thrift .
>>
>>
>>
>> Waiting for background workers to finish.
>>
>> 15:33:39 00:32 [complete]
>>
>> FAILURE
>>
>> Exception caught: (<class 'pex.resolver.Unsatisfiable'>)
>>
>>
>>
>> Exception message: Could not satisfy all requirements for
>> mesos.native==0.28.2:
>>
>> mesos.native==0.28.2
>>
>>
>>
>> Which is kind of strange, because I have actually already installed mesos
>> 0.28.2
>>
>>
>>
>> On 18 July 2016 at 16:16, John Sirois <jo...@gmail.com> wrote:
>>
>> On Jul 18, 2016 9:04 AM, "Yiannis Gkoufas" <jo...@gmail.com> wrote:
>> >
>> > Hi John,
>> >
>> > just to confirm: this is how this part of pants.ini should look like:
>> >
>> > backend_packages: [
>> > [binaries]
>> > baseurls: +['http://localhost:9999']
>> > path_by_id: {('linux', 'armv7l'): ['local']}
>> > ]
>> >
>> > right?
>>
>> Not quite, kill the surrounding backend_packages, so:
>>
>> [binaries]
>> baseurls: +['http://localhost:9999']
>> path_by_id: {('linux', 'armv7l'): ['local']}
>>
>> >
>>
>>
>> > On 17 July 2016 at 11:58, Yiannis Gkoufas <jo...@gmail.com> wrote:
>> >>
>> >> Wow, thanks a lot John!
>> >> Will give it a shot and get back to you!
>> >>
>> >> On 15 July 2016 at 18:05, John Sirois <jo...@gmail.com> wrote:
>> >>>
>> >>> Here's a workaround for you to try. If this works, I can modify the
>> Aurora pants build without modifying pants itself to do all this
>> automatically.
>> >>>
>> >>> The basic idea is that pants probes a list of URLs for pre-built
>> binaries. You can add a local HTTP server to the list and point that
>> server at the thrift binary Aurora builds for gradle via make.
>> >>> Below is how you do this:
>> >>>
>> >>> 0.) Ensure the Aurora gradle thrift is built.
>> >>> $ ./build-support/thrift/thriftw 0.9.1 --version
>> >>> Thrift version 0.9.1
>> >>>
>> >>> 1.) Just for reference, the pants options in-play here
>> >>> $ ./pants options | grep binaries
>> >>> binaries.baseurls = ['
>> https://dl.bintray.com/pantsbuild/bin/build-support'] (from HARDCODED)
>> >>> binaries.fetch_timeout_secs = 30 (from HARDCODED)
>> >>> binaries.path_by_id = {} (from NONE)
>> >>> $ ./pants options | grep binary
>> >>> thrift-binary.supportdir = bin/thrift (from HARDCODED)
>> >>> thrift-binary.version = 0.9.1 (from CONFIG in pants.ini)
>> >>>
>> >>> 2.) Grab machine name pants will try to find your thrift binary under
>> (here is an example from my machine with result 'x86_64' - yours will be
>> different and arm-ey):
>> >>> $ python2 -c 'import os; sysname, _, _, _, machine = os.uname();
>> print sysname.lower(), machine'
>> >>> linux x86_64
>> >>>
>> >>> 3.) Edit pants.ini to tell pants where to look for binaries for your
>> machine:
>> >>> $ git diff -U1
>> >>> diff --git a/pants.ini b/pants.ini
>> >>> index de0ab30..e2cd94e 100644
>> >>> --- a/pants.ini
>> >>> +++ b/pants.ini
>> >>> @@ -32,2 +32,6 @@ backend_packages: [
>> >>>
>> >>> +[binaries]
>> >>> +baseurls: +['http://localhost:9999']
>> >>> +path_by_id: {('linux', 'x86_64'): ['local']}
>> >>> +
>> >>> [thrift-binary]
>> >>>
>> >>> 4.) Setup the URL you told pants about above:
>> >>> $ mkdir -p build-support/thrift/serve/bin/thrift/local/0.9.1
>> >>> $ ln -s $PWD/build-support/thrift/thrift-0.9.1/compiler/cpp/thrift
>> build-support/thrift/serve/bin/thrift/local/0.9.1/thrift
>> >>> $ cd build-support/thrift/serve && (nohup python2 -m SimpleHTTPServer
>> 9999 >&/tmp/thrift.serve &) && cd -
>> >>>
>> >>> 5.) Run pants and confirm download of a thrift binary for your
>> machine:
>> >>> $ ./pants binary src/main/python/::
>> >>>
>> >>> 10:58:44 00:00 [main]
>> >>> (To run a reporting server: ./pants server)
>> >>> 10:58:44 00:00 [setup]
>> >>> 10:58:44 00:00 [parse]
>> >>> Executing tasks in goals: binary
>> >>> 10:58:44 00:00 [binary]
>> >>> 10:58:44 00:00 [binary-prep-command]
>> >>> 10:58:44 00:00 [python-binary-create]
>> >>> 10:58:44 00:00 [cache]
>> >>> No cached artifacts for 8 targets.
>> >>> Invalidated 8 targets.
>> >>> 10:58:44 00:00 [chroot]INFO] Attempting to fetch thrift binary
>> from: http://localhost:9999/bin/thrift/local/0.9.1/thrift ...
>> >>> 100% .................................................. 31661 KB
>> 0.364s
>> >>> INFO] Fetched thrift binary from:
>> http://localhost:9999/bin/thrift/local/0.9.1/thrift .
>> >>> ...
>> >>>
>> >>> Let me know what you get!
>> >>>
>> >>> On Thu, Jul 14, 2016 at 6:24 AM, Yiannis Gkoufas <
>> johngouf85@gmail.com> wrote:
>> >>>>
>> >>>> Cool, can you imagine a way to work around this?
>> >>>> I have to customize pants itself right?
>> >>>>
>> >>>> On 14 July 2016 at 11:54, Erb, Stephan <St...@blue-yonder.com>
>> wrote:
>> >>>>>
>> >>>>> Yeah, that sounds like it could be the issue here.
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>> From: Yiannis Gkoufas <jo...@gmail.com>
>> >>>>> Reply-To: "user@aurora.apache.org" <us...@aurora.apache.org>
>> >>>>> Date: Thursday 14 July 2016 at 12:28
>> >>>>> To: "user@aurora.apache.org" <us...@aurora.apache.org>
>> >>>>> Subject: Re: Build thermos executor on ARM
>> >>>>>
>> >>>>>
>> >>>>>>
>> >>>>>> Hi there,
>> >>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>>> sure:
>> >>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>>> 10:26:45 00:00 [binary]
>> >>>>>>
>> >>>>>> 10:26:45 00:00 [binary-jvm-prep-command]
>> >>>>>>
>> >>>>>> 10:26:46 00:01 [jvm_prep_command]
>> >>>>>>
>> >>>>>> 10:26:46 00:01 [binary-prep-command]
>> >>>>>>
>> >>>>>> 10:26:46 00:01 [python-binary-create]
>> >>>>>>
>> >>>>>> Detected interpreter /usr/bin/python2.7:
>> CPython-2.7.6
>> >>>>>>
>> >>>>>> 10:26:46 00:01 [chroot]
>> >>>>>>
>> >>>>>> Building chroot for
>> [PythonBinary(BuildFileAddress(BuildFile(src/main/python/apache/aurora/executor/BUILD,
>> FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)),
>> thermos_executor))]:
>> >>>>>>
>> >>>>>> Dumping library:
>> PythonLibrary(BuildFileAddress(BuildFile(src/main/python/apache/aurora/executor/BUILD,
>> FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)), _executor))
>> >>>>>>
>> >>>>>> Dumping library:
>> PythonLibrary(BuildFileAddress(BuildFile(api/src/main/thrift/org/apache/aurora/gen/BUILD,
>> FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)), gen))
>> >>>>>>
>> >>>>>> Dumping library:
>> PythonLibrary(BuildFileAddress(BuildFile(src/main/python/apache/aurora/common/BUILD,
>> FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)), common))
>> >>>>>>
>> >>>>>> Dumping library:
>> PythonLibrary(BuildFileAddress(BuildFile(src/main/python/apache/aurora/config/BUILD,
>> FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)), config))
>> >>>>>>
>> >>>>>> Dumping library:
>> PythonLibrary(BuildFileAddress(BuildFile(src/main/python/apache/thermos/config/BUILD,
>> FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)), config))
>> >>>>>>
>> >>>>>> Dumping library:
>> PythonLibrary(BuildFileAddress(BuildFile(src/main/python/apache/thermos/common/BUILD,
>> FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)), common))
>> >>>>>>
>> >>>>>> Dumping library:
>> PythonLibrary(BuildFileAddress(BuildFile(src/main/python/apache/thermos/core/BUILD,
>> FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)), core))
>> >>>>>>
>> >>>>>> Dumping library:
>> PythonLibrary(BuildFileAddress(BuildFile(src/main/python/apache/thermos/monitoring/BUILD,
>> FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)), monitoring))
>> >>>>>>
>> >>>>>> Dumping library:
>> PythonBinary(BuildFileAddress(BuildFile(src/main/python/apache/aurora/executor/BUILD,
>> FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)),
>> thermos_executor))
>> >>>>>>
>> >>>>>> FAILURE: Update --binaries-path-by-id to find binaries for Linux
>> armv7l 3.10.40
>> >>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>>> If I am not completely wrong, I think pants is looking one of the
>> architectures outlined here:
>> >>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>>>
>> https://github.com/pantsbuild/pants/blob/master/src/python/pants/binaries/binary_util.py#L24
>> >>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>>> and probably that's why it fails
>> >>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>>> On 14 July 2016 at 11:17, Erb, Stephan <
>> Stephan.Erb@blue-yonder.com> wrote:
>> >>>>>>>
>> >>>>>>> Cool, ARM J. Please try to run it like this, so that we get a
>> little bit more output:
>> >>>>>>>
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> PEX_VERBOSE=1 ./pants -ldebug binary
>> src/main/python/apache/aurora/executor:thermos_executor
>> >>>>>>>
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> From: Yiannis Gkoufas <jo...@gmail.com>
>> >>>>>>> Reply-To: "user@aurora.apache.org" <us...@aurora.apache.org>
>> >>>>>>> Date: Thursday 14 July 2016 at 10:44
>> >>>>>>> To: "user@aurora.apache.org" <us...@aurora.apache.org>
>> >>>>>>> Subject: Build thermos executor on ARM
>> >>>>>>>
>> >>>>>>>
>> >>>>>>>>
>> >>>>>>>> Hi guys,
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>> I have been trying to get aurora working on ARM processors.
>> >>>>>>>>
>> >>>>>>>> I had no problem with the scheduler, but I didn't manage to
>> compile the thermos executor.
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>> When I give:
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>> ./pants binary
>> src/main/python/apache/aurora/executor:thermos_executor
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>> I get back:
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>> FAILURE: Update --binaries-path-by-id to find binaries for Linux
>> armv7l 3.10.40
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>> Let me know if you have any hints on how to workaround this.
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>> Thanks!
>> >>>>>>
>> >>>>>>
>> >>>>
>> >>>>
>> >>>
>> >>
>> >
>>
>>
>>
>>
>
Re: Build thermos executor on ARM
Posted by Yiannis Gkoufas <jo...@gmail.com>.
great, so if I understood correctly, it just needs the *.egg files right?
since I have built it myself, what is the directory in which I would have
to copy them?
On 18 July 2016 at 16:52, Erb, Stephan <St...@blue-yonder.com> wrote:
> The Thermos executor is bundling the Mesos driver necessary for
> communication with to the agent. To make this work, you have to build
> Python egg files containing the Mesos driver. This is the script we
> normally use to achieve this:
> https://github.com/apache/aurora/blob/master/build-support/python/make-mesos-native-egg
>
>
>
>
>
> *From: *Yiannis Gkoufas <jo...@gmail.com>
> *Reply-To: *"user@aurora.apache.org" <us...@aurora.apache.org>
> *Date: *Monday 18 July 2016 at 17:35
>
> *To: *"user@aurora.apache.org" <us...@aurora.apache.org>
> *Subject: *Re: Build thermos executor on ARM
>
>
>
> So the error I am getting now is:
>
>
>
> 15:33:32 00:25 [chroot]INFO] Attempting to fetch thrift binary from:
> https://dl.bintray.com/pantsbuild/bin/build-support/bin/thrift/local/0.9.1/thrift
> ...
>
> INFO] Attempting to fetch thrift binary from:
> http://localhost:9999/bin/thrift/local/0.9.1/thrift ...
>
> 100% .................................................. 20270 KB 0.285s
>
> INFO] Fetched thrift binary from:
> http://localhost:9999/bin/thrift/local/0.9.1/thrift .
>
>
>
> Waiting for background workers to finish.
>
> 15:33:39 00:32 [complete]
>
> FAILURE
>
> Exception caught: (<class 'pex.resolver.Unsatisfiable'>)
>
>
>
> Exception message: Could not satisfy all requirements for
> mesos.native==0.28.2:
>
> mesos.native==0.28.2
>
>
>
> Which is kind of strange, because I have actually already installed mesos
> 0.28.2
>
>
>
> On 18 July 2016 at 16:16, John Sirois <jo...@gmail.com> wrote:
>
> On Jul 18, 2016 9:04 AM, "Yiannis Gkoufas" <jo...@gmail.com> wrote:
> >
> > Hi John,
> >
> > just to confirm: this is how this part of pants.ini should look like:
> >
> > backend_packages: [
> > [binaries]
> > baseurls: +['http://localhost:9999']
> > path_by_id: {('linux', 'armv7l'): ['local']}
> > ]
> >
> > right?
>
> Not quite, kill the surrounding backend_packages, so:
>
> [binaries]
> baseurls: +['http://localhost:9999']
> path_by_id: {('linux', 'armv7l'): ['local']}
>
> >
>
>
> > On 17 July 2016 at 11:58, Yiannis Gkoufas <jo...@gmail.com> wrote:
> >>
> >> Wow, thanks a lot John!
> >> Will give it a shot and get back to you!
> >>
> >> On 15 July 2016 at 18:05, John Sirois <jo...@gmail.com> wrote:
> >>>
> >>> Here's a workaround for you to try. If this works, I can modify the
> Aurora pants build without modifying pants itself to do all this
> automatically.
> >>>
> >>> The basic idea is that pants probes a list of URLs for pre-built
> binaries. You can add a local HTTP server to the list and point that
> server at the thrift binary Aurora builds for gradle via make.
> >>> Below is how you do this:
> >>>
> >>> 0.) Ensure the Aurora gradle thrift is built.
> >>> $ ./build-support/thrift/thriftw 0.9.1 --version
> >>> Thrift version 0.9.1
> >>>
> >>> 1.) Just for reference, the pants options in-play here
> >>> $ ./pants options | grep binaries
> >>> binaries.baseurls = ['
> https://dl.bintray.com/pantsbuild/bin/build-support'] (from HARDCODED)
> >>> binaries.fetch_timeout_secs = 30 (from HARDCODED)
> >>> binaries.path_by_id = {} (from NONE)
> >>> $ ./pants options | grep binary
> >>> thrift-binary.supportdir = bin/thrift (from HARDCODED)
> >>> thrift-binary.version = 0.9.1 (from CONFIG in pants.ini)
> >>>
> >>> 2.) Grab machine name pants will try to find your thrift binary under
> (here is an example from my machine with result 'x86_64' - yours will be
> different and arm-ey):
> >>> $ python2 -c 'import os; sysname, _, _, _, machine = os.uname(); print
> sysname.lower(), machine'
> >>> linux x86_64
> >>>
> >>> 3.) Edit pants.ini to tell pants where to look for binaries for your
> machine:
> >>> $ git diff -U1
> >>> diff --git a/pants.ini b/pants.ini
> >>> index de0ab30..e2cd94e 100644
> >>> --- a/pants.ini
> >>> +++ b/pants.ini
> >>> @@ -32,2 +32,6 @@ backend_packages: [
> >>>
> >>> +[binaries]
> >>> +baseurls: +['http://localhost:9999']
> >>> +path_by_id: {('linux', 'x86_64'): ['local']}
> >>> +
> >>> [thrift-binary]
> >>>
> >>> 4.) Setup the URL you told pants about above:
> >>> $ mkdir -p build-support/thrift/serve/bin/thrift/local/0.9.1
> >>> $ ln -s $PWD/build-support/thrift/thrift-0.9.1/compiler/cpp/thrift
> build-support/thrift/serve/bin/thrift/local/0.9.1/thrift
> >>> $ cd build-support/thrift/serve && (nohup python2 -m SimpleHTTPServer
> 9999 >&/tmp/thrift.serve &) && cd -
> >>>
> >>> 5.) Run pants and confirm download of a thrift binary for your machine:
> >>> $ ./pants binary src/main/python/::
> >>>
> >>> 10:58:44 00:00 [main]
> >>> (To run a reporting server: ./pants server)
> >>> 10:58:44 00:00 [setup]
> >>> 10:58:44 00:00 [parse]
> >>> Executing tasks in goals: binary
> >>> 10:58:44 00:00 [binary]
> >>> 10:58:44 00:00 [binary-prep-command]
> >>> 10:58:44 00:00 [python-binary-create]
> >>> 10:58:44 00:00 [cache]
> >>> No cached artifacts for 8 targets.
> >>> Invalidated 8 targets.
> >>> 10:58:44 00:00 [chroot]INFO] Attempting to fetch thrift binary
> from: http://localhost:9999/bin/thrift/local/0.9.1/thrift ...
> >>> 100% .................................................. 31661 KB 0.364s
> >>> INFO] Fetched thrift binary from:
> http://localhost:9999/bin/thrift/local/0.9.1/thrift .
> >>> ...
> >>>
> >>> Let me know what you get!
> >>>
> >>> On Thu, Jul 14, 2016 at 6:24 AM, Yiannis Gkoufas <jo...@gmail.com>
> wrote:
> >>>>
> >>>> Cool, can you imagine a way to work around this?
> >>>> I have to customize pants itself right?
> >>>>
> >>>> On 14 July 2016 at 11:54, Erb, Stephan <St...@blue-yonder.com>
> wrote:
> >>>>>
> >>>>> Yeah, that sounds like it could be the issue here.
> >>>>>
> >>>>>
> >>>>>
> >>>>> From: Yiannis Gkoufas <jo...@gmail.com>
> >>>>> Reply-To: "user@aurora.apache.org" <us...@aurora.apache.org>
> >>>>> Date: Thursday 14 July 2016 at 12:28
> >>>>> To: "user@aurora.apache.org" <us...@aurora.apache.org>
> >>>>> Subject: Re: Build thermos executor on ARM
> >>>>>
> >>>>>
> >>>>>>
> >>>>>> Hi there,
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> sure:
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> 10:26:45 00:00 [binary]
> >>>>>>
> >>>>>> 10:26:45 00:00 [binary-jvm-prep-command]
> >>>>>>
> >>>>>> 10:26:46 00:01 [jvm_prep_command]
> >>>>>>
> >>>>>> 10:26:46 00:01 [binary-prep-command]
> >>>>>>
> >>>>>> 10:26:46 00:01 [python-binary-create]
> >>>>>>
> >>>>>> Detected interpreter /usr/bin/python2.7:
> CPython-2.7.6
> >>>>>>
> >>>>>> 10:26:46 00:01 [chroot]
> >>>>>>
> >>>>>> Building chroot for
> [PythonBinary(BuildFileAddress(BuildFile(src/main/python/apache/aurora/executor/BUILD,
> FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)),
> thermos_executor))]:
> >>>>>>
> >>>>>> Dumping library:
> PythonLibrary(BuildFileAddress(BuildFile(src/main/python/apache/aurora/executor/BUILD,
> FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)), _executor))
> >>>>>>
> >>>>>> Dumping library:
> PythonLibrary(BuildFileAddress(BuildFile(api/src/main/thrift/org/apache/aurora/gen/BUILD,
> FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)), gen))
> >>>>>>
> >>>>>> Dumping library:
> PythonLibrary(BuildFileAddress(BuildFile(src/main/python/apache/aurora/common/BUILD,
> FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)), common))
> >>>>>>
> >>>>>> Dumping library:
> PythonLibrary(BuildFileAddress(BuildFile(src/main/python/apache/aurora/config/BUILD,
> FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)), config))
> >>>>>>
> >>>>>> Dumping library:
> PythonLibrary(BuildFileAddress(BuildFile(src/main/python/apache/thermos/config/BUILD,
> FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)), config))
> >>>>>>
> >>>>>> Dumping library:
> PythonLibrary(BuildFileAddress(BuildFile(src/main/python/apache/thermos/common/BUILD,
> FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)), common))
> >>>>>>
> >>>>>> Dumping library:
> PythonLibrary(BuildFileAddress(BuildFile(src/main/python/apache/thermos/core/BUILD,
> FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)), core))
> >>>>>>
> >>>>>> Dumping library:
> PythonLibrary(BuildFileAddress(BuildFile(src/main/python/apache/thermos/monitoring/BUILD,
> FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)), monitoring))
> >>>>>>
> >>>>>> Dumping library:
> PythonBinary(BuildFileAddress(BuildFile(src/main/python/apache/aurora/executor/BUILD,
> FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)),
> thermos_executor))
> >>>>>>
> >>>>>> FAILURE: Update --binaries-path-by-id to find binaries for Linux
> armv7l 3.10.40
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> If I am not completely wrong, I think pants is looking one of the
> architectures outlined here:
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> https://github.com/pantsbuild/pants/blob/master/src/python/pants/binaries/binary_util.py#L24
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> and probably that's why it fails
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> On 14 July 2016 at 11:17, Erb, Stephan <St...@blue-yonder.com>
> wrote:
> >>>>>>>
> >>>>>>> Cool, ARM J. Please try to run it like this, so that we get a
> little bit more output:
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> PEX_VERBOSE=1 ./pants -ldebug binary
> src/main/python/apache/aurora/executor:thermos_executor
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> From: Yiannis Gkoufas <jo...@gmail.com>
> >>>>>>> Reply-To: "user@aurora.apache.org" <us...@aurora.apache.org>
> >>>>>>> Date: Thursday 14 July 2016 at 10:44
> >>>>>>> To: "user@aurora.apache.org" <us...@aurora.apache.org>
> >>>>>>> Subject: Build thermos executor on ARM
> >>>>>>>
> >>>>>>>
> >>>>>>>>
> >>>>>>>> Hi guys,
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> I have been trying to get aurora working on ARM processors.
> >>>>>>>>
> >>>>>>>> I had no problem with the scheduler, but I didn't manage to
> compile the thermos executor.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> When I give:
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> ./pants binary
> src/main/python/apache/aurora/executor:thermos_executor
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> I get back:
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> FAILURE: Update --binaries-path-by-id to find binaries for Linux
> armv7l 3.10.40
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> Let me know if you have any hints on how to workaround this.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> Thanks!
> >>>>>>
> >>>>>>
> >>>>
> >>>>
> >>>
> >>
> >
>
>
>
>
Re: Build thermos executor on ARM
Posted by "Erb, Stephan" <St...@blue-yonder.com>.
The Thermos executor is bundling the Mesos driver necessary for communication with to the agent. To make this work, you have to build Python egg files containing the Mesos driver. This is the script we normally use to achieve this: https://github.com/apache/aurora/blob/master/build-support/python/make-mesos-native-egg
From: Yiannis Gkoufas <jo...@gmail.com>
Reply-To: "user@aurora.apache.org" <us...@aurora.apache.org>
Date: Monday 18 July 2016 at 17:35
To: "user@aurora.apache.org" <us...@aurora.apache.org>
Subject: Re: Build thermos executor on ARM
So the error I am getting now is:
15:33:32 00:25 [chroot]INFO] Attempting to fetch thrift binary from: https://dl.bintray.com/pantsbuild/bin/build-support/bin/thrift/local/0.9.1/thrift ...
INFO] Attempting to fetch thrift binary from: http://localhost:9999/bin/thrift/local/0.9.1/thrift ...
100% .................................................. 20270 KB 0.285s
INFO] Fetched thrift binary from: http://localhost:9999/bin/thrift/local/0.9.1/thrift .
Waiting for background workers to finish.
15:33:39 00:32 [complete]
FAILURE
Exception caught: (<class 'pex.resolver.Unsatisfiable'>)
Exception message: Could not satisfy all requirements for mesos.native==0.28.2:
mesos.native==0.28.2
Which is kind of strange, because I have actually already installed mesos 0.28.2
On 18 July 2016 at 16:16, John Sirois <jo...@gmail.com>> wrote:
On Jul 18, 2016 9:04 AM, "Yiannis Gkoufas" <jo...@gmail.com>> wrote:
>
> Hi John,
>
> just to confirm: this is how this part of pants.ini should look like:
>
> backend_packages: [
> [binaries]
> baseurls: +['http://localhost:9999']
> path_by_id: {('linux', 'armv7l'): ['local']}
> ]
>
> right?
Not quite, kill the surrounding backend_packages, so:
[binaries]
baseurls: +['http://localhost:9999']
path_by_id: {('linux', 'armv7l'): ['local']}
>
> On 17 July 2016 at 11:58, Yiannis Gkoufas <jo...@gmail.com>> wrote:
>>
>> Wow, thanks a lot John!
>> Will give it a shot and get back to you!
>>
>> On 15 July 2016 at 18:05, John Sirois <jo...@gmail.com>> wrote:
>>>
>>> Here's a workaround for you to try. If this works, I can modify the Aurora pants build without modifying pants itself to do all this automatically.
>>>
>>> The basic idea is that pants probes a list of URLs for pre-built binaries. You can add a local HTTP server to the list and point that server at the thrift binary Aurora builds for gradle via make.
>>> Below is how you do this:
>>>
>>> 0.) Ensure the Aurora gradle thrift is built.
>>> $ ./build-support/thrift/thriftw 0.9.1 --version
>>> Thrift version 0.9.1
>>>
>>> 1.) Just for reference, the pants options in-play here
>>> $ ./pants options | grep binaries
>>> binaries.baseurls = ['https://dl.bintray.com/pantsbuild/bin/build-support'] (from HARDCODED)
>>> binaries.fetch_timeout_secs = 30 (from HARDCODED)
>>> binaries.path_by_id = {} (from NONE)
>>> $ ./pants options | grep binary
>>> thrift-binary.supportdir = bin/thrift (from HARDCODED)
>>> thrift-binary.version = 0.9.1 (from CONFIG in pants.ini)
>>>
>>> 2.) Grab machine name pants will try to find your thrift binary under (here is an example from my machine with result 'x86_64' - yours will be different and arm-ey):
>>> $ python2 -c 'import os; sysname, _, _, _, machine = os.uname(); print sysname.lower(), machine'
>>> linux x86_64
>>>
>>> 3.) Edit pants.ini to tell pants where to look for binaries for your machine:
>>> $ git diff -U1
>>> diff --git a/pants.ini b/pants.ini
>>> index de0ab30..e2cd94e 100644
>>> --- a/pants.ini
>>> +++ b/pants.ini
>>> @@ -32,2 +32,6 @@ backend_packages: [
>>>
>>> +[binaries]
>>> +baseurls: +['http://localhost:9999']
>>> +path_by_id: {('linux', 'x86_64'): ['local']}
>>> +
>>> [thrift-binary]
>>>
>>> 4.) Setup the URL you told pants about above:
>>> $ mkdir -p build-support/thrift/serve/bin/thrift/local/0.9.1
>>> $ ln -s $PWD/build-support/thrift/thrift-0.9.1/compiler/cpp/thrift build-support/thrift/serve/bin/thrift/local/0.9.1/thrift
>>> $ cd build-support/thrift/serve && (nohup python2 -m SimpleHTTPServer 9999 >&/tmp/thrift.serve &) && cd -
>>>
>>> 5.) Run pants and confirm download of a thrift binary for your machine:
>>> $ ./pants binary src/main/python/::
>>>
>>> 10:58:44 00:00 [main]
>>> (To run a reporting server: ./pants server)
>>> 10:58:44 00:00 [setup]
>>> 10:58:44 00:00 [parse]
>>> Executing tasks in goals: binary
>>> 10:58:44 00:00 [binary]
>>> 10:58:44 00:00 [binary-prep-command]
>>> 10:58:44 00:00 [python-binary-create]
>>> 10:58:44 00:00 [cache]
>>> No cached artifacts for 8 targets.
>>> Invalidated 8 targets.
>>> 10:58:44 00:00 [chroot]INFO] Attempting to fetch thrift binary from: http://localhost:9999/bin/thrift/local/0.9.1/thrift ...
>>> 100% .................................................. 31661 KB 0.364s
>>> INFO] Fetched thrift binary from: http://localhost:9999/bin/thrift/local/0.9.1/thrift .
>>> ...
>>>
>>> Let me know what you get!
>>>
>>> On Thu, Jul 14, 2016 at 6:24 AM, Yiannis Gkoufas <jo...@gmail.com>> wrote:
>>>>
>>>> Cool, can you imagine a way to work around this?
>>>> I have to customize pants itself right?
>>>>
>>>> On 14 July 2016 at 11:54, Erb, Stephan <St...@blue-yonder.com>> wrote:
>>>>>
>>>>> Yeah, that sounds like it could be the issue here.
>>>>>
>>>>>
>>>>>
>>>>> From: Yiannis Gkoufas <jo...@gmail.com>>
>>>>> Reply-To: "user@aurora.apache.org<ma...@aurora.apache.org>" <us...@aurora.apache.org>>
>>>>> Date: Thursday 14 July 2016 at 12:28
>>>>> To: "user@aurora.apache.org<ma...@aurora.apache.org>" <us...@aurora.apache.org>>
>>>>> Subject: Re: Build thermos executor on ARM
>>>>>
>>>>>
>>>>>>
>>>>>> Hi there,
>>>>>>
>>>>>>
>>>>>>
>>>>>> sure:
>>>>>>
>>>>>>
>>>>>>
>>>>>> 10:26:45 00:00 [binary]
>>>>>>
>>>>>> 10:26:45 00:00 [binary-jvm-prep-command]
>>>>>>
>>>>>> 10:26:46 00:01 [jvm_prep_command]
>>>>>>
>>>>>> 10:26:46 00:01 [binary-prep-command]
>>>>>>
>>>>>> 10:26:46 00:01 [python-binary-create]
>>>>>>
>>>>>> Detected interpreter /usr/bin/python2.7: CPython-2.7.6
>>>>>>
>>>>>> 10:26:46 00:01 [chroot]
>>>>>>
>>>>>> Building chroot for [PythonBinary(BuildFileAddress(BuildFile(src/main/python/apache/aurora/executor/BUILD, FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)), thermos_executor))]:
>>>>>>
>>>>>> Dumping library: PythonLibrary(BuildFileAddress(BuildFile(src/main/python/apache/aurora/executor/BUILD, FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)), _executor))
>>>>>>
>>>>>> Dumping library: PythonLibrary(BuildFileAddress(BuildFile(api/src/main/thrift/org/apache/aurora/gen/BUILD, FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)), gen))
>>>>>>
>>>>>> Dumping library: PythonLibrary(BuildFileAddress(BuildFile(src/main/python/apache/aurora/common/BUILD, FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)), common))
>>>>>>
>>>>>> Dumping library: PythonLibrary(BuildFileAddress(BuildFile(src/main/python/apache/aurora/config/BUILD, FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)), config))
>>>>>>
>>>>>> Dumping library: PythonLibrary(BuildFileAddress(BuildFile(src/main/python/apache/thermos/config/BUILD, FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)), config))
>>>>>>
>>>>>> Dumping library: PythonLibrary(BuildFileAddress(BuildFile(src/main/python/apache/thermos/common/BUILD, FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)), common))
>>>>>>
>>>>>> Dumping library: PythonLibrary(BuildFileAddress(BuildFile(src/main/python/apache/thermos/core/BUILD, FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)), core))
>>>>>>
>>>>>> Dumping library: PythonLibrary(BuildFileAddress(BuildFile(src/main/python/apache/thermos/monitoring/BUILD, FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)), monitoring))
>>>>>>
>>>>>> Dumping library: PythonBinary(BuildFileAddress(BuildFile(src/main/python/apache/aurora/executor/BUILD, FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)), thermos_executor))
>>>>>>
>>>>>> FAILURE: Update --binaries-path-by-id to find binaries for Linux armv7l 3.10.40
>>>>>>
>>>>>>
>>>>>>
>>>>>> If I am not completely wrong, I think pants is looking one of the architectures outlined here:
>>>>>>
>>>>>>
>>>>>>
>>>>>> https://github.com/pantsbuild/pants/blob/master/src/python/pants/binaries/binary_util.py#L24
>>>>>>
>>>>>>
>>>>>>
>>>>>> and probably that's why it fails
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 14 July 2016 at 11:17, Erb, Stephan <St...@blue-yonder.com>> wrote:
>>>>>>>
>>>>>>> Cool, ARM J. Please try to run it like this, so that we get a little bit more output:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> PEX_VERBOSE=1 ./pants -ldebug binary src/main/python/apache/aurora/executor:thermos_executor
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> From: Yiannis Gkoufas <jo...@gmail.com>>
>>>>>>> Reply-To: "user@aurora.apache.org<ma...@aurora.apache.org>" <us...@aurora.apache.org>>
>>>>>>> Date: Thursday 14 July 2016 at 10:44
>>>>>>> To: "user@aurora.apache.org<ma...@aurora.apache.org>" <us...@aurora.apache.org>>
>>>>>>> Subject: Build thermos executor on ARM
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>> Hi guys,
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> I have been trying to get aurora working on ARM processors.
>>>>>>>>
>>>>>>>> I had no problem with the scheduler, but I didn't manage to compile the thermos executor.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> When I give:
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> ./pants binary src/main/python/apache/aurora/executor:thermos_executor
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> I get back:
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> FAILURE: Update --binaries-path-by-id to find binaries for Linux armv7l 3.10.40
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Let me know if you have any hints on how to workaround this.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Thanks!
>>>>>>
>>>>>>
>>>>
>>>>
>>>
>>
>
Re: Build thermos executor on ARM
Posted by Yiannis Gkoufas <jo...@gmail.com>.
So the error I am getting now is:
15:33:32 00:25 [chroot]INFO] Attempting to fetch thrift binary from:
https://dl.bintray.com/pantsbuild/bin/build-support/bin/thrift/local/0.9.1/thrift
...
INFO] Attempting to fetch thrift binary from:
http://localhost:9999/bin/thrift/local/0.9.1/thrift ...
100% .................................................. 20270 KB 0.285s
INFO] Fetched thrift binary from:
http://localhost:9999/bin/thrift/local/0.9.1/thrift .
Waiting for background workers to finish.
15:33:39 00:32 [complete]
FAILURE
Exception caught: (<class 'pex.resolver.Unsatisfiable'>)
Exception message: Could not satisfy all requirements for
mesos.native==0.28.2:
mesos.native==0.28.2
Which is kind of strange, because I have actually already installed mesos
0.28.2
On 18 July 2016 at 16:16, John Sirois <jo...@gmail.com> wrote:
> On Jul 18, 2016 9:04 AM, "Yiannis Gkoufas" <jo...@gmail.com> wrote:
> >
> > Hi John,
> >
> > just to confirm: this is how this part of pants.ini should look like:
> >
> > backend_packages: [
> > [binaries]
> > baseurls: +['http://localhost:9999']
> > path_by_id: {('linux', 'armv7l'): ['local']}
> > ]
> >
> > right?
>
> Not quite, kill the surrounding backend_packages, so:
>
> [binaries]
> baseurls: +['http://localhost:9999']
> path_by_id: {('linux', 'armv7l'): ['local']}
>
> >
>
> > On 17 July 2016 at 11:58, Yiannis Gkoufas <jo...@gmail.com> wrote:
> >>
> >> Wow, thanks a lot John!
> >> Will give it a shot and get back to you!
> >>
> >> On 15 July 2016 at 18:05, John Sirois <jo...@gmail.com> wrote:
> >>>
> >>> Here's a workaround for you to try. If this works, I can modify the
> Aurora pants build without modifying pants itself to do all this
> automatically.
> >>>
> >>> The basic idea is that pants probes a list of URLs for pre-built
> binaries. You can add a local HTTP server to the list and point that
> server at the thrift binary Aurora builds for gradle via make.
> >>> Below is how you do this:
> >>>
> >>> 0.) Ensure the Aurora gradle thrift is built.
> >>> $ ./build-support/thrift/thriftw 0.9.1 --version
> >>> Thrift version 0.9.1
> >>>
> >>> 1.) Just for reference, the pants options in-play here
> >>> $ ./pants options | grep binaries
> >>> binaries.baseurls = ['
> https://dl.bintray.com/pantsbuild/bin/build-support'] (from HARDCODED)
> >>> binaries.fetch_timeout_secs = 30 (from HARDCODED)
> >>> binaries.path_by_id = {} (from NONE)
> >>> $ ./pants options | grep binary
> >>> thrift-binary.supportdir = bin/thrift (from HARDCODED)
> >>> thrift-binary.version = 0.9.1 (from CONFIG in pants.ini)
> >>>
> >>> 2.) Grab machine name pants will try to find your thrift binary under
> (here is an example from my machine with result 'x86_64' - yours will be
> different and arm-ey):
> >>> $ python2 -c 'import os; sysname, _, _, _, machine = os.uname(); print
> sysname.lower(), machine'
> >>> linux x86_64
> >>>
> >>> 3.) Edit pants.ini to tell pants where to look for binaries for your
> machine:
> >>> $ git diff -U1
> >>> diff --git a/pants.ini b/pants.ini
> >>> index de0ab30..e2cd94e 100644
> >>> --- a/pants.ini
> >>> +++ b/pants.ini
> >>> @@ -32,2 +32,6 @@ backend_packages: [
> >>>
> >>> +[binaries]
> >>> +baseurls: +['http://localhost:9999']
> >>> +path_by_id: {('linux', 'x86_64'): ['local']}
> >>> +
> >>> [thrift-binary]
> >>>
> >>> 4.) Setup the URL you told pants about above:
> >>> $ mkdir -p build-support/thrift/serve/bin/thrift/local/0.9.1
> >>> $ ln -s $PWD/build-support/thrift/thrift-0.9.1/compiler/cpp/thrift
> build-support/thrift/serve/bin/thrift/local/0.9.1/thrift
> >>> $ cd build-support/thrift/serve && (nohup python2 -m SimpleHTTPServer
> 9999 >&/tmp/thrift.serve &) && cd -
> >>>
> >>> 5.) Run pants and confirm download of a thrift binary for your machine:
> >>> $ ./pants binary src/main/python/::
> >>>
> >>> 10:58:44 00:00 [main]
> >>> (To run a reporting server: ./pants server)
> >>> 10:58:44 00:00 [setup]
> >>> 10:58:44 00:00 [parse]
> >>> Executing tasks in goals: binary
> >>> 10:58:44 00:00 [binary]
> >>> 10:58:44 00:00 [binary-prep-command]
> >>> 10:58:44 00:00 [python-binary-create]
> >>> 10:58:44 00:00 [cache]
> >>> No cached artifacts for 8 targets.
> >>> Invalidated 8 targets.
> >>> 10:58:44 00:00 [chroot]INFO] Attempting to fetch thrift binary
> from: http://localhost:9999/bin/thrift/local/0.9.1/thrift ...
> >>> 100% .................................................. 31661 KB 0.364s
> >>> INFO] Fetched thrift binary from:
> http://localhost:9999/bin/thrift/local/0.9.1/thrift .
> >>> ...
> >>>
> >>> Let me know what you get!
> >>>
> >>> On Thu, Jul 14, 2016 at 6:24 AM, Yiannis Gkoufas <jo...@gmail.com>
> wrote:
> >>>>
> >>>> Cool, can you imagine a way to work around this?
> >>>> I have to customize pants itself right?
> >>>>
> >>>> On 14 July 2016 at 11:54, Erb, Stephan <St...@blue-yonder.com>
> wrote:
> >>>>>
> >>>>> Yeah, that sounds like it could be the issue here.
> >>>>>
> >>>>>
> >>>>>
> >>>>> From: Yiannis Gkoufas <jo...@gmail.com>
> >>>>> Reply-To: "user@aurora.apache.org" <us...@aurora.apache.org>
> >>>>> Date: Thursday 14 July 2016 at 12:28
> >>>>> To: "user@aurora.apache.org" <us...@aurora.apache.org>
> >>>>> Subject: Re: Build thermos executor on ARM
> >>>>>
> >>>>>
> >>>>>>
> >>>>>> Hi there,
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> sure:
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> 10:26:45 00:00 [binary]
> >>>>>>
> >>>>>> 10:26:45 00:00 [binary-jvm-prep-command]
> >>>>>>
> >>>>>> 10:26:46 00:01 [jvm_prep_command]
> >>>>>>
> >>>>>> 10:26:46 00:01 [binary-prep-command]
> >>>>>>
> >>>>>> 10:26:46 00:01 [python-binary-create]
> >>>>>>
> >>>>>> Detected interpreter /usr/bin/python2.7:
> CPython-2.7.6
> >>>>>>
> >>>>>> 10:26:46 00:01 [chroot]
> >>>>>>
> >>>>>> Building chroot for
> [PythonBinary(BuildFileAddress(BuildFile(src/main/python/apache/aurora/executor/BUILD,
> FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)),
> thermos_executor))]:
> >>>>>>
> >>>>>> Dumping library:
> PythonLibrary(BuildFileAddress(BuildFile(src/main/python/apache/aurora/executor/BUILD,
> FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)), _executor))
> >>>>>>
> >>>>>> Dumping library:
> PythonLibrary(BuildFileAddress(BuildFile(api/src/main/thrift/org/apache/aurora/gen/BUILD,
> FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)), gen))
> >>>>>>
> >>>>>> Dumping library:
> PythonLibrary(BuildFileAddress(BuildFile(src/main/python/apache/aurora/common/BUILD,
> FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)), common))
> >>>>>>
> >>>>>> Dumping library:
> PythonLibrary(BuildFileAddress(BuildFile(src/main/python/apache/aurora/config/BUILD,
> FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)), config))
> >>>>>>
> >>>>>> Dumping library:
> PythonLibrary(BuildFileAddress(BuildFile(src/main/python/apache/thermos/config/BUILD,
> FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)), config))
> >>>>>>
> >>>>>> Dumping library:
> PythonLibrary(BuildFileAddress(BuildFile(src/main/python/apache/thermos/common/BUILD,
> FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)), common))
> >>>>>>
> >>>>>> Dumping library:
> PythonLibrary(BuildFileAddress(BuildFile(src/main/python/apache/thermos/core/BUILD,
> FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)), core))
> >>>>>>
> >>>>>> Dumping library:
> PythonLibrary(BuildFileAddress(BuildFile(src/main/python/apache/thermos/monitoring/BUILD,
> FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)), monitoring))
> >>>>>>
> >>>>>> Dumping library:
> PythonBinary(BuildFileAddress(BuildFile(src/main/python/apache/aurora/executor/BUILD,
> FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)),
> thermos_executor))
> >>>>>>
> >>>>>> FAILURE: Update --binaries-path-by-id to find binaries for Linux
> armv7l 3.10.40
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> If I am not completely wrong, I think pants is looking one of the
> architectures outlined here:
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> https://github.com/pantsbuild/pants/blob/master/src/python/pants/binaries/binary_util.py#L24
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> and probably that's why it fails
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> On 14 July 2016 at 11:17, Erb, Stephan <St...@blue-yonder.com>
> wrote:
> >>>>>>>
> >>>>>>> Cool, ARM J. Please try to run it like this, so that we get a
> little bit more output:
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> PEX_VERBOSE=1 ./pants -ldebug binary
> src/main/python/apache/aurora/executor:thermos_executor
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> From: Yiannis Gkoufas <jo...@gmail.com>
> >>>>>>> Reply-To: "user@aurora.apache.org" <us...@aurora.apache.org>
> >>>>>>> Date: Thursday 14 July 2016 at 10:44
> >>>>>>> To: "user@aurora.apache.org" <us...@aurora.apache.org>
> >>>>>>> Subject: Build thermos executor on ARM
> >>>>>>>
> >>>>>>>
> >>>>>>>>
> >>>>>>>> Hi guys,
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> I have been trying to get aurora working on ARM processors.
> >>>>>>>>
> >>>>>>>> I had no problem with the scheduler, but I didn't manage to
> compile the thermos executor.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> When I give:
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> ./pants binary
> src/main/python/apache/aurora/executor:thermos_executor
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> I get back:
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> FAILURE: Update --binaries-path-by-id to find binaries for Linux
> armv7l 3.10.40
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> Let me know if you have any hints on how to workaround this.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> Thanks!
> >>>>>>
> >>>>>>
> >>>>
> >>>>
> >>>
> >>
> >
>
>
Re: Build thermos executor on ARM
Posted by John Sirois <jo...@gmail.com>.
On Jul 18, 2016 9:04 AM, "Yiannis Gkoufas" <jo...@gmail.com> wrote:
>
> Hi John,
>
> just to confirm: this is how this part of pants.ini should look like:
>
> backend_packages: [
> [binaries]
> baseurls: +['http://localhost:9999']
> path_by_id: {('linux', 'armv7l'): ['local']}
> ]
>
> right?
Not quite, kill the surrounding backend_packages, so:
[binaries]
baseurls: +['http://localhost:9999']
path_by_id: {('linux', 'armv7l'): ['local']}
>
> On 17 July 2016 at 11:58, Yiannis Gkoufas <jo...@gmail.com> wrote:
>>
>> Wow, thanks a lot John!
>> Will give it a shot and get back to you!
>>
>> On 15 July 2016 at 18:05, John Sirois <jo...@gmail.com> wrote:
>>>
>>> Here's a workaround for you to try. If this works, I can modify the
Aurora pants build without modifying pants itself to do all this
automatically.
>>>
>>> The basic idea is that pants probes a list of URLs for pre-built
binaries. You can add a local HTTP server to the list and point that
server at the thrift binary Aurora builds for gradle via make.
>>> Below is how you do this:
>>>
>>> 0.) Ensure the Aurora gradle thrift is built.
>>> $ ./build-support/thrift/thriftw 0.9.1 --version
>>> Thrift version 0.9.1
>>>
>>> 1.) Just for reference, the pants options in-play here
>>> $ ./pants options | grep binaries
>>> binaries.baseurls = ['
https://dl.bintray.com/pantsbuild/bin/build-support'] (from HARDCODED)
>>> binaries.fetch_timeout_secs = 30 (from HARDCODED)
>>> binaries.path_by_id = {} (from NONE)
>>> $ ./pants options | grep binary
>>> thrift-binary.supportdir = bin/thrift (from HARDCODED)
>>> thrift-binary.version = 0.9.1 (from CONFIG in pants.ini)
>>>
>>> 2.) Grab machine name pants will try to find your thrift binary under
(here is an example from my machine with result 'x86_64' - yours will be
different and arm-ey):
>>> $ python2 -c 'import os; sysname, _, _, _, machine = os.uname(); print
sysname.lower(), machine'
>>> linux x86_64
>>>
>>> 3.) Edit pants.ini to tell pants where to look for binaries for your
machine:
>>> $ git diff -U1
>>> diff --git a/pants.ini b/pants.ini
>>> index de0ab30..e2cd94e 100644
>>> --- a/pants.ini
>>> +++ b/pants.ini
>>> @@ -32,2 +32,6 @@ backend_packages: [
>>>
>>> +[binaries]
>>> +baseurls: +['http://localhost:9999']
>>> +path_by_id: {('linux', 'x86_64'): ['local']}
>>> +
>>> [thrift-binary]
>>>
>>> 4.) Setup the URL you told pants about above:
>>> $ mkdir -p build-support/thrift/serve/bin/thrift/local/0.9.1
>>> $ ln -s $PWD/build-support/thrift/thrift-0.9.1/compiler/cpp/thrift
build-support/thrift/serve/bin/thrift/local/0.9.1/thrift
>>> $ cd build-support/thrift/serve && (nohup python2 -m SimpleHTTPServer
9999 >&/tmp/thrift.serve &) && cd -
>>>
>>> 5.) Run pants and confirm download of a thrift binary for your machine:
>>> $ ./pants binary src/main/python/::
>>>
>>> 10:58:44 00:00 [main]
>>> (To run a reporting server: ./pants server)
>>> 10:58:44 00:00 [setup]
>>> 10:58:44 00:00 [parse]
>>> Executing tasks in goals: binary
>>> 10:58:44 00:00 [binary]
>>> 10:58:44 00:00 [binary-prep-command]
>>> 10:58:44 00:00 [python-binary-create]
>>> 10:58:44 00:00 [cache]
>>> No cached artifacts for 8 targets.
>>> Invalidated 8 targets.
>>> 10:58:44 00:00 [chroot]INFO] Attempting to fetch thrift binary
from: http://localhost:9999/bin/thrift/local/0.9.1/thrift ...
>>> 100% .................................................. 31661 KB 0.364s
>>> INFO] Fetched thrift binary from:
http://localhost:9999/bin/thrift/local/0.9.1/thrift .
>>> ...
>>>
>>> Let me know what you get!
>>>
>>> On Thu, Jul 14, 2016 at 6:24 AM, Yiannis Gkoufas <jo...@gmail.com>
wrote:
>>>>
>>>> Cool, can you imagine a way to work around this?
>>>> I have to customize pants itself right?
>>>>
>>>> On 14 July 2016 at 11:54, Erb, Stephan <St...@blue-yonder.com>
wrote:
>>>>>
>>>>> Yeah, that sounds like it could be the issue here.
>>>>>
>>>>>
>>>>>
>>>>> From: Yiannis Gkoufas <jo...@gmail.com>
>>>>> Reply-To: "user@aurora.apache.org" <us...@aurora.apache.org>
>>>>> Date: Thursday 14 July 2016 at 12:28
>>>>> To: "user@aurora.apache.org" <us...@aurora.apache.org>
>>>>> Subject: Re: Build thermos executor on ARM
>>>>>
>>>>>
>>>>>>
>>>>>> Hi there,
>>>>>>
>>>>>>
>>>>>>
>>>>>> sure:
>>>>>>
>>>>>>
>>>>>>
>>>>>> 10:26:45 00:00 [binary]
>>>>>>
>>>>>> 10:26:45 00:00 [binary-jvm-prep-command]
>>>>>>
>>>>>> 10:26:46 00:01 [jvm_prep_command]
>>>>>>
>>>>>> 10:26:46 00:01 [binary-prep-command]
>>>>>>
>>>>>> 10:26:46 00:01 [python-binary-create]
>>>>>>
>>>>>> Detected interpreter /usr/bin/python2.7:
CPython-2.7.6
>>>>>>
>>>>>> 10:26:46 00:01 [chroot]
>>>>>>
>>>>>> Building chroot for
[PythonBinary(BuildFileAddress(BuildFile(src/main/python/apache/aurora/executor/BUILD,
FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)),
thermos_executor))]:
>>>>>>
>>>>>> Dumping library:
PythonLibrary(BuildFileAddress(BuildFile(src/main/python/apache/aurora/executor/BUILD,
FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)), _executor))
>>>>>>
>>>>>> Dumping library:
PythonLibrary(BuildFileAddress(BuildFile(api/src/main/thrift/org/apache/aurora/gen/BUILD,
FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)), gen))
>>>>>>
>>>>>> Dumping library:
PythonLibrary(BuildFileAddress(BuildFile(src/main/python/apache/aurora/common/BUILD,
FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)), common))
>>>>>>
>>>>>> Dumping library:
PythonLibrary(BuildFileAddress(BuildFile(src/main/python/apache/aurora/config/BUILD,
FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)), config))
>>>>>>
>>>>>> Dumping library:
PythonLibrary(BuildFileAddress(BuildFile(src/main/python/apache/thermos/config/BUILD,
FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)), config))
>>>>>>
>>>>>> Dumping library:
PythonLibrary(BuildFileAddress(BuildFile(src/main/python/apache/thermos/common/BUILD,
FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)), common))
>>>>>>
>>>>>> Dumping library:
PythonLibrary(BuildFileAddress(BuildFile(src/main/python/apache/thermos/core/BUILD,
FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)), core))
>>>>>>
>>>>>> Dumping library:
PythonLibrary(BuildFileAddress(BuildFile(src/main/python/apache/thermos/monitoring/BUILD,
FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)), monitoring))
>>>>>>
>>>>>> Dumping library:
PythonBinary(BuildFileAddress(BuildFile(src/main/python/apache/aurora/executor/BUILD,
FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)),
thermos_executor))
>>>>>>
>>>>>> FAILURE: Update --binaries-path-by-id to find binaries for Linux
armv7l 3.10.40
>>>>>>
>>>>>>
>>>>>>
>>>>>> If I am not completely wrong, I think pants is looking one of the
architectures outlined here:
>>>>>>
>>>>>>
>>>>>>
>>>>>>
https://github.com/pantsbuild/pants/blob/master/src/python/pants/binaries/binary_util.py#L24
>>>>>>
>>>>>>
>>>>>>
>>>>>> and probably that's why it fails
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 14 July 2016 at 11:17, Erb, Stephan <St...@blue-yonder.com>
wrote:
>>>>>>>
>>>>>>> Cool, ARM J. Please try to run it like this, so that we get a
little bit more output:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> PEX_VERBOSE=1 ./pants -ldebug binary
src/main/python/apache/aurora/executor:thermos_executor
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> From: Yiannis Gkoufas <jo...@gmail.com>
>>>>>>> Reply-To: "user@aurora.apache.org" <us...@aurora.apache.org>
>>>>>>> Date: Thursday 14 July 2016 at 10:44
>>>>>>> To: "user@aurora.apache.org" <us...@aurora.apache.org>
>>>>>>> Subject: Build thermos executor on ARM
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>> Hi guys,
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> I have been trying to get aurora working on ARM processors.
>>>>>>>>
>>>>>>>> I had no problem with the scheduler, but I didn't manage to
compile the thermos executor.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> When I give:
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> ./pants binary
src/main/python/apache/aurora/executor:thermos_executor
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> I get back:
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> FAILURE: Update --binaries-path-by-id to find binaries for Linux
armv7l 3.10.40
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Let me know if you have any hints on how to workaround this.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Thanks!
>>>>>>
>>>>>>
>>>>
>>>>
>>>
>>
>
Re: Build thermos executor on ARM
Posted by Yiannis Gkoufas <jo...@gmail.com>.
Hi John,
just to confirm: this is how this part of pants.ini should look like:
backend_packages: [
[binaries]
baseurls: +['http://localhost:9999']
path_by_id: {('linux', 'armv7l'): ['local']}
]
right?
On 17 July 2016 at 11:58, Yiannis Gkoufas <jo...@gmail.com> wrote:
> Wow, thanks a lot John!
> Will give it a shot and get back to you!
>
> On 15 July 2016 at 18:05, John Sirois <jo...@gmail.com> wrote:
>
>> Here's a workaround for you to try. If this works, I can modify the
>> Aurora pants build without modifying pants itself to do all this
>> automatically.
>>
>> The basic idea is that pants probes a list of URLs for pre-built
>> binaries. You can add a local HTTP server to the list and point that
>> server at the thrift binary Aurora builds for gradle via make.
>> Below is how you do this:
>>
>> 0.) Ensure the Aurora gradle thrift is built.
>> $ ./build-support/thrift/thriftw 0.9.1 --version
>> Thrift version 0.9.1
>>
>> 1.) Just for reference, the pants options in-play here
>> $ ./pants options | grep binaries
>> binaries.baseurls = ['https://dl.bintray.com/pantsbuild/bin/build-support']
>> (from HARDCODED)
>> binaries.fetch_timeout_secs = 30 (from HARDCODED)
>> binaries.path_by_id = {} (from NONE)
>> $ ./pants options | grep binary
>> thrift-binary.supportdir = bin/thrift (from HARDCODED)
>> thrift-binary.version = 0.9.1 (from CONFIG in pants.ini)
>>
>> 2.) Grab machine name pants will try to find your thrift binary under
>> (here is an example from my machine with result 'x86_64' - yours will be
>> different and arm-ey):
>> $ python2 -c 'import os; sysname, _, _, _, machine = os.uname(); print
>> sysname.lower(), machine'
>> linux x86_64
>>
>> 3.) Edit pants.ini to tell pants where to look for binaries for your
>> machine:
>> $ git diff -U1
>> diff --git a/pants.ini b/pants.ini
>> index de0ab30..e2cd94e 100644
>> --- a/pants.ini
>> +++ b/pants.ini
>> @@ -32,2 +32,6 @@ backend_packages: [
>>
>> +[binaries]
>> +baseurls: +['http://localhost:9999']
>> +path_by_id: {('linux', 'x86_64'): ['local']}
>> +
>> [thrift-binary]
>>
>> 4.) Setup the URL you told pants about above:
>> $ mkdir -p build-support/thrift/serve/bin/thrift/local/0.9.1
>> $ ln -s $PWD/build-support/thrift/thrift-0.9.1/compiler/cpp/thrift
>> build-support/thrift/serve/bin/thrift/local/0.9.1/thrift
>> $ cd build-support/thrift/serve && (nohup python2 -m SimpleHTTPServer
>> 9999 >&/tmp/thrift.serve &) && cd -
>>
>> 5.) Run pants and confirm download of a thrift binary for your machine:
>> $ ./pants binary src/main/python/::
>>
>> 10:58:44 00:00 [main]
>> (To run a reporting server: ./pants server)
>> 10:58:44 00:00 [setup]
>> 10:58:44 00:00 [parse]
>> Executing tasks in goals: binary
>> 10:58:44 00:00 [binary]
>> 10:58:44 00:00 [binary-prep-command]
>> 10:58:44 00:00 [python-binary-create]
>> 10:58:44 00:00 [cache]
>> No cached artifacts for 8 targets.
>> Invalidated 8 targets.
>> 10:58:44 00:00 [chroot]INFO] Attempting to fetch thrift binary
>> from: http://localhost:9999/bin/thrift/local/0.9.1/thrift ...
>> 100% .................................................. 31661 KB 0.364s
>> INFO] Fetched thrift binary from:
>> http://localhost:9999/bin/thrift/local/0.9.1/thrift .
>> ...
>>
>> Let me know what you get!
>>
>> On Thu, Jul 14, 2016 at 6:24 AM, Yiannis Gkoufas <jo...@gmail.com>
>> wrote:
>>
>>> Cool, can you imagine a way to work around this?
>>> I have to customize pants itself right?
>>>
>>> On 14 July 2016 at 11:54, Erb, Stephan <St...@blue-yonder.com>
>>> wrote:
>>>
>>>> Yeah, that sounds like it could be the issue here.
>>>>
>>>>
>>>>
>>>> *From: *Yiannis Gkoufas <jo...@gmail.com>
>>>> *Reply-To: *"user@aurora.apache.org" <us...@aurora.apache.org>
>>>> *Date: *Thursday 14 July 2016 at 12:28
>>>> *To: *"user@aurora.apache.org" <us...@aurora.apache.org>
>>>> *Subject: *Re: Build thermos executor on ARM
>>>>
>>>>
>>>>
>>>> Hi there,
>>>>
>>>>
>>>>
>>>> sure:
>>>>
>>>>
>>>>
>>>> 10:26:45 00:00 [binary]
>>>>
>>>> 10:26:45 00:00 [binary-jvm-prep-command]
>>>>
>>>> 10:26:46 00:01 [jvm_prep_command]
>>>>
>>>> 10:26:46 00:01 [binary-prep-command]
>>>>
>>>> 10:26:46 00:01 [python-binary-create]
>>>>
>>>> Detected interpreter /usr/bin/python2.7:
>>>> CPython-2.7.6
>>>>
>>>> 10:26:46 00:01 [chroot]
>>>>
>>>> Building chroot for
>>>> [PythonBinary(BuildFileAddress(BuildFile(src/main/python/apache/aurora/executor/BUILD,
>>>> FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)),
>>>> thermos_executor))]:
>>>>
>>>> Dumping library:
>>>> PythonLibrary(BuildFileAddress(BuildFile(src/main/python/apache/aurora/executor/BUILD,
>>>> FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)), _executor))
>>>>
>>>> Dumping library:
>>>> PythonLibrary(BuildFileAddress(BuildFile(api/src/main/thrift/org/apache/aurora/gen/BUILD,
>>>> FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)), gen))
>>>>
>>>> Dumping library:
>>>> PythonLibrary(BuildFileAddress(BuildFile(src/main/python/apache/aurora/common/BUILD,
>>>> FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)), common))
>>>>
>>>> Dumping library:
>>>> PythonLibrary(BuildFileAddress(BuildFile(src/main/python/apache/aurora/config/BUILD,
>>>> FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)), config))
>>>>
>>>> Dumping library:
>>>> PythonLibrary(BuildFileAddress(BuildFile(src/main/python/apache/thermos/config/BUILD,
>>>> FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)), config))
>>>>
>>>> Dumping library:
>>>> PythonLibrary(BuildFileAddress(BuildFile(src/main/python/apache/thermos/common/BUILD,
>>>> FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)), common))
>>>>
>>>> Dumping library:
>>>> PythonLibrary(BuildFileAddress(BuildFile(src/main/python/apache/thermos/core/BUILD,
>>>> FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)), core))
>>>>
>>>> Dumping library:
>>>> PythonLibrary(BuildFileAddress(BuildFile(src/main/python/apache/thermos/monitoring/BUILD,
>>>> FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)), monitoring))
>>>>
>>>> Dumping library:
>>>> PythonBinary(BuildFileAddress(BuildFile(src/main/python/apache/aurora/executor/BUILD,
>>>> FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)),
>>>> thermos_executor))
>>>>
>>>> FAILURE: Update --binaries-path-by-id to find binaries for Linux armv7l
>>>> 3.10.40
>>>>
>>>>
>>>>
>>>> If I am not completely wrong, I think pants is looking one of the
>>>> architectures outlined here:
>>>>
>>>>
>>>>
>>>>
>>>> https://github.com/pantsbuild/pants/blob/master/src/python/pants/binaries/binary_util.py#L24
>>>>
>>>>
>>>>
>>>> and probably that's why it fails
>>>>
>>>>
>>>>
>>>> On 14 July 2016 at 11:17, Erb, Stephan <St...@blue-yonder.com>
>>>> wrote:
>>>>
>>>> Cool, ARM J. Please try to run it like this, so that we get a little
>>>> bit more output:
>>>>
>>>>
>>>>
>>>> PEX_VERBOSE=1 ./pants -ldebug binary
>>>> src/main/python/apache/aurora/executor:thermos_executor
>>>>
>>>>
>>>>
>>>> *From: *Yiannis Gkoufas <jo...@gmail.com>
>>>> *Reply-To: *"user@aurora.apache.org" <us...@aurora.apache.org>
>>>> *Date: *Thursday 14 July 2016 at 10:44
>>>> *To: *"user@aurora.apache.org" <us...@aurora.apache.org>
>>>> *Subject: *Build thermos executor on ARM
>>>>
>>>>
>>>>
>>>> Hi guys,
>>>>
>>>>
>>>>
>>>> I have been trying to get aurora working on ARM processors.
>>>>
>>>> I had no problem with the scheduler, but I didn't manage to compile the
>>>> thermos executor.
>>>>
>>>>
>>>>
>>>> When I give:
>>>>
>>>>
>>>>
>>>> ./pants binary src/main/python/apache/aurora/executor:thermos_executor
>>>>
>>>>
>>>>
>>>> I get back:
>>>>
>>>>
>>>>
>>>> FAILURE: Update --binaries-path-by-id to find binaries for Linux armv7l
>>>> 3.10.40
>>>>
>>>>
>>>>
>>>> Let me know if you have any hints on how to workaround this.
>>>>
>>>>
>>>>
>>>> Thanks!
>>>>
>>>>
>>>>
>>>>
>>>
>>
>
Re: Build thermos executor on ARM
Posted by Yiannis Gkoufas <jo...@gmail.com>.
Wow, thanks a lot John!
Will give it a shot and get back to you!
On 15 July 2016 at 18:05, John Sirois <jo...@gmail.com> wrote:
> Here's a workaround for you to try. If this works, I can modify the
> Aurora pants build without modifying pants itself to do all this
> automatically.
>
> The basic idea is that pants probes a list of URLs for pre-built
> binaries. You can add a local HTTP server to the list and point that
> server at the thrift binary Aurora builds for gradle via make.
> Below is how you do this:
>
> 0.) Ensure the Aurora gradle thrift is built.
> $ ./build-support/thrift/thriftw 0.9.1 --version
> Thrift version 0.9.1
>
> 1.) Just for reference, the pants options in-play here
> $ ./pants options | grep binaries
> binaries.baseurls = ['https://dl.bintray.com/pantsbuild/bin/build-support']
> (from HARDCODED)
> binaries.fetch_timeout_secs = 30 (from HARDCODED)
> binaries.path_by_id = {} (from NONE)
> $ ./pants options | grep binary
> thrift-binary.supportdir = bin/thrift (from HARDCODED)
> thrift-binary.version = 0.9.1 (from CONFIG in pants.ini)
>
> 2.) Grab machine name pants will try to find your thrift binary under
> (here is an example from my machine with result 'x86_64' - yours will be
> different and arm-ey):
> $ python2 -c 'import os; sysname, _, _, _, machine = os.uname(); print
> sysname.lower(), machine'
> linux x86_64
>
> 3.) Edit pants.ini to tell pants where to look for binaries for your
> machine:
> $ git diff -U1
> diff --git a/pants.ini b/pants.ini
> index de0ab30..e2cd94e 100644
> --- a/pants.ini
> +++ b/pants.ini
> @@ -32,2 +32,6 @@ backend_packages: [
>
> +[binaries]
> +baseurls: +['http://localhost:9999']
> +path_by_id: {('linux', 'x86_64'): ['local']}
> +
> [thrift-binary]
>
> 4.) Setup the URL you told pants about above:
> $ mkdir -p build-support/thrift/serve/bin/thrift/local/0.9.1
> $ ln -s $PWD/build-support/thrift/thrift-0.9.1/compiler/cpp/thrift
> build-support/thrift/serve/bin/thrift/local/0.9.1/thrift
> $ cd build-support/thrift/serve && (nohup python2 -m SimpleHTTPServer 9999
> >&/tmp/thrift.serve &) && cd -
>
> 5.) Run pants and confirm download of a thrift binary for your machine:
> $ ./pants binary src/main/python/::
>
> 10:58:44 00:00 [main]
> (To run a reporting server: ./pants server)
> 10:58:44 00:00 [setup]
> 10:58:44 00:00 [parse]
> Executing tasks in goals: binary
> 10:58:44 00:00 [binary]
> 10:58:44 00:00 [binary-prep-command]
> 10:58:44 00:00 [python-binary-create]
> 10:58:44 00:00 [cache]
> No cached artifacts for 8 targets.
> Invalidated 8 targets.
> 10:58:44 00:00 [chroot]INFO] Attempting to fetch thrift binary from:
> http://localhost:9999/bin/thrift/local/0.9.1/thrift ...
> 100% .................................................. 31661 KB 0.364s
> INFO] Fetched thrift binary from:
> http://localhost:9999/bin/thrift/local/0.9.1/thrift .
> ...
>
> Let me know what you get!
>
> On Thu, Jul 14, 2016 at 6:24 AM, Yiannis Gkoufas <jo...@gmail.com>
> wrote:
>
>> Cool, can you imagine a way to work around this?
>> I have to customize pants itself right?
>>
>> On 14 July 2016 at 11:54, Erb, Stephan <St...@blue-yonder.com>
>> wrote:
>>
>>> Yeah, that sounds like it could be the issue here.
>>>
>>>
>>>
>>> *From: *Yiannis Gkoufas <jo...@gmail.com>
>>> *Reply-To: *"user@aurora.apache.org" <us...@aurora.apache.org>
>>> *Date: *Thursday 14 July 2016 at 12:28
>>> *To: *"user@aurora.apache.org" <us...@aurora.apache.org>
>>> *Subject: *Re: Build thermos executor on ARM
>>>
>>>
>>>
>>> Hi there,
>>>
>>>
>>>
>>> sure:
>>>
>>>
>>>
>>> 10:26:45 00:00 [binary]
>>>
>>> 10:26:45 00:00 [binary-jvm-prep-command]
>>>
>>> 10:26:46 00:01 [jvm_prep_command]
>>>
>>> 10:26:46 00:01 [binary-prep-command]
>>>
>>> 10:26:46 00:01 [python-binary-create]
>>>
>>> Detected interpreter /usr/bin/python2.7: CPython-2.7.6
>>>
>>> 10:26:46 00:01 [chroot]
>>>
>>> Building chroot for
>>> [PythonBinary(BuildFileAddress(BuildFile(src/main/python/apache/aurora/executor/BUILD,
>>> FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)),
>>> thermos_executor))]:
>>>
>>> Dumping library:
>>> PythonLibrary(BuildFileAddress(BuildFile(src/main/python/apache/aurora/executor/BUILD,
>>> FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)), _executor))
>>>
>>> Dumping library:
>>> PythonLibrary(BuildFileAddress(BuildFile(api/src/main/thrift/org/apache/aurora/gen/BUILD,
>>> FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)), gen))
>>>
>>> Dumping library:
>>> PythonLibrary(BuildFileAddress(BuildFile(src/main/python/apache/aurora/common/BUILD,
>>> FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)), common))
>>>
>>> Dumping library:
>>> PythonLibrary(BuildFileAddress(BuildFile(src/main/python/apache/aurora/config/BUILD,
>>> FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)), config))
>>>
>>> Dumping library:
>>> PythonLibrary(BuildFileAddress(BuildFile(src/main/python/apache/thermos/config/BUILD,
>>> FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)), config))
>>>
>>> Dumping library:
>>> PythonLibrary(BuildFileAddress(BuildFile(src/main/python/apache/thermos/common/BUILD,
>>> FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)), common))
>>>
>>> Dumping library:
>>> PythonLibrary(BuildFileAddress(BuildFile(src/main/python/apache/thermos/core/BUILD,
>>> FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)), core))
>>>
>>> Dumping library:
>>> PythonLibrary(BuildFileAddress(BuildFile(src/main/python/apache/thermos/monitoring/BUILD,
>>> FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)), monitoring))
>>>
>>> Dumping library:
>>> PythonBinary(BuildFileAddress(BuildFile(src/main/python/apache/aurora/executor/BUILD,
>>> FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)),
>>> thermos_executor))
>>>
>>> FAILURE: Update --binaries-path-by-id to find binaries for Linux armv7l
>>> 3.10.40
>>>
>>>
>>>
>>> If I am not completely wrong, I think pants is looking one of the
>>> architectures outlined here:
>>>
>>>
>>>
>>>
>>> https://github.com/pantsbuild/pants/blob/master/src/python/pants/binaries/binary_util.py#L24
>>>
>>>
>>>
>>> and probably that's why it fails
>>>
>>>
>>>
>>> On 14 July 2016 at 11:17, Erb, Stephan <St...@blue-yonder.com>
>>> wrote:
>>>
>>> Cool, ARM J. Please try to run it like this, so that we get a little
>>> bit more output:
>>>
>>>
>>>
>>> PEX_VERBOSE=1 ./pants -ldebug binary
>>> src/main/python/apache/aurora/executor:thermos_executor
>>>
>>>
>>>
>>> *From: *Yiannis Gkoufas <jo...@gmail.com>
>>> *Reply-To: *"user@aurora.apache.org" <us...@aurora.apache.org>
>>> *Date: *Thursday 14 July 2016 at 10:44
>>> *To: *"user@aurora.apache.org" <us...@aurora.apache.org>
>>> *Subject: *Build thermos executor on ARM
>>>
>>>
>>>
>>> Hi guys,
>>>
>>>
>>>
>>> I have been trying to get aurora working on ARM processors.
>>>
>>> I had no problem with the scheduler, but I didn't manage to compile the
>>> thermos executor.
>>>
>>>
>>>
>>> When I give:
>>>
>>>
>>>
>>> ./pants binary src/main/python/apache/aurora/executor:thermos_executor
>>>
>>>
>>>
>>> I get back:
>>>
>>>
>>>
>>> FAILURE: Update --binaries-path-by-id to find binaries for Linux armv7l
>>> 3.10.40
>>>
>>>
>>>
>>> Let me know if you have any hints on how to workaround this.
>>>
>>>
>>>
>>> Thanks!
>>>
>>>
>>>
>>>
>>
>
Re: Build thermos executor on ARM
Posted by John Sirois <jo...@gmail.com>.
Here's a workaround for you to try. If this works, I can modify the Aurora
pants build without modifying pants itself to do all this automatically.
The basic idea is that pants probes a list of URLs for pre-built binaries.
You can add a local HTTP server to the list and point that server at the
thrift binary Aurora builds for gradle via make.
Below is how you do this:
0.) Ensure the Aurora gradle thrift is built.
$ ./build-support/thrift/thriftw 0.9.1 --version
Thrift version 0.9.1
1.) Just for reference, the pants options in-play here
$ ./pants options | grep binaries
binaries.baseurls = ['https://dl.bintray.com/pantsbuild/bin/build-support']
(from HARDCODED)
binaries.fetch_timeout_secs = 30 (from HARDCODED)
binaries.path_by_id = {} (from NONE)
$ ./pants options | grep binary
thrift-binary.supportdir = bin/thrift (from HARDCODED)
thrift-binary.version = 0.9.1 (from CONFIG in pants.ini)
2.) Grab machine name pants will try to find your thrift binary under (here
is an example from my machine with result 'x86_64' - yours will be
different and arm-ey):
$ python2 -c 'import os; sysname, _, _, _, machine = os.uname(); print
sysname.lower(), machine'
linux x86_64
3.) Edit pants.ini to tell pants where to look for binaries for your
machine:
$ git diff -U1
diff --git a/pants.ini b/pants.ini
index de0ab30..e2cd94e 100644
--- a/pants.ini
+++ b/pants.ini
@@ -32,2 +32,6 @@ backend_packages: [
+[binaries]
+baseurls: +['http://localhost:9999']
+path_by_id: {('linux', 'x86_64'): ['local']}
+
[thrift-binary]
4.) Setup the URL you told pants about above:
$ mkdir -p build-support/thrift/serve/bin/thrift/local/0.9.1
$ ln -s $PWD/build-support/thrift/thrift-0.9.1/compiler/cpp/thrift
build-support/thrift/serve/bin/thrift/local/0.9.1/thrift
$ cd build-support/thrift/serve && (nohup python2 -m SimpleHTTPServer 9999
>&/tmp/thrift.serve &) && cd -
5.) Run pants and confirm download of a thrift binary for your machine:
$ ./pants binary src/main/python/::
10:58:44 00:00 [main]
(To run a reporting server: ./pants server)
10:58:44 00:00 [setup]
10:58:44 00:00 [parse]
Executing tasks in goals: binary
10:58:44 00:00 [binary]
10:58:44 00:00 [binary-prep-command]
10:58:44 00:00 [python-binary-create]
10:58:44 00:00 [cache]
No cached artifacts for 8 targets.
Invalidated 8 targets.
10:58:44 00:00 [chroot]INFO] Attempting to fetch thrift binary from:
http://localhost:9999/bin/thrift/local/0.9.1/thrift ...
100% .................................................. 31661 KB 0.364s
INFO] Fetched thrift binary from:
http://localhost:9999/bin/thrift/local/0.9.1/thrift .
...
Let me know what you get!
On Thu, Jul 14, 2016 at 6:24 AM, Yiannis Gkoufas <jo...@gmail.com>
wrote:
> Cool, can you imagine a way to work around this?
> I have to customize pants itself right?
>
> On 14 July 2016 at 11:54, Erb, Stephan <St...@blue-yonder.com>
> wrote:
>
>> Yeah, that sounds like it could be the issue here.
>>
>>
>>
>> *From: *Yiannis Gkoufas <jo...@gmail.com>
>> *Reply-To: *"user@aurora.apache.org" <us...@aurora.apache.org>
>> *Date: *Thursday 14 July 2016 at 12:28
>> *To: *"user@aurora.apache.org" <us...@aurora.apache.org>
>> *Subject: *Re: Build thermos executor on ARM
>>
>>
>>
>> Hi there,
>>
>>
>>
>> sure:
>>
>>
>>
>> 10:26:45 00:00 [binary]
>>
>> 10:26:45 00:00 [binary-jvm-prep-command]
>>
>> 10:26:46 00:01 [jvm_prep_command]
>>
>> 10:26:46 00:01 [binary-prep-command]
>>
>> 10:26:46 00:01 [python-binary-create]
>>
>> Detected interpreter /usr/bin/python2.7: CPython-2.7.6
>>
>> 10:26:46 00:01 [chroot]
>>
>> Building chroot for
>> [PythonBinary(BuildFileAddress(BuildFile(src/main/python/apache/aurora/executor/BUILD,
>> FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)),
>> thermos_executor))]:
>>
>> Dumping library:
>> PythonLibrary(BuildFileAddress(BuildFile(src/main/python/apache/aurora/executor/BUILD,
>> FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)), _executor))
>>
>> Dumping library:
>> PythonLibrary(BuildFileAddress(BuildFile(api/src/main/thrift/org/apache/aurora/gen/BUILD,
>> FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)), gen))
>>
>> Dumping library:
>> PythonLibrary(BuildFileAddress(BuildFile(src/main/python/apache/aurora/common/BUILD,
>> FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)), common))
>>
>> Dumping library:
>> PythonLibrary(BuildFileAddress(BuildFile(src/main/python/apache/aurora/config/BUILD,
>> FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)), config))
>>
>> Dumping library:
>> PythonLibrary(BuildFileAddress(BuildFile(src/main/python/apache/thermos/config/BUILD,
>> FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)), config))
>>
>> Dumping library:
>> PythonLibrary(BuildFileAddress(BuildFile(src/main/python/apache/thermos/common/BUILD,
>> FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)), common))
>>
>> Dumping library:
>> PythonLibrary(BuildFileAddress(BuildFile(src/main/python/apache/thermos/core/BUILD,
>> FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)), core))
>>
>> Dumping library:
>> PythonLibrary(BuildFileAddress(BuildFile(src/main/python/apache/thermos/monitoring/BUILD,
>> FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)), monitoring))
>>
>> Dumping library:
>> PythonBinary(BuildFileAddress(BuildFile(src/main/python/apache/aurora/executor/BUILD,
>> FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)),
>> thermos_executor))
>>
>> FAILURE: Update --binaries-path-by-id to find binaries for Linux armv7l
>> 3.10.40
>>
>>
>>
>> If I am not completely wrong, I think pants is looking one of the
>> architectures outlined here:
>>
>>
>>
>>
>> https://github.com/pantsbuild/pants/blob/master/src/python/pants/binaries/binary_util.py#L24
>>
>>
>>
>> and probably that's why it fails
>>
>>
>>
>> On 14 July 2016 at 11:17, Erb, Stephan <St...@blue-yonder.com>
>> wrote:
>>
>> Cool, ARM J. Please try to run it like this, so that we get a little bit
>> more output:
>>
>>
>>
>> PEX_VERBOSE=1 ./pants -ldebug binary
>> src/main/python/apache/aurora/executor:thermos_executor
>>
>>
>>
>> *From: *Yiannis Gkoufas <jo...@gmail.com>
>> *Reply-To: *"user@aurora.apache.org" <us...@aurora.apache.org>
>> *Date: *Thursday 14 July 2016 at 10:44
>> *To: *"user@aurora.apache.org" <us...@aurora.apache.org>
>> *Subject: *Build thermos executor on ARM
>>
>>
>>
>> Hi guys,
>>
>>
>>
>> I have been trying to get aurora working on ARM processors.
>>
>> I had no problem with the scheduler, but I didn't manage to compile the
>> thermos executor.
>>
>>
>>
>> When I give:
>>
>>
>>
>> ./pants binary src/main/python/apache/aurora/executor:thermos_executor
>>
>>
>>
>> I get back:
>>
>>
>>
>> FAILURE: Update --binaries-path-by-id to find binaries for Linux armv7l
>> 3.10.40
>>
>>
>>
>> Let me know if you have any hints on how to workaround this.
>>
>>
>>
>> Thanks!
>>
>>
>>
>>
>
Re: Build thermos executor on ARM
Posted by Yiannis Gkoufas <jo...@gmail.com>.
Cool, can you imagine a way to work around this?
I have to customize pants itself right?
On 14 July 2016 at 11:54, Erb, Stephan <St...@blue-yonder.com> wrote:
> Yeah, that sounds like it could be the issue here.
>
>
>
> *From: *Yiannis Gkoufas <jo...@gmail.com>
> *Reply-To: *"user@aurora.apache.org" <us...@aurora.apache.org>
> *Date: *Thursday 14 July 2016 at 12:28
> *To: *"user@aurora.apache.org" <us...@aurora.apache.org>
> *Subject: *Re: Build thermos executor on ARM
>
>
>
> Hi there,
>
>
>
> sure:
>
>
>
> 10:26:45 00:00 [binary]
>
> 10:26:45 00:00 [binary-jvm-prep-command]
>
> 10:26:46 00:01 [jvm_prep_command]
>
> 10:26:46 00:01 [binary-prep-command]
>
> 10:26:46 00:01 [python-binary-create]
>
> Detected interpreter /usr/bin/python2.7: CPython-2.7.6
>
> 10:26:46 00:01 [chroot]
>
> Building chroot for
> [PythonBinary(BuildFileAddress(BuildFile(src/main/python/apache/aurora/executor/BUILD,
> FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)),
> thermos_executor))]:
>
> Dumping library:
> PythonLibrary(BuildFileAddress(BuildFile(src/main/python/apache/aurora/executor/BUILD,
> FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)), _executor))
>
> Dumping library:
> PythonLibrary(BuildFileAddress(BuildFile(api/src/main/thrift/org/apache/aurora/gen/BUILD,
> FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)), gen))
>
> Dumping library:
> PythonLibrary(BuildFileAddress(BuildFile(src/main/python/apache/aurora/common/BUILD,
> FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)), common))
>
> Dumping library:
> PythonLibrary(BuildFileAddress(BuildFile(src/main/python/apache/aurora/config/BUILD,
> FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)), config))
>
> Dumping library:
> PythonLibrary(BuildFileAddress(BuildFile(src/main/python/apache/thermos/config/BUILD,
> FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)), config))
>
> Dumping library:
> PythonLibrary(BuildFileAddress(BuildFile(src/main/python/apache/thermos/common/BUILD,
> FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)), common))
>
> Dumping library:
> PythonLibrary(BuildFileAddress(BuildFile(src/main/python/apache/thermos/core/BUILD,
> FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)), core))
>
> Dumping library:
> PythonLibrary(BuildFileAddress(BuildFile(src/main/python/apache/thermos/monitoring/BUILD,
> FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)), monitoring))
>
> Dumping library:
> PythonBinary(BuildFileAddress(BuildFile(src/main/python/apache/aurora/executor/BUILD,
> FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)),
> thermos_executor))
>
> FAILURE: Update --binaries-path-by-id to find binaries for Linux armv7l
> 3.10.40
>
>
>
> If I am not completely wrong, I think pants is looking one of the
> architectures outlined here:
>
>
>
>
> https://github.com/pantsbuild/pants/blob/master/src/python/pants/binaries/binary_util.py#L24
>
>
>
> and probably that's why it fails
>
>
>
> On 14 July 2016 at 11:17, Erb, Stephan <St...@blue-yonder.com>
> wrote:
>
> Cool, ARM J. Please try to run it like this, so that we get a little bit
> more output:
>
>
>
> PEX_VERBOSE=1 ./pants -ldebug binary
> src/main/python/apache/aurora/executor:thermos_executor
>
>
>
> *From: *Yiannis Gkoufas <jo...@gmail.com>
> *Reply-To: *"user@aurora.apache.org" <us...@aurora.apache.org>
> *Date: *Thursday 14 July 2016 at 10:44
> *To: *"user@aurora.apache.org" <us...@aurora.apache.org>
> *Subject: *Build thermos executor on ARM
>
>
>
> Hi guys,
>
>
>
> I have been trying to get aurora working on ARM processors.
>
> I had no problem with the scheduler, but I didn't manage to compile the
> thermos executor.
>
>
>
> When I give:
>
>
>
> ./pants binary src/main/python/apache/aurora/executor:thermos_executor
>
>
>
> I get back:
>
>
>
> FAILURE: Update --binaries-path-by-id to find binaries for Linux armv7l
> 3.10.40
>
>
>
> Let me know if you have any hints on how to workaround this.
>
>
>
> Thanks!
>
>
>
>
Re: Build thermos executor on ARM
Posted by "Erb, Stephan" <St...@blue-yonder.com>.
Yeah, that sounds like it could be the issue here.
From: Yiannis Gkoufas <jo...@gmail.com>
Reply-To: "user@aurora.apache.org" <us...@aurora.apache.org>
Date: Thursday 14 July 2016 at 12:28
To: "user@aurora.apache.org" <us...@aurora.apache.org>
Subject: Re: Build thermos executor on ARM
Hi there,
sure:
10:26:45 00:00 [binary]
10:26:45 00:00 [binary-jvm-prep-command]
10:26:46 00:01 [jvm_prep_command]
10:26:46 00:01 [binary-prep-command]
10:26:46 00:01 [python-binary-create]
Detected interpreter /usr/bin/python2.7: CPython-2.7.6
10:26:46 00:01 [chroot]
Building chroot for [PythonBinary(BuildFileAddress(BuildFile(src/main/python/apache/aurora/executor/BUILD, FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)), thermos_executor))]:
Dumping library: PythonLibrary(BuildFileAddress(BuildFile(src/main/python/apache/aurora/executor/BUILD, FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)), _executor))
Dumping library: PythonLibrary(BuildFileAddress(BuildFile(api/src/main/thrift/org/apache/aurora/gen/BUILD, FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)), gen))
Dumping library: PythonLibrary(BuildFileAddress(BuildFile(src/main/python/apache/aurora/common/BUILD, FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)), common))
Dumping library: PythonLibrary(BuildFileAddress(BuildFile(src/main/python/apache/aurora/config/BUILD, FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)), config))
Dumping library: PythonLibrary(BuildFileAddress(BuildFile(src/main/python/apache/thermos/config/BUILD, FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)), config))
Dumping library: PythonLibrary(BuildFileAddress(BuildFile(src/main/python/apache/thermos/common/BUILD, FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)), common))
Dumping library: PythonLibrary(BuildFileAddress(BuildFile(src/main/python/apache/thermos/core/BUILD, FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)), core))
Dumping library: PythonLibrary(BuildFileAddress(BuildFile(src/main/python/apache/thermos/monitoring/BUILD, FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)), monitoring))
Dumping library: PythonBinary(BuildFileAddress(BuildFile(src/main/python/apache/aurora/executor/BUILD, FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)), thermos_executor))
FAILURE: Update --binaries-path-by-id to find binaries for Linux armv7l 3.10.40
If I am not completely wrong, I think pants is looking one of the architectures outlined here:
https://github.com/pantsbuild/pants/blob/master/src/python/pants/binaries/binary_util.py#L24
and probably that's why it fails
On 14 July 2016 at 11:17, Erb, Stephan <St...@blue-yonder.com>> wrote:
Cool, ARM ☺. Please try to run it like this, so that we get a little bit more output:
PEX_VERBOSE=1 ./pants -ldebug binary src/main/python/apache/aurora/executor:thermos_executor
From: Yiannis Gkoufas <jo...@gmail.com>>
Reply-To: "user@aurora.apache.org<ma...@aurora.apache.org>" <us...@aurora.apache.org>>
Date: Thursday 14 July 2016 at 10:44
To: "user@aurora.apache.org<ma...@aurora.apache.org>" <us...@aurora.apache.org>>
Subject: Build thermos executor on ARM
Hi guys,
I have been trying to get aurora working on ARM processors.
I had no problem with the scheduler, but I didn't manage to compile the thermos executor.
When I give:
./pants binary src/main/python/apache/aurora/executor:thermos_executor
I get back:
FAILURE: Update --binaries-path-by-id to find binaries for Linux armv7l 3.10.40
Let me know if you have any hints on how to workaround this.
Thanks!
Re: Build thermos executor on ARM
Posted by Yiannis Gkoufas <jo...@gmail.com>.
Hi there,
sure:
10:26:45 00:00 [binary]
10:26:45 00:00 [binary-jvm-prep-command]
10:26:46 00:01 [jvm_prep_command]
10:26:46 00:01 [binary-prep-command]
10:26:46 00:01 [python-binary-create]
Detected interpreter /usr/bin/python2.7: CPython-2.7.6
10:26:46 00:01 [chroot]
Building chroot for
[PythonBinary(BuildFileAddress(BuildFile(src/main/python/apache/aurora/executor/BUILD,
FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)),
thermos_executor))]:
Dumping library:
PythonLibrary(BuildFileAddress(BuildFile(src/main/python/apache/aurora/executor/BUILD,
FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)), _executor))
Dumping library:
PythonLibrary(BuildFileAddress(BuildFile(api/src/main/thrift/org/apache/aurora/gen/BUILD,
FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)), gen))
Dumping library:
PythonLibrary(BuildFileAddress(BuildFile(src/main/python/apache/aurora/common/BUILD,
FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)), common))
Dumping library:
PythonLibrary(BuildFileAddress(BuildFile(src/main/python/apache/aurora/config/BUILD,
FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)), config))
Dumping library:
PythonLibrary(BuildFileAddress(BuildFile(src/main/python/apache/thermos/config/BUILD,
FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)), config))
Dumping library:
PythonLibrary(BuildFileAddress(BuildFile(src/main/python/apache/thermos/common/BUILD,
FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)), common))
Dumping library:
PythonLibrary(BuildFileAddress(BuildFile(src/main/python/apache/thermos/core/BUILD,
FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)), core))
Dumping library:
PythonLibrary(BuildFileAddress(BuildFile(src/main/python/apache/thermos/monitoring/BUILD,
FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)), monitoring))
Dumping library:
PythonBinary(BuildFileAddress(BuildFile(src/main/python/apache/aurora/executor/BUILD,
FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)),
thermos_executor))
FAILURE: Update --binaries-path-by-id to find binaries for Linux armv7l
3.10.40
If I am not completely wrong, I think pants is looking one of the
architectures outlined here:
https://github.com/pantsbuild/pants/blob/master/src/python/pants/binaries/binary_util.py#L24
and probably that's why it fails
On 14 July 2016 at 11:17, Erb, Stephan <St...@blue-yonder.com> wrote:
> Cool, ARM J. Please try to run it like this, so that we get a little bit
> more output:
>
>
>
> PEX_VERBOSE=1 ./pants -ldebug binary
> src/main/python/apache/aurora/executor:thermos_executor
>
>
>
> *From: *Yiannis Gkoufas <jo...@gmail.com>
> *Reply-To: *"user@aurora.apache.org" <us...@aurora.apache.org>
> *Date: *Thursday 14 July 2016 at 10:44
> *To: *"user@aurora.apache.org" <us...@aurora.apache.org>
> *Subject: *Build thermos executor on ARM
>
>
>
> Hi guys,
>
>
>
> I have been trying to get aurora working on ARM processors.
>
> I had no problem with the scheduler, but I didn't manage to compile the
> thermos executor.
>
>
>
> When I give:
>
>
>
> ./pants binary src/main/python/apache/aurora/executor:thermos_executor
>
>
>
> I get back:
>
>
>
> FAILURE: Update --binaries-path-by-id to find binaries for Linux armv7l
> 3.10.40
>
>
>
> Let me know if you have any hints on how to workaround this.
>
>
>
> Thanks!
>
>
Re: Build thermos executor on ARM
Posted by "Erb, Stephan" <St...@blue-yonder.com>.
Cool, ARM ☺. Please try to run it like this, so that we get a little bit more output:
PEX_VERBOSE=1 ./pants -ldebug binary src/main/python/apache/aurora/executor:thermos_executor
From: Yiannis Gkoufas <jo...@gmail.com>
Reply-To: "user@aurora.apache.org" <us...@aurora.apache.org>
Date: Thursday 14 July 2016 at 10:44
To: "user@aurora.apache.org" <us...@aurora.apache.org>
Subject: Build thermos executor on ARM
Hi guys,
I have been trying to get aurora working on ARM processors.
I had no problem with the scheduler, but I didn't manage to compile the thermos executor.
When I give:
./pants binary src/main/python/apache/aurora/executor:thermos_executor
I get back:
FAILURE: Update --binaries-path-by-id to find binaries for Linux armv7l 3.10.40
Let me know if you have any hints on how to workaround this.
Thanks!