You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@qpid.apache.org by Alan Conway <ac...@redhat.com> on 2010/03/16 19:00:33 UTC
Why do we create the cpp/src/tests/python dir?
The C++ builds install python files in src/tests/python. This is kind of a pain
for at least 2 reasons:
- src/tests/python can go stale - not sure how but it just happened to me.
- debugging python code, error messages take you to the src/tests/python copy
rather than the real source file.
- python gets re-installed every time you build.
Whats the reason for doing this? Can we get the same effect some other way that
doesn't involve duplicating all the .py files?
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org
Re: Why do we create the cpp/src/tests/python dir?
Posted by Andrew Stitcher <as...@redhat.com>.
On Fri, 2010-03-19 at 07:59 -0400, Rafael Schloming wrote:
> As for why they get re-installed every time and how they can go stale,
> I'm not sure about that. I believe andrew was the one who modified the
> cpp build to do this.
I plead not guilty m'lud.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org
Re: Why do we create the cpp/src/tests/python dir?
Posted by Rafael Schloming <ra...@redhat.com>.
Gordon Sim wrote:
> On 03/19/2010 02:23 PM, Alan Conway wrote:
>> I think the clean way out is to make the python project a dependency of
>> the C++ project just like it depends on boost or other tools. I.e. to
>> run the tests you must *install* the python project, and the tests pick
>> up components via standard PATH and PYTHONPATH env vars.
>
> Assuming I understand this correctly, we would have to reinstall every
> time we needed to pick up any change in the python client or tools.
> Unlike with boost or other tools, that's likely to occur quite frequently.
>
> I'm not arguing against your suggestion however, just making an
> observation on the impact as I see it.
There is also a dependency on tests, which would probably be more
problematic since it's not uncommon to fix/add something to the broker
and also change these tests in a single commit.
--Rafael
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org
Re: Why do we create the cpp/src/tests/python dir?
Posted by Gordon Sim <gs...@redhat.com>.
On 03/19/2010 02:23 PM, Alan Conway wrote:
> I think the clean way out is to make the python project a dependency of
> the C++ project just like it depends on boost or other tools. I.e. to
> run the tests you must *install* the python project, and the tests pick
> up components via standard PATH and PYTHONPATH env vars.
Assuming I understand this correctly, we would have to reinstall every
time we needed to pick up any change in the python client or tools.
Unlike with boost or other tools, that's likely to occur quite frequently.
I'm not arguing against your suggestion however, just making an
observation on the impact as I see it.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org
Re: Why do we create the cpp/src/tests/python dir?
Posted by Alan Conway <ac...@redhat.com>.
On 03/19/2010 07:59 AM, Rafael Schloming wrote:
> Alan Conway wrote:
>> The C++ builds install python files in src/tests/python. This is kind
>> of a pain for at least 2 reasons:
>> - src/tests/python can go stale - not sure how but it just happened to
>> me.
>> - debugging python code, error messages take you to the
>> src/tests/python copy rather than the real source file.
>> - python gets re-installed every time you build.
>>
>> Whats the reason for doing this? Can we get the same effect some other
>> way that doesn't involve duplicating all the .py files?
>
> The root cause for this is that the python client needs to support
> python 2.3, and to do this it needs to do some preprocessing on the
> files before you can run them (assuming you're on a system with python
> 2.3). So it's not actually copying the files it's actually "compiling"
> them, and the reason this is done under the cpp build root is simply to
> not pollute the source tree.
>
> As for why they get re-installed every time and how they can go stale,
> I'm not sure about that. I believe andrew was the one who modified the
> cpp build to do this. I'm guessing it could be done in a smarter way
> that wouldn't rebuild and wouldn't allow things to go stale.
>
I think this is a good case study to think about how we do dependencies between
sub-projects in qpid.
The C++ build currently has dependencies on (at least)
- python
- tools
- extras
And it picks them up via a variety of routes. In an svn checkout it has a bunch
of coded-in assumptions about looking for things in ../python, it's got hackery
to run the python install process mentioned above and it's all quite nasty. In a
distribution tree it picks up qpid-tool and the like from PATH if they're
installed, quietly skips the python tests if it can't find the ../python tree,
randomly fails if there happens to be a ../python directory that's not what the
build expected etc.
I think the clean way out is to make the python project a dependency of the C++
project just like it depends on boost or other tools. I.e. to run the tests you
must *install* the python project, and the tests pick up components via standard
PATH and PYTHONPATH env vars. A developer would set their install --prefix to
their own directory rather than the system one and install the dependencies
there. A user building from a distro could do this or install in standard system
places as they wish.
To keep things simple for developers we would have a script that install all the
C++ deps for you in an SVN checkout.
This idea is in the half-baked stage so please point to holes & suggest
improvements. There are a couple drawbacks I can think of:
- extra step for developers to build, although could consider this part of
installing pre-reqs so not really a new step.
- risk of picking up stuff installed in standard places by accident, getting
wrong version. Don't know how to avoid this.
- not sure how this plays in the windows world.
Thoughts?
Alan.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org
Re: Why do we create the cpp/src/tests/python dir?
Posted by Rafael Schloming <ra...@redhat.com>.
Alan Conway wrote:
> The C++ builds install python files in src/tests/python. This is kind of
> a pain for at least 2 reasons:
> - src/tests/python can go stale - not sure how but it just happened to me.
> - debugging python code, error messages take you to the src/tests/python
> copy rather than the real source file.
> - python gets re-installed every time you build.
>
> Whats the reason for doing this? Can we get the same effect some other
> way that doesn't involve duplicating all the .py files?
The root cause for this is that the python client needs to support
python 2.3, and to do this it needs to do some preprocessing on the
files before you can run them (assuming you're on a system with python
2.3). So it's not actually copying the files it's actually "compiling"
them, and the reason this is done under the cpp build root is simply to
not pollute the source tree.
As for why they get re-installed every time and how they can go stale,
I'm not sure about that. I believe andrew was the one who modified the
cpp build to do this. I'm guessing it could be done in a smarter way
that wouldn't rebuild and wouldn't allow things to go stale.
--Rafael
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org