You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cloudstack.apache.org by David Nalley <da...@gnsa.us> on 2012/07/04 22:20:35 UTC
[DISCUSS] tools directory
Hi folks,
I spent some time poking around in the tools directory.
There are a number of software packages in this directory, and I am
not sure that it is appropriate to have them there, and specifically
some of them are prohibited items.
I am proposing to delete the following directories:
ant
gcc
junit
migration
mockito
tooljars
vhd-tools
If this offends you, please let us know why it must stay.
I suppose the remaining can/should stay, but if you know a reason why
not, please let us know that too.
--David
RE: [DISCUSS] tools directory
Posted by Frank Zhang <Fr...@citrix.com>.
> > Migration: this is/was a tool to assist 1.0.x -> 2.1.x migration. Can
> > be nuked
>
> No no。 We can't nuke this one. There are still 1.0.x deployments out there.
> This is their only way to upgrade.
We can maintain it in somewhere but not ASF repo. It uses some python library against ASF license.
>
> > ant: sure -- your dev OS should have it, unless hudson/jenkins relies
> > on this folder
> > gcc: this seems to be used by the UI javascript?
> > Junit/mockito: both libraries are used for unit tests, not sure this
> > is safe to nuke
> Mockito is not used yet so it can be nuked.
>
> > Tooljars: not sure if folks are using selenium. Haven't seen any test
> > cases driven by selenium
> >
>
> Tools directory is intended for tools that assist in building and testing. None
> of them should be deployable on a production system. We need to move
> everything that are deployable such as migration and vhdutil to a more
> appropriate directory.
>
> --Alex
RE: [DISCUSS] tools directory
Posted by Alex Huang <Al...@citrix.com>.
Yeah. As long as it's not removed from older branches, it should be ok.
--Alex
> -----Original Message-----
> From: David Nalley [mailto:david@gnsa.us]
> Sent: Thursday, July 05, 2012 3:20 PM
> To: cloudstack-dev@incubator.apache.org
> Subject: Re: [DISCUSS] tools directory
>
> On Thu, Jul 5, 2012 at 5:14 PM, Alex Huang <Al...@citrix.com> wrote:
> >> Migration: this is/was a tool to assist 1.0.x -> 2.1.x migration. Can
> >> be nuked
> >
> > No no。 We can't nuke this one. There are still 1.0.x deployments out
> there. This is their only way to upgrade.
> >
>
> But wouldn't they have to move to 2.x first? which should have this in place
> I'd hope?
Re: [DISCUSS] tools directory
Posted by David Nalley <da...@gnsa.us>.
On Thu, Jul 5, 2012 at 5:14 PM, Alex Huang <Al...@citrix.com> wrote:
>> Migration: this is/was a tool to assist 1.0.x -> 2.1.x migration. Can be nuked
>
> No no。 We can't nuke this one. There are still 1.0.x deployments out there. This is their only way to upgrade.
>
But wouldn't they have to move to 2.x first? which should have this in
place I'd hope?
RE: [DISCUSS] tools directory
Posted by Alex Huang <Al...@citrix.com>.
> Migration: this is/was a tool to assist 1.0.x -> 2.1.x migration. Can be nuked
No no。 We can't nuke this one. There are still 1.0.x deployments out there. This is their only way to upgrade.
> ant: sure -- your dev OS should have it, unless hudson/jenkins relies on this
> folder
> gcc: this seems to be used by the UI javascript?
> Junit/mockito: both libraries are used for unit tests, not sure this is safe to
> nuke
Mockito is not used yet so it can be nuked.
> Tooljars: not sure if folks are using selenium. Haven't seen any test cases
> driven by selenium
>
Tools directory is intended for tools that assist in building and testing. None of them should be deployable on a production system. We need to move everything that are deployable such as migration and vhdutil to a more appropriate directory.
--Alex
RE: [DISCUSS] tools directory
Posted by Frank Zhang <Fr...@citrix.com>.
>
> On 7/4/12 10:12 PM, "Prasanna Santhanam"
> <Pr...@citrix.com>
> wrote:
>
> >On Wed, Jul 04, 2012 at 04:20:35PM -0400, David Nalley wrote:
> >> Hi folks,
> >>
> >> I spent some time poking around in the tools directory.
> >>
> >> There are a number of software packages in this directory, and I am
> >> not sure that it is appropriate to have them there, and specifically
> >> some of them are prohibited items.
> >>
> >> I am proposing to delete the following directories:
> >>
> >> ant
> >> gcc
> >> junit
> >> migration
> >> mockito
> >> tooljars
> >> vhd-tools
> >>
> >> If this offends you, please let us know why it must stay.
> >>
> >> I suppose the remaining can/should stay, but if you know a reason why
> >> not, please let us know that too.
> >>
> >> --David
> >
> >The copyright notice script in tools/build/copyright.py can be hacked
> >to have the license headers for ASF be automatically added. Anyone
> >already have any plan for this? Will this be useful David?
>
> Migration: this is/was a tool to assist 1.0.x -> 2.1.x migration. Can be nuked
> ant: sure -- your dev OS should have it, unless hudson/jenkins relies on this
> folder
I suggest keeping ant. Our ant build uses a lot of ant plugins which are not installed with Ant itself by default.
and once you face a build error because of missing plugin, it costs you much time to figure out which RPM contains it as
the plugin name and RPM package name usually vary a lot.
This ant in tool/ is self-contained.
> gcc: this seems to be used by the UI javascript?
> Junit/mockito: both libraries are used for unit tests, not sure this is safe to
> nuke
> Tooljars: not sure if folks are using selenium. Haven't seen any test cases
> driven by selenium
>
Re: [DISCUSS] tools directory
Posted by Robert Schweikert <rj...@suse.com>.
On 07/06/2012 01:31 PM, David Nalley wrote:
> On Fri, Jul 6, 2012 at 1:07 PM, Robert Schweikert <rj...@suse.com> wrote:
>> On 07/06/2012 12:54 PM, Kevin Kluge wrote:
>>>
>>> I understood GCC to have a AL 2.0 license and to be used by the UI, as
>>> described in [1]. So it should at least be acceptable per license. I
>>> don't see why it has to be kept in the source tree but a removal of it
>>> should also include build changes to download a known-stable version for use
>>> in the build,
>>
>>
>> Downloading things during build will not work in all cases, thus a "download
>> something to build the stack" is not a good option.
>>
>> Things in the source tree should be either in source format or describable
>> as an dependency. If a "download it automatically" feature is needed for
>> developer convenience then it should be an explicit feature that can be
>> enabled.
>>
>> I am thinking here primarily of package build systems such as OBS (Open
>> Build Service) that do not provide network access when packages are being
>> built.
>
> If it's a dependency, can we just move it to the deps directory?
I would prefer not to have a deps directory. From my point of view
anything the code (source) depends on falls into two buckets.
1.) Functionality provided in another source file that is part of the
code base.
2.) Functionality provided by some code external to the project.
Handling of stuff that falls into bucket 1 is pretty straight forward as
it is under the control of the project, although even here dependencies
can be tricky to manage. Everything else that falls into bucket 2 is
external to the project and should be treated as such, i.e. keep it out
of the tree and document the dependency. The documented dependencies
fall into 2 categories:
a.) build dependencies; such as ant and other tools
b.) run dependencies (a runtime dependency may also be a build
time dependency); such as libvirt-java
From a distribution perspective everything that falls into the
dependency category can and should be provided as a package. This in the
end is the packagers responsibility and not the responsibility of the
project. There's another angle to this for developers I agree, more on
this below.
Part of my issue as new-comer to the project and trying to get decent
quality packages for SUSE is that things are mingled together and there
are layered build systems, waf ontop of ant and maybe other stuff. While
the build works, thanks to the work done for fedora by David, I am not
really happy with this I cannot wrap my head around how things are put
together. It would be nice to get to a point where things are clearly
delineated and we can have a README file that clearly spells out what
the dependencies are and how to build the various components Alex is
working on, (avoiding the overloaded word "package" as in the distro
context int means something different).
>
> Fedora's build systems are the same way.
> Moreover, while we can have a system that downloads prohibited
> dependencies, it must be a non-default option.
> http://www.apache.org/legal/3party.html#options
>
This is where I see the "convenience function" for developers come into
play. For those that just what to hack on the code base and do not care
much about how things are built etc. it is nice to have an "automated"
way to pull in all these dependencies, ant setup-buildenv, for example
might be a good target that could provide all these dependencies.
However, with the dependencies out of the tree and explicitly stated it
is also easy for packages to create -devel packages that let CloudStack
developers install all the devel dependencies and still have them
managed by their distro's package manager.
Yes, the implication of this is that all .jar files that are currently
in the repository go away.
-> find . -name "*.jar" | wc
162 162 6604
ouch.....
And again there are 2 options:
- the .jar file is superfluous in which case it is simple to remove
- the dependency is real and falls into a.) or b.) above, this implies
~ we have to document the dependency
~ get rid of the jar file
~ let developers know they are responsible for managing their
build system accordingly.
I am not certain how this plays into the setup of the continuous
integration server, but that should be manageable in a reasonable way.
Robert
--
Robert Schweikert MAY THE SOURCE BE WITH YOU
SUSE-IBM Software Integration Center LINUX
Tech Lead
rjschwei@suse.com
rschweik@ca.ibm.com
781-464-8147
Re: [DISCUSS] tools directory
Posted by David Nalley <da...@gnsa.us>.
On Fri, Jul 6, 2012 at 1:07 PM, Robert Schweikert <rj...@suse.com> wrote:
> On 07/06/2012 12:54 PM, Kevin Kluge wrote:
>>
>> I understood GCC to have a AL 2.0 license and to be used by the UI, as
>> described in [1]. So it should at least be acceptable per license. I
>> don't see why it has to be kept in the source tree but a removal of it
>> should also include build changes to download a known-stable version for use
>> in the build,
>
>
> Downloading things during build will not work in all cases, thus a "download
> something to build the stack" is not a good option.
>
> Things in the source tree should be either in source format or describable
> as an dependency. If a "download it automatically" feature is needed for
> developer convenience then it should be an explicit feature that can be
> enabled.
>
> I am thinking here primarily of package build systems such as OBS (Open
> Build Service) that do not provide network access when packages are being
> built.
If it's a dependency, can we just move it to the deps directory?
Fedora's build systems are the same way.
Moreover, while we can have a system that downloads prohibited
dependencies, it must be a non-default option.
http://www.apache.org/legal/3party.html#options
Re: [DISCUSS] tools directory
Posted by Robert Schweikert <rj...@suse.com>.
On 07/06/2012 12:54 PM, Kevin Kluge wrote:
> I understood GCC to have a AL 2.0 license and to be used by the UI, as described in [1]. So it should at least be acceptable per license. I don't see why it has to be kept in the source tree but a removal of it should also include build changes to download a known-stable version for use in the build,
Downloading things during build will not work in all cases, thus a
"download something to build the stack" is not a good option.
Things in the source tree should be either in source format or
describable as an dependency. If a "download it automatically" feature
is needed for developer convenience then it should be an explicit
feature that can be enabled.
I am thinking here primarily of package build systems such as OBS (Open
Build Service) that do not provide network access when packages are
being built.
Regards,
Robert
--
Robert Schweikert MAY THE SOURCE BE WITH YOU
SUSE-IBM Software Integration Center LINUX
Tech Lead
rjschwei@suse.com
rschweik@ca.ibm.com
781-464-8147
RE: [DISCUSS] tools directory
Posted by Kevin Kluge <Ke...@citrix.com>.
I understood GCC to have a AL 2.0 license and to be used by the UI, as described in [1]. So it should at least be acceptable per license. I don't see why it has to be kept in the source tree but a removal of it should also include build changes to download a known-stable version for use in the build, or some testing to prove that the perf increase it creates is too small to care about.
[1] http://confluence.cloudstack.org/display/dev/Moving+dependencies+to+ASF+approved+licenses.
> -----Original Message-----
> From: Chiradeep Vittal [mailto:Chiradeep.Vittal@citrix.com]
> Sent: Thursday, July 05, 2012 11:02 AM
> To: CloudStack DeveloperList
> Subject: Re: [DISCUSS] tools directory
>
>
>
> On 7/4/12 10:12 PM, "Prasanna Santhanam"
> <Pr...@citrix.com>
> wrote:
>
> >On Wed, Jul 04, 2012 at 04:20:35PM -0400, David Nalley wrote:
> >> Hi folks,
> >>
> >> I spent some time poking around in the tools directory.
> >>
> >> There are a number of software packages in this directory, and I am
> >> not sure that it is appropriate to have them there, and specifically
> >> some of them are prohibited items.
> >>
> >> I am proposing to delete the following directories:
> >>
> >> ant
> >> gcc
> >> junit
> >> migration
> >> mockito
> >> tooljars
> >> vhd-tools
> >>
> >> If this offends you, please let us know why it must stay.
> >>
> >> I suppose the remaining can/should stay, but if you know a reason why
> >> not, please let us know that too.
> >>
> >> --David
> >
> >The copyright notice script in tools/build/copyright.py can be hacked
> >to have the license headers for ASF be automatically added. Anyone
> >already have any plan for this? Will this be useful David?
>
> Migration: this is/was a tool to assist 1.0.x -> 2.1.x migration. Can be nuked
> ant: sure -- your dev OS should have it, unless hudson/jenkins relies on this
> folder
> gcc: this seems to be used by the UI javascript?
> Junit/mockito: both libraries are used for unit tests, not sure this is safe to
> nuke
> Tooljars: not sure if folks are using selenium. Haven't seen any test cases
> driven by selenium
>
RE: [DISCUSS] tools directory
Posted by Sudha Ponnaganti <su...@citrix.com>.
Please leave Selenium related tools in place. We may revive this for UI automation in future.
-----Original Message-----
From: Chiradeep Vittal [mailto:Chiradeep.Vittal@citrix.com]
Sent: Thursday, July 05, 2012 11:02 AM
To: CloudStack DeveloperList
Subject: Re: [DISCUSS] tools directory
On 7/4/12 10:12 PM, "Prasanna Santhanam" <Pr...@citrix.com>
wrote:
>On Wed, Jul 04, 2012 at 04:20:35PM -0400, David Nalley wrote:
>> Hi folks,
>>
>> I spent some time poking around in the tools directory.
>>
>> There are a number of software packages in this directory, and I am
>> not sure that it is appropriate to have them there, and specifically
>> some of them are prohibited items.
>>
>> I am proposing to delete the following directories:
>>
>> ant
>> gcc
>> junit
>> migration
>> mockito
>> tooljars
>> vhd-tools
>>
>> If this offends you, please let us know why it must stay.
>>
>> I suppose the remaining can/should stay, but if you know a reason why
>> not, please let us know that too.
>>
>> --David
>
>The copyright notice script in tools/build/copyright.py can be hacked
>to have the license headers for ASF be automatically added. Anyone
>already have any plan for this? Will this be useful David?
Migration: this is/was a tool to assist 1.0.x -> 2.1.x migration. Can be nuked
ant: sure -- your dev OS should have it, unless hudson/jenkins relies on this folder
gcc: this seems to be used by the UI javascript?
Junit/mockito: both libraries are used for unit tests, not sure this is safe to nuke
Tooljars: not sure if folks are using selenium. Haven't seen any test cases driven by selenium
Re: [DISCUSS] tools directory
Posted by Chiradeep Vittal <Ch...@citrix.com>.
On 7/4/12 10:12 PM, "Prasanna Santhanam" <Pr...@citrix.com>
wrote:
>On Wed, Jul 04, 2012 at 04:20:35PM -0400, David Nalley wrote:
>> Hi folks,
>>
>> I spent some time poking around in the tools directory.
>>
>> There are a number of software packages in this directory, and I am
>> not sure that it is appropriate to have them there, and specifically
>> some of them are prohibited items.
>>
>> I am proposing to delete the following directories:
>>
>> ant
>> gcc
>> junit
>> migration
>> mockito
>> tooljars
>> vhd-tools
>>
>> If this offends you, please let us know why it must stay.
>>
>> I suppose the remaining can/should stay, but if you know a reason why
>> not, please let us know that too.
>>
>> --David
>
>The copyright notice script in tools/build/copyright.py can be hacked
>to have the license headers for ASF be automatically added. Anyone
>already have any plan for this? Will this be useful David?
Migration: this is/was a tool to assist 1.0.x -> 2.1.x migration. Can be
nuked
ant: sure -- your dev OS should have it, unless hudson/jenkins relies on
this folder
gcc: this seems to be used by the UI javascript?
Junit/mockito: both libraries are used for unit tests, not sure this is
safe to nuke
Tooljars: not sure if folks are using selenium. Haven't seen any test
cases driven by selenium
Re: [DISCUSS] tools directory
Posted by Prasanna Santhanam <Pr...@citrix.com>.
On Wed, Jul 04, 2012 at 04:20:35PM -0400, David Nalley wrote:
> Hi folks,
>
> I spent some time poking around in the tools directory.
>
> There are a number of software packages in this directory, and I am
> not sure that it is appropriate to have them there, and specifically
> some of them are prohibited items.
>
> I am proposing to delete the following directories:
>
> ant
> gcc
> junit
> migration
> mockito
> tooljars
> vhd-tools
>
> If this offends you, please let us know why it must stay.
>
> I suppose the remaining can/should stay, but if you know a reason why
> not, please let us know that too.
>
> --David
The copyright notice script in tools/build/copyright.py can be hacked
to have the license headers for ASF be automatically added. Anyone
already have any plan for this? Will this be useful David?
--
Prasanna.,
Re: [DISCUSS] tools directory
Posted by John Kinsella <jl...@stratosec.co>.
+1
Just noticed some of those myself and was wondering about that.
On Jul 4, 2012, at 1:20 PM, David Nalley wrote:
> Hi folks,
>
> I spent some time poking around in the tools directory.
>
> There are a number of software packages in this directory, and I am
> not sure that it is appropriate to have them there, and specifically
> some of them are prohibited items.
>
> I am proposing to delete the following directories:
>
> ant
> gcc
> junit
> migration
> mockito
> tooljars
> vhd-tools
>
> If this offends you, please let us know why it must stay.
>
> I suppose the remaining can/should stay, but if you know a reason why
> not, please let us know that too.
>
> --David
Stratosec - Secure Infrastructure as a Service
o: 415.315.9385
@johnlkinsella
RE: [DISCUSS] tools directory
Posted by Anthony Xu <Xu...@citrix.com>.
I'll try, before the patch is accepted by Xen community, I may need to keep this directory somewhere else.
Anthony
> -----Original Message-----
> From: Kevin Kluge [mailto:Kevin.Kluge@citrix.com]
> Sent: Friday, July 06, 2012 9:56 AM
> To: cloudstack-dev@incubator.apache.org
> Subject: RE: [DISCUSS] tools directory
>
> Anthony, would you be able to reach out to the Xen Community to see if
> they will take this upstream, or introduce a config that allows it to
> be disabled?
>
> -kevin
>
>
> > -----Original Message-----
> > From: Anthony Xu [mailto:Xuefei.Xu@citrix.com]
> > Sent: Thursday, July 05, 2012 12:56 PM
> > To: cloudstack-dev@incubator.apache.org
> > Subject: RE: [DISCUSS] tools directory
> >
> > vhd-util uses local time as timestamp when creating vhd, it may be
> okay with
> > XenServer, but in cloudstack, vhd files are moved around to different
> cluster,
> > the timestamp in vhd may be after the current time in the host.
> >
> > CloudStack only uses vhd-util to do vhd sanity check, The patch
> removes the
> > timestamp check, which might not be acceptable for xen community.
> >
> > It is not in upstream.
> >
> >
> > Anthony
> >
> > > -----Original Message-----
> > > From: David Nalley [mailto:david@gnsa.us]
> > > Sent: Thursday, July 05, 2012 12:11 PM
> > > To: cloudstack-dev@incubator.apache.org
> > > Subject: Re: [DISCUSS] tools directory
> > >
> > > On Thu, Jul 5, 2012 at 3:07 PM, Anthony Xu <Xu...@citrix.com>
> wrote:
> > > >> vhd-tools,
> > > >
> > > > We have some patches in this directory.
> > > > You may need put this directory somewhere else if you want to
> remove
> > > it from ASF.
> > > >
> > > > Anthony
> > >
> > >
> > > Have they been upstreamed??
> > >
> > > --David
RE: [DISCUSS] tools directory
Posted by Kevin Kluge <Ke...@citrix.com>.
Anthony, would you be able to reach out to the Xen Community to see if they will take this upstream, or introduce a config that allows it to be disabled?
-kevin
> -----Original Message-----
> From: Anthony Xu [mailto:Xuefei.Xu@citrix.com]
> Sent: Thursday, July 05, 2012 12:56 PM
> To: cloudstack-dev@incubator.apache.org
> Subject: RE: [DISCUSS] tools directory
>
> vhd-util uses local time as timestamp when creating vhd, it may be okay with
> XenServer, but in cloudstack, vhd files are moved around to different cluster,
> the timestamp in vhd may be after the current time in the host.
>
> CloudStack only uses vhd-util to do vhd sanity check, The patch removes the
> timestamp check, which might not be acceptable for xen community.
>
> It is not in upstream.
>
>
> Anthony
>
> > -----Original Message-----
> > From: David Nalley [mailto:david@gnsa.us]
> > Sent: Thursday, July 05, 2012 12:11 PM
> > To: cloudstack-dev@incubator.apache.org
> > Subject: Re: [DISCUSS] tools directory
> >
> > On Thu, Jul 5, 2012 at 3:07 PM, Anthony Xu <Xu...@citrix.com> wrote:
> > >> vhd-tools,
> > >
> > > We have some patches in this directory.
> > > You may need put this directory somewhere else if you want to remove
> > it from ASF.
> > >
> > > Anthony
> >
> >
> > Have they been upstreamed??
> >
> > --David
RE: [DISCUSS] tools directory
Posted by Anthony Xu <Xu...@citrix.com>.
vhd-util uses local time as timestamp when creating vhd, it may be okay with XenServer, but in cloudstack, vhd files are moved around to
different cluster, the timestamp in vhd may be after the current time in the host.
CloudStack only uses vhd-util to do vhd sanity check, The patch removes the timestamp check, which might not be acceptable for xen community.
It is not in upstream.
Anthony
> -----Original Message-----
> From: David Nalley [mailto:david@gnsa.us]
> Sent: Thursday, July 05, 2012 12:11 PM
> To: cloudstack-dev@incubator.apache.org
> Subject: Re: [DISCUSS] tools directory
>
> On Thu, Jul 5, 2012 at 3:07 PM, Anthony Xu <Xu...@citrix.com> wrote:
> >> vhd-tools,
> >
> > We have some patches in this directory.
> > You may need put this directory somewhere else if you want to remove
> it from ASF.
> >
> > Anthony
>
>
> Have they been upstreamed??
>
> --David
Re: [DISCUSS] tools directory
Posted by David Nalley <da...@gnsa.us>.
On Thu, Jul 5, 2012 at 3:07 PM, Anthony Xu <Xu...@citrix.com> wrote:
>> vhd-tools,
>
> We have some patches in this directory.
> You may need put this directory somewhere else if you want to remove it from ASF.
>
> Anthony
Have they been upstreamed??
--David
RE: [DISCUSS] tools directory
Posted by Anthony Xu <Xu...@citrix.com>.
> vhd-tools,
We have some patches in this directory.
You may need put this directory somewhere else if you want to remove it from ASF.
Anthony
> -----Original Message-----
> From: David Nalley [mailto:david@gnsa.us]
> Sent: Wednesday, July 04, 2012 1:21 PM
> To: cloudstack-dev@incubator.apache.org
> Subject: [DISCUSS] tools directory
>
> Hi folks,
>
> I spent some time poking around in the tools directory.
>
> There are a number of software packages in this directory, and I am
> not sure that it is appropriate to have them there, and specifically
> some of them are prohibited items.
>
> I am proposing to delete the following directories:
>
> ant
> gcc
> junit
> migration
> mockito
> tooljars
> vhd-tools
>
> If this offends you, please let us know why it must stay.
>
> I suppose the remaining can/should stay, but if you know a reason why
> not, please let us know that too.
>
> --David