Incubator Proposal: Pig

[Olga Natkovich <ol...@yahoo-inc.com>]

Hi,
 
Yahoo! research and development teams have developed a proposal below. The
proposal is also available on wiki at
<http://wiki.apache.org/incubator/PigProposal>
http://wiki.apache.org/incubator/PigProposal.
We would like to ask that the ASF consider forming a podling according to
the proposal.

Thanks,

Olga Natkovich
 <ma...@yahoo-inc.com> olgan@yahoo-inc.com

----------------------------------------------------------------------------
---------

= Pig Open Source Proposal =

== Abstract ==

Pig is a platform for analyzing large data sets. 

== Proposal ==

The Pig project consists of high-level languages for expressing data
analysis programs, coupled with infrastructure for evaluating these
programs. The salient property of Pig programs is that their structure is
amenable to substantial parallelization, which in turns enables them to
handle very large data sets.

At the present time, Pig's infrastructure layer consists of a compiler that
produces sequences of Map-Reduce programs, for which large-scale parallel
implementations already exist (e.g., the Hadoop subproject). Pig's language
layer currently consists of a textual language called Pig Latin, which has
the following key properties:

 1. ''Ease of programming''. It is trivial to achieve parallel execution of
simple, "embarrassingly parallel" data analysis tasks. Complex tasks
comprised of multiple interrelated data transformations are explicitly
encoded as data flow sequences, making them easy to write, understand, and
maintain.
 2. ''Optimization opportunities''. The way in which tasks are encoded
permits the system to optimize their execution automatically, allowing the
user to focus on semantics rather than efficiency.
 3. ''Extensibility''. Users can create their own functions to do
special-purpose processing. 

== Background ==

Pig started as a research project at Yahoo! in May of 2006 to combine ideas
in parallel databases and distributed computing. The first internal release
took place in July 2006. The first release was a simple front-end to the
Hadoop Map/Reduce framework. The following releases added new features and
evolved the language based on user feedback. In July 2007, pig was taken
over by a development team and the first production version is due to be
released on 9/28/07.

Since its inception, we had observed a steady growth of the user community
within Yahoo!.  In April 2007, Pig was released under a BSD-type license.
Several external parties are using this version and have expressed interest
in collaborating on its development.

== Rationale ==

In an information-centric world, innovation is driven by ad-hoc analysis of
large data sets. For example, search engine companies routinely deploy and
refine services based on analyzing the recorded behavior of users,
publishers, and advertisers. The rate of innovation depends on the
efficiency with which data can be
analyzed.

To analyze large data sets efficiently, one needs parallelism. The cheapest
and most scalable form of parallelism is cluster computing. Unfortunately,
programming for a cluster computing environment is difficult and
time-consuming. Pig makes it easy to harness the power of cluster computing
for ad-hoc data analysis. 

While other language exist that try to achieve the same goals, we believe
that Pig provides more flexibility and gives more control to the end user. 

SQL typically requires (1) importing data from a user's preferred format
into a database system's internal format (2) well-structured, normalized
data with a declared schema, and (3) programs expressed in declarative
SELECT-FROM-WHERE blocks. In contrast, Pig Latin facilitates (1)
interoperability, i.e. data may be read/written in a format accepted by
other applications such as text editors or graph generators (2) flexibility,
i.e. data may be loosely structured or have structure that is
defined operationally, and (3) adoption by programmers who find procedural
programming more natural than declarative programming.

Sawzall is a scripting language used at Google on top of Map-Reduce. A
sawzall program has a fairly rigid structure consisting of a filtering phase
(the map step) followed by an aggregation phase (the reduce step).
Furthermore, only the filtering phase can be written by the user, and only a
pre-built set of aggregations are available (new ones are non-trivial to
add). While Pig Latin has similar higher level primitives like filtering and
aggregation, an arbitrary number of them can be flexibly chained together in
a Pig Latin program, and all primitives can use user-defined functions with
equal ease. Further, Pig Latin has additional primitives such as cogrouping,
that allow operations such as joins (which require multiple programs in
Sawzall) to be written in a single line in Pig Latin. Further, Pig Latin is
designed
to be embedded into other languages, and can use functions written in other
languages. Thus, in contrast to Sawzall, it directly caters to a large
community of developers without having to make them learn an entirely new
programming language.

== Current Status ==

=== Meritocracy ===

Pig was started as a project that was developed by Yahoo! research team.
Recently we have added a development team that works in harmony with the
research team with both teams actively and successfully contributing to the
project. We are planning to create the environment that encourages
meritocracy and is consistent with the meritocracy principles of Apache.
Within the team we have people actively participating in the Hadoop
subproject.

=== Community ===

Pig has an active user community within Yahoo! that has been steadily
growing. Pig also attracted external users since its release under a
BSD-type license.  Several external parties are using the product and have
expressed interest in collaborating on its development.

Also, since the current version of Pig is built on top of the Hadoop we
believe that we will be able to quickly extend our community by attracting
both the Hadoop users and developers to the project.

=== Core Developers ===

Our contributors come from both research and development world and most have
background in database internals and large scale distributed systems.

=== Alignment ===

Yahoo! seeks to develop Pig collaboratively with others, not to control and
maintain it independently.  Apache offers the best legal and social
framework for such community-based software development.  

Also, the current version of Pig runs on top of the Hadoop's Map-Reduce
infrastructure which is part of Apache. We believe there would be a lot of
synergy between the projects both in terms of users and developers.

== Known Risks ==
=== Orphaned products ===

All current contributors are part of Yahoo which is a major player in the
space and is committed to grid computing. Also we expect high degree of
synergy with Hadoop subproject.

=== Inexperience with Open Source ===

Two of the committers have extensive experience with open source and Apache.
The rest are new to open source and will be guided through the process by
the team members with experience.

=== Homogenous Developers ===

The current list of committers is confined to Yahoo employees. Our plan is
to recruit more committers once the project gets on the way.

=== Reliance on Salaried Developers ===

Currently, all contributors are Yahoo employees. By extending the
development community we are hoping to mitigate this risk.

=== Relationships with Other Apache Products ===

Pig is built on top of Hadoop and we expect deep collaboration with Hadoop
subproject.

=== An Excessive Fascination with the Apache Brand ===

Yahoo already have a strong brand and is not interested in Apache as a way
to gain visibility. Yahoo! seeks to develop Pig collaboratively with others,
not to control and maintain it independently.  Apache offers the best legal
and social framework for such community-based software development.  

== Documentation ==

http://research.yahoo.com/project/pig

== Initial Source ==

The initial source will be donated by Yahoo Inc. The donating company will
contribute the initial code base once the proposal is accepted and necessary
infrastructure has been set up. 

== External Dependencies ==

 1. bzip2:  <http://www.kohsuke.org/bzip2/:Apache>
http://www.kohsuke.org/bzip2/:Apache license
 2. javacc:  <https://javacc.dev.java.net/:BSD>
https://javacc.dev.java.net/:BSD license
 3. hadoop:  <http://lucene.apache.org/hadoop/:Apache>
http://lucene.apache.org/hadoop/:Apache license
 4. log4j:  <http://logging.apache.org/log4j/>
http://logging.apache.org/log4j/: Apache license
 5. jsch:  <http://www.jcraft.com/jsch> http://www.jcraft.com/jsch: BSD
style license:  <http://www.jcraft.com/jsch/LICENSE.txt>
http://www.jcraft.com/jsch/LICENSE.txt 

== Required Resources ==
== Mailing lists ==

We would need the following mailing lists
 1. pig-private (with moderated subscriptions)
 2. pig-dev
 3. pig-commits
 4. pig-user

=== Subversion Directory ===

https://svn.apache.org/repos/asf/incubator/pig

=== Issue Tracking ===

JIRA PIG (PIG)

== Initial Committers ==

 1. Nigel Daley ( <ma...@yahoo-inc.com> ndaley@yahoo-inc.com)
 2. Alan Gates ( <ma...@yahoo-inc.com> gates@yahoo-inc.com)
 3. Olga Natkovich ( <ma...@yahoo-inc.com> olgan@yahoo-inc.com)
 4. Chris Olston ( <ma...@yahoo-inc.com> olston@yahoo-inc.com)
 5. Owen O'Malley ( <ma...@yahoo-inc.com> oom@yahoo-inc.com)
 6. Ben Reed ( <ma...@yahoo-inc.com> breed@yahoo-inc.com)
 7. Utkarsh Srivastava ( <ma...@yahoo-inc.com>
utkarsh@yahoo-inc.com)

== Affiliation ==

All initial committers are affiliated with Yahoo!

== Sponsors ==

=== Champion ===

Doug Cutting 

=== Nominated Mentors ===

Doug Cutting 

=== Sponsoring Entity ===

Incubator

Re: Incubator Proposal: Pig

[Craig L Russell <Cr...@Sun.COM>]

On Sep 18, 2007, at 2:45 PM, Doug Cutting wrote:

> Garrett Rooney wrote:
>> (if anything it seems like a new TLP
>> associated with hadoop and generic distributed computing tools like
>> pig that are built on top of hadoop seems like it would make more
>> sense than just a pig TLP),
>
> Yes, I agree.  But that's not happened yet, and the Pig folks are  
> ready to enter the incubator now.
>
>> but if that's what the people involved
>> want I don't see anything wrong with it.  I mean it's not like a
>> decision on a final home for the project has to happen now anyway.
>
> Exactly.  If, when Pig is ready to graduate, there is a more Hadoop- 
> specific TLP, then it may make sense to have Pig join that as a sub- 
> project, or it may not.

I agree. One of the objectives of incubation is to decide exactly  
where in Apache a project (or sub-project) belongs. Let's get it into  
incubation and see what kind of community/synergy Pig can build with  
other projects.

Craig

> But, for now, the folks involved have elected to aim for TLP rather  
> than Lucene sub-project (the available options).
>
> Doug
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
>

Craig Russell
Architect, Sun Java Enterprise System http://java.sun.com/products/jdo
408 276-5638 mailto:Craig.Russell@sun.com
P.S. A good JDO? O, Gasp!

Re: Incubator Proposal: Pig

[Doug Cutting <cu...@apache.org>]

Garrett Rooney wrote:
> (if anything it seems like a new TLP
> associated with hadoop and generic distributed computing tools like
> pig that are built on top of hadoop seems like it would make more
> sense than just a pig TLP),

Yes, I agree.  But that's not happened yet, and the Pig folks are ready 
to enter the incubator now.

> but if that's what the people involved
> want I don't see anything wrong with it.  I mean it's not like a
> decision on a final home for the project has to happen now anyway.

Exactly.  If, when Pig is ready to graduate, there is a more 
Hadoop-specific TLP, then it may make sense to have Pig join that as a 
sub-project, or it may not.  But, for now, the folks involved have 
elected to aim for TLP rather than Lucene sub-project (the available 
options).

Doug

---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org

Re: Incubator Proposal: Pig

[Garrett Rooney <ro...@electricjellyfish.net>]

On 9/18/07, Doug Cutting <cu...@apache.org> wrote:
> Garrett Rooney wrote:
> > Is there any particular reason you want this podling to be sponsored
> > by the Incubator PMC (which is generally done for projects that intend
> > to turn into their own top level project) rather than having it
> > sponsored by the Lucene PMC (where Hadoop currently resides)?  It
> > seems to me that the close relationship between Pig and Hadoop implies
> > that they very well might best be served under the same roof.
>
> The existing contributor base is largely disjoint from the Hadoop
> contributor base, and they expect that to mostly remain the case.
> Nigel, Owen & I, Hadoop committers, will mostly just help the Pig crew
> out with Apache ways, and don't expect to become significant
> contributors to Pig.  Pig builds on Hadoop, and the communities may
> overlap a bit, but, to the primary folks involved, it feels like a
> separate community and they'd prefer to aim for a TLP.

Well, it seems a little odd to me (if anything it seems like a new TLP
associated with hadoop and generic distributed computing tools like
pig that are built on top of hadoop seems like it would make more
sense than just a pig TLP), but if that's what the people involved
want I don't see anything wrong with it.  I mean it's not like a
decision on a final home for the project has to happen now anyway.

In any event, +1 from me, this is a neat project and I'd be happy to
see it here.

-garrett

---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org

Re: Incubator Proposal: Pig

[Doug Cutting <cu...@apache.org>]

Yoav Shapira wrote:
> It should be clear to everyone involved, though, that part of the goal
> of incubation is to diversify the project's community so that it's not
> disjoint from everyone else.  I hope to have a bunch of non-Yahoo
> people contributing to the project.

Indeed.  That's the primary reason to move this to Apache: to be able to 
collaborate with others outside Y!.  If Y! didn't want to diversify the 
community it could just keep posting code dumps under BSD as it does today.

Doug

---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org

Re: Incubator Proposal: Pig

[Yoav Shapira <yo...@apache.org>]

Hey,

On 9/18/07, Doug Cutting <cu...@apache.org> wrote:
> overlap a bit, but, to the primary folks involved, it feels like a
> separate community and they'd prefer to aim for a TLP.

It should be clear to everyone involved, though, that part of the goal
of incubation is to diversify the project's community so that it's not
disjoint from everyone else.  I hope to have a bunch of non-Yahoo
people contributing to the project.

Yoav

---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org

Re: Incubator Proposal: Pig

[Doug Cutting <cu...@apache.org>]

Garrett Rooney wrote:
> Is there any particular reason you want this podling to be sponsored
> by the Incubator PMC (which is generally done for projects that intend
> to turn into their own top level project) rather than having it
> sponsored by the Lucene PMC (where Hadoop currently resides)?  It
> seems to me that the close relationship between Pig and Hadoop implies
> that they very well might best be served under the same roof.

The existing contributor base is largely disjoint from the Hadoop 
contributor base, and they expect that to mostly remain the case. 
Nigel, Owen & I, Hadoop committers, will mostly just help the Pig crew 
out with Apache ways, and don't expect to become significant 
contributors to Pig.  Pig builds on Hadoop, and the communities may 
overlap a bit, but, to the primary folks involved, it feels like a 
separate community and they'd prefer to aim for a TLP.

Doug

---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org

Re: Incubator Proposal: Pig

[Garrett Rooney <ro...@electricjellyfish.net>]

On 9/18/07, Lawrence Mandel <lm...@ca.ibm.com> wrote:
> >Is there any particular reason you want this podling to be sponsored
> >by the Incubator PMC (which is generally done for projects that intend
> >to turn into their own top level project)
>
> Is this true? I thought all new projects had to go through the incubator.
> Woden [1] is an incubator project that plans to graduate and join the WS
> PMC.
>
> [1] http://incubator.apache.org/woden/

All projects need to go through the incubation process, but not all
are sponsored by the incubator PMC, many are sponsored by an existing
PMC outside the incubator.  I don't recall for sure, but I'd expect
that woden entered the incubator after being sponsored by the WS PMC
(at least, that's how things tend to work today if I understand
correctly, it's quite possible that woden predates that practice, I'm
really not sure).

-garrett

---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org

Re: Incubator Proposal: Pig

[Lawrence Mandel <lm...@ca.ibm.com>]

>Is there any particular reason you want this podling to be sponsored
>by the Incubator PMC (which is generally done for projects that intend
>to turn into their own top level project)

Is this true? I thought all new projects had to go through the incubator. 
Woden [1] is an incubator project that plans to graduate and join the WS 
PMC.

[1] http://incubator.apache.org/woden/

Lawrence




"Garrett Rooney" <ro...@electricjellyfish.net> 
Sent by: rooneg@gmail.com
09/18/2007 04:02 PM
Please respond to
general@incubator.apache.org


To
general@incubator.apache.org
cc

Subject
Re: Incubator Proposal: Pig






On 9/18/07, Olga Natkovich <ol...@yahoo-inc.com> wrote:
> Hi,
>
> Yahoo! research and development teams have developed a proposal below. 
The
> proposal is also available on wiki at
> <http://wiki.apache.org/incubator/PigProposal>
> http://wiki.apache.org/incubator/PigProposal.
> We would like to ask that the ASF consider forming a podling according 
to
> the proposal.

Is there any particular reason you want this podling to be sponsored
by the Incubator PMC (which is generally done for projects that intend
to turn into their own top level project) rather than having it
sponsored by the Lucene PMC (where Hadoop currently resides)?  It
seems to me that the close relationship between Pig and Hadoop implies
that they very well might best be served under the same roof.

-garrett

---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org




---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org

Re: Incubator Proposal: Pig

[Garrett Rooney <ro...@electricjellyfish.net>]

On 9/18/07, Olga Natkovich <ol...@yahoo-inc.com> wrote:
> Hi,
>
> Yahoo! research and development teams have developed a proposal below. The
> proposal is also available on wiki at
> <http://wiki.apache.org/incubator/PigProposal>
> http://wiki.apache.org/incubator/PigProposal.
> We would like to ask that the ASF consider forming a podling according to
> the proposal.

Is there any particular reason you want this podling to be sponsored
by the Incubator PMC (which is generally done for projects that intend
to turn into their own top level project) rather than having it
sponsored by the Lucene PMC (where Hadoop currently resides)?  It
seems to me that the close relationship between Pig and Hadoop implies
that they very well might best be served under the same roof.

-garrett

---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org

Re: Incubator Proposal: Pig

[Ted Dunning <te...@gmail.com>]

+1 as well.

I would be happy to help with code contributions and user testing.


Yoav Shapira-2 wrote:
> 
> Hey,
> 
> On 9/18/07, Olga Natkovich <ol...@yahoo-inc.com> wrote:
>> Yahoo! research and development teams have developed a proposal below.
>> The
>> proposal is also available on wiki at
>> <http://wiki.apache.org/incubator/PigProposal>
> 
> Looks very cool to me.  +1 to accepting Pig as an Incubator project.
> 
> I'll also gladly volunteer as a mentor.
> 
> Yoav
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
> 
> 
> 

-- 
View this message in context: http://www.nabble.com/Incubator-Proposal%3A-Pig-tf4476730.html#a12766208
Sent from the Apache Incubator - General mailing list archive at Nabble.com.


---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org

Re: Incubator Proposal: Pig

[Yoav Shapira <yo...@apache.org>]

Hey,

On 9/18/07, Olga Natkovich <ol...@yahoo-inc.com> wrote:
> Yahoo! research and development teams have developed a proposal below. The
> proposal is also available on wiki at
> <http://wiki.apache.org/incubator/PigProposal>

Looks very cool to me.  +1 to accepting Pig as an Incubator project.

I'll also gladly volunteer as a mentor.

Yoav

---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org

Re: Incubator Proposal: Pig

[Jim Hurley <Ji...@Sun.COM>]

+1

-Jim

On Sep 18, 2007, at 3:52 PM, Olga Natkovich wrote:
> Hi,
>
> Yahoo! research and development teams have developed a proposal  
> below. The
> proposal is also available on wiki at
> <http://wiki.apache.org/incubator/PigProposal>
> http://wiki.apache.org/incubator/PigProposal.
> We would like to ask that the ASF consider forming a podling  
> according to
> the proposal.
>
> Thanks,
>
> Olga Natkovich
>  <ma...@yahoo-inc.com> olgan@yahoo-inc.com
>
> ---------------------------------------------------------------------- 
> ------
> ---------
>
> = Pig Open Source Proposal =
>
> == Abstract ==
>
> Pig is a platform for analyzing large data sets.
>
> == Proposal ==
>
> The Pig project consists of high-level languages for expressing data
> analysis programs, coupled with infrastructure for evaluating these
:
:
:

---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org

Re: Incubator Proposal: Pig

[Grant Ingersoll <gs...@apache.org>]

On Sep 24, 2007, at 4:44 AM, Sylvain Wallez wrote:

> Olga Natkovich wrote:
>> Hi,
>>
>> Yahoo! research and development teams have developed a proposal  
>> below. The
>> proposal is also available on wiki at
>> http://wiki.apache.org/incubator/PigProposal.
>> We would like to ask that the ASF consider forming a podling  
>> according to
>> the proposal.
>>

+1


---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org

Re: Effects on corporate backing withdrawals [was: Incubator Proposal: Pig]

[Matthieu Riou <ma...@offthelip.org>]

On 9/25/07, Guillaume Nodet <gn...@gmail.com> wrote:
>
> One of the purpose of the incubator is to ensure that there is a
> sustainable developer community, so I don't see failure of incubating
> projects as a real problem.


Actually I think it's showing the incubator's success even if it's always
sad to see a project die. I would see a problem if a project graduated
successfully from the incubator and then got abandoned a few months later,
which I haven't seen so far. So a strong corporate backing is maybe just a
sign that more attention is going to be needed from mentors and the IPMC but
I don't see it as worrying either.

Matthieu

On 9/25/07, Noel J. Bergman < noel@devtech.com> wrote:
> > Dims wrote:
> > > Niclas Hedhman asked:
> > > > Do we have any examples where corporate backing has been withdrawn,
> and
> > how
> > > > the project was affected, whether inside or outside ASF?
> > > TSIK -> Verisign folks lost interest, community did not form, project
> > shelved.
> >
> > Plus Kabuki, Heraldry, and possibly Lokahi.  We still hope to save the
> > latter, as there is consistently a lot of user interest, but the
> developer
> > input has dwindled, and we need developers!
> >
> >         --- Noel
> >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> > For additional commands, e-mail: general-help@incubator.apache.org
> >
> >
>
>
> --
> Cheers,
> Guillaume Nodet
> ------------------------
> Blog: http://gnodet.blogspot.com/
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
>
>

Re: Effects on corporate backing withdrawals [was: Incubator Proposal: Pig]

[Robert Burrell Donkin <ro...@gmail.com>]

On 9/25/07, Craig L Russell <Cr...@sun.com> wrote:
>
> On Sep 25, 2007, at 8:28 AM, Guillaume Nodet wrote:
>
> > One of the purpose of the incubator is to ensure that there is a
> > sustainable developer community, so I don't see failure of incubating
> > projects as a real problem.
>
> +1.
>
> If we knew for sure that a project would be able to attract a
> community, we would have much less need for incubation.

+1

- robert

---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org

Re: Effects on corporate backing withdrawals [was: Incubator Proposal: Pig]

[Craig L Russell <Cr...@Sun.COM>]

On Sep 25, 2007, at 8:28 AM, Guillaume Nodet wrote:

> One of the purpose of the incubator is to ensure that there is a
> sustainable developer community, so I don't see failure of incubating
> projects as a real problem.

+1.

If we knew for sure that a project would be able to attract a  
community, we would have much less need for incubation.

Craig

"I have not failed. I've just found 10,000 ways that won't work."  
Thomas Alva Edison

> On 9/25/07, Noel J. Bergman <no...@devtech.com> wrote:
>> Dims wrote:
>>> Niclas Hedhman asked:
>>>> Do we have any examples where corporate backing has been  
>>>> withdrawn, and
>> how
>>>> the project was affected, whether inside or outside ASF?
>>> TSIK -> Verisign folks lost interest, community did not form,  
>>> project
>> shelved.
>>
>> Plus Kabuki, Heraldry, and possibly Lokahi.  We still hope to save  
>> the
>> latter, as there is consistently a lot of user interest, but the  
>> developer
>> input has dwindled, and we need developers!
>>
>>         --- Noel
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
>> For additional commands, e-mail: general-help@incubator.apache.org
>>
>>
>
>
> -- 
> Cheers,
> Guillaume Nodet
> ------------------------
> Blog: http://gnodet.blogspot.com/
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
>

Craig Russell
Architect, Sun Java Enterprise System http://java.sun.com/products/jdo
408 276-5638 mailto:Craig.Russell@sun.com
P.S. A good JDO? O, Gasp!

Re: Effects on corporate backing withdrawals [was: Incubator Proposal: Pig]

[Jim Jagielski <ji...@jaguNET.com>]

On Sep 25, 2007, at 11:28 AM, Guillaume Nodet wrote:

> One of the purpose of the incubator is to ensure that there is a
> sustainable developer community, so I don't see failure of incubating
> projects as a real problem.
>

Neither do I...


---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org

Re: HttpClient, HttpComponents (was Jakarta)

[Robert Burrell Donkin <ro...@gmail.com>]

On 10/12/07, Roland Weber <os...@dubioso.net> wrote:

<snip>

> > I'd keep server side separate, but a rest-centric server side stack
> > (with no attempts to support WSDL) would be appealing, though restlets
> > exist for that purpose.
>
> Hmm. Can't say that I've ever perceived REST as much more than a
> buzzword,

FWIW REST is one of the better defined architectural styles
(http://www.ics.uci.edu/~fielding/pubs/dissertation/top.htm). if you
want to understand why ROA is becoming more important, read RESTful
Web Services (see http://www.crummy.com/writing/RESTful-Web-Services/,
for example).

- robert

---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org

Re: HttpClient, HttpComponents (was Jakarta)

[Eelco Hillenius <ee...@gmail.com>]

> Hmm. Can't say that I've ever perceived REST as much more than a
> buzzword

One of these things that started out as an architectural pattern, and
got hijacked for a thousand different interpretations. If it 'has'
REST, it's gotta be good ;-)

Eelco

---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org

Re: HttpClient, HttpComponents (was Jakarta)

[Roland Weber <os...@dubioso.net>]

Hi Steve, Niclas,

>>> Any Java stack that does SOAP or REST should be using
>>> HttpClient
>>
>> That seems to indicate that it should be a TLP in its own right.
> 
> you want that, you get my support.

Thanks for your encouragement. I'm currently drowned in work,
but I intend to kick off the (hopefully last) TLP discussion
on httpcomponents-dev@jakarta before the end of this month.
Unless Oleg beats me to it :-)

> axis, ant-contrib, maven, jetty, wss4j, groovy. So not xerces or xalan.
> Maybe they are hoping someone will fix java.net.HttpUrlConnection now it
> is OSS/

Xerces and Xalan have plug-in interfaces for resolving external
references. They wouldn't do themselves a favour by adding an
external dependency for the default implementation. Just as we
wouldn't do ourselves a favour by shipping HttpComponents with
a specific SSL implementation.

http://xerces.apache.org/xerces2-j/javadocs/api/org/xml/sax/EntityResolver.html
http://xerces.apache.org/xerces2-j/javadocs/api/org/xml/sax/ext/EntityResolver2.html
http://xerces.apache.org/xerces2-j/javadocs/api/javax/xml/transform/URIResolver.html
http://xml.apache.org/xalan-j/apidocs/javax/xml/transform/URIResolver.html


>> And if the charter is extended to http server side as well (without
>> the servlet cruft ;o) ), we have a pretty immense project.

That already happened when HttpComponents left Commons
to become a separate Jakarta sub-project two years ago:
http://jakarta.apache.org/httpcomponents/charter.html

HttpComponents Core is agnostic, while HttpComponents Client is
the future replacement for the client-side 3.x codebase we'd like
to bury. The charter also includes specific statements that we will
not be developing server-side application layer APIs. I believe that
was necessary because others developing server-side HTTP (Tomcat?)
felt that we were stepping on their toes. At the time, it was
Apache policy to not allow competing projects, or so I've heard.

Your help to define the scope for the new project will be most
welcome.

> I'd keep server side separate, but a rest-centric server side stack
> (with no attempts to support WSDL) would be appealing, though restlets
> exist for that purpose.

Hmm. Can't say that I've ever perceived REST as much more than a
buzzword, but you're welcome to share your ideas. Even more so if
you bring in some folks who want to contribute code ;-) The current
developers are maxed out with pushing the two components we've got
towards production quality.

cheers,
  Roland


---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org

Re: HttpClient, HttpComponents (was Jakarta)

[Steve Loughran <st...@apache.org>]

Niclas Hedhman wrote:
> On Monday 08 October 2007 18:37, Steve Loughran wrote:
>> Any Java stack that does SOAP or REST should be using
>> HttpClient
> 
> That seems to indicate that it should be a TLP in its own right.

you want that, you get my support.

> 
> Besides WS, we have;
>  - Maven do the http client stuff... the naive version??
>  - Ivy would be in the same boat.
>  - Any XML parser and XSL transformer must be supporting externalizations.
> the list as you initiated will probably be long at exhaustive search.

gump shows some dependencies; a full walk of the m2 metadata will show 
the rest

http://vmgump.apache.org/gump/public/httpcomponents/commons-httpclient/details.html

axis, ant-contrib, maven, jetty, wss4j, groovy. So not xerces or xalan. 
Maybe they are hoping someone will fix java.net.HttpUrlConnection now it 
is OSS/

> 
> And if the charter is extended to http server side as well (without the 
> servlet cruft ;o) ), we have a pretty immense project.

lovely servlet cruft. Much better than what sun builds in to java6.

I'd keep server side separate, but a rest-centric server side stack 
(with no attempts to support WSDL) would be appealing, though restlets 
exist for that purpose.

-steve



---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org

Re: HttpClient, HttpComponents (was Jakarta)

[Niclas Hedhman <ni...@hedhman.org>]

On Monday 08 October 2007 18:37, Steve Loughran wrote:
> Any Java stack that does SOAP or REST should be using
> HttpClient

That seems to indicate that it should be a TLP in its own right.

Besides WS, we have;
 - Maven do the http client stuff... the naive version??
 - Ivy would be in the same boat.
 - Any XML parser and XSL transformer must be supporting externalizations.
the list as you initiated will probably be long at exhaustive search.

And if the charter is extended to http server side as well (without the 
servlet cruft ;o) ), we have a pretty immense project.


Cheers
-- 
Niclas Hedhman, Software Developer

I  live here; http://tinyurl.com/2qq9er
I  work here; http://tinyurl.com/2ymelc
I relax here; http://tinyurl.com/2cgsug

---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org

Re: HttpClient, HttpComponents (was Jakarta)

[Steve Loughran <st...@apache.org>]

Roland Weber wrote:
> A fallback option is to move from Jakarta to
> WebServices, exchanging one umbrella for another. That wouldn't
> move us forward, and the impression we got from Jakarta is that
> umbrellas are not in favour at the board.
> As a TLP, we will have a fighting chance to grow the project
> to the point where it no longer depends on just the two of us.
> Until that is achieved, we'll be one of those projects that
> Niall referred to, with community issues because they didn't
> pass through the Incubator. That's why I thought it was a good
> occasion to ask for suggestions. The new HttpComponents as well
> as the old HttpClient we maintain are being used by Apache
> projects, so coming into the Incubator is not an option for
> HttpComponents.

I dont consider http support in webservices as a general purpose 
umbrella. Any Java stack that does SOAP or REST should be using 
HttpClient unless they are naiive fools who believe that there is 
someone that actually maintains java.net.HttpUrl, and that persion has 
actually read the HTTP RFCs.

1. it is used in Axis, XFire, Alpine. I think the only mainstream stack 
that doesnt use it is the Sun one, but they have the ability to fix bits 
of java.net, and/or the naiive optimism I mentioned earlier.


2. It would be a good way of kickstarting more pure RESTy stuff in the 
WS group.

3. this has been raised on ws pmc, we'd love to have the 
HttpComponents/HttpClient team(s) on board if you want to join.

-steve

(still on the WS PMC, incidentally)

---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org

Re: Jakarta [was: Effects on corporate backing withdrawals [was: Incubator Proposal: Pig]]

[Erik Abele <er...@codefaktor.de>]

On 01.10.2007, at 18:43, Roland Weber wrote:

> Erik Abele wrote:
>> Sure, am happy to help (as a satisfied user of both,  
>> HttpComponents and
>> JMeter); just let me know where you'd like to see me subscribed... (I
>> assume httpcomponents-dev@j.a.o and jmeter-dev-@j.a.o?)
>
> That's great! Yes, those will be the interesting lists in terms
> of future directions for both subprojects.

Done.

> If the traffic on
> either list is too high, you could subscribing to general@jakarta
> instead.

Already subscribed.

> I'll make sure to post there when discussions get on
> the way. For HttpComponents, we're planning to prepare the TLP
> proposal for the December board meeting.

Nice.

> You can find some older
> discussions in the mailing list archives.

Ok, will have a look, thx for the pointers.

Cheers,
Erik

---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org

Re: Jakarta [was: Effects on corporate backing withdrawals [was: Incubator Proposal: Pig]]

[Roland Weber <os...@dubioso.net>]

Erik Abele wrote:
> Sure, am happy to help (as a satisfied user of both, HttpComponents and
> JMeter); just let me know where you'd like to see me subscribed... (I
> assume httpcomponents-dev@j.a.o and jmeter-dev-@j.a.o?)

That's great! Yes, those will be the interesting lists in terms
of future directions for both subprojects. If the traffic on
either list is too high, you could subscribing to general@jakarta
instead. I'll make sure to post there when discussions get on
the way. For HttpComponents, we're planning to prepare the TLP
proposal for the December board meeting. You can find some older
discussions in the mailing list archives.

thanks!
  Roland




---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org

Re: Jakarta [was: Effects on corporate backing withdrawals [was: Incubator Proposal: Pig]]

[Erik Abele <er...@codefaktor.de>]

On 30.09.2007, at 18:17, Roland Weber wrote:

> Niclas Hedhman wrote:
>> I don't know what to suggest, but perhaps recruiting one or more  
>> veteran
>> ASFer, either just off the member's list or some experienced  
>> Incubator
>> mentor, feeling this being important could just join the PMC and  
>> at least
>> ensure process with 3 pairs of eye balls.
>
> Yeah, we'll try to get a veteran (though not a member) to help us  
> out as
> the chair for the initial phase. (speaking for HttpComponents, not  
> JMeter)
>
> If anyone here feels like keeping an eye on us too, you're most  
> welcome.
> We know our way through the code and public processes up to PMC,  
> but we
> currently don't have an ASF member on board who is familiar with  
> what's
> going on beyond PMC. (speaking for both)

Sure, am happy to help (as a satisfied user of both, HttpComponents  
and JMeter); just let me know where you'd like to see me  
subscribed... (I assume httpcomponents-dev@j.a.o and jmeter-dev-@j.a.o?)

Cheers,
Erik

---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org

Re: Jakarta [was: Effects on corporate backing withdrawals [was: Incubator Proposal: Pig]]

[Roland Weber <os...@dubioso.net>]

Niclas Hedhman wrote:
> I don't know what to suggest, but perhaps recruiting one or more veteran 
> ASFer, either just off the member's list or some experienced Incubator 
> mentor, feeling this being important could just join the PMC and at least 
> ensure process with 3 pairs of eye balls.

Yeah, we'll try to get a veteran (though not a member) to help us out as
the chair for the initial phase. (speaking for HttpComponents, not JMeter)

If anyone here feels like keeping an eye on us too, you're most welcome.
We know our way through the code and public processes up to PMC, but we
currently don't have an ASF member on board who is familiar with what's
going on beyond PMC. (speaking for both)

thanks for taking the time,
  Roland


---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org

Re: Jakarta [was: Effects on corporate backing withdrawals [was: Incubator Proposal: Pig]]

[Niclas Hedhman <ni...@hedhman.org>]

On Sunday 30 September 2007 01:19, Roland Weber wrote:
> The new HttpComponents as well
> as the old HttpClient we maintain are being used by Apache
> projects, so coming into the Incubator is not an option for
> HttpComponents.

I agree. And typically, TLPs receive somewhat more exposure than sub-projects 
and a better chance of building a stronger community.

I don't know what to suggest, but perhaps recruiting one or more veteran 
ASFer, either just off the member's list or some experienced Incubator 
mentor, feeling this being important could just join the PMC and at least 
ensure process with 3 pairs of eye balls.
It sounds to me there should be such interest...

Cheers
-- 
Niclas Hedhman, Software Developer

I  live here; http://tinyurl.com/2qq9er
I  work here; http://tinyurl.com/2ymelc
I relax here; http://tinyurl.com/2cgsug

---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org

Re: Jakarta [was: Effects on corporate backing withdrawals [was: Incubator Proposal: Pig]]

[Roland Weber <os...@dubioso.net>]

Hello Niclas,

>> Staying at Jakarta will buy some time, but won't last forever.
>> >
>> > If you have ideas on what to do with these small but active
>> > projects, please come over to general@jakarta and share your
>> > thoughts.
> 
> Can't Jakarta just be revitalized as a home for small, but mature and stable 
> projects??

I feel Jakarta has to downsize some more before we can think
about reviving it in a new role. A home for small, mature and
stable projects is surely an option, though I don't believe
it will be easy.
The two projects I care about, however, do not match that profile.
They are highly active, evolving, and growing. You can see that
from Jakarta's September board report[1]: 5 releases, all of them
from JMeter and HttpComponents. JMeter released 2.3 final today,
and HttpComponents has three more releases in the pipeline until
the end of the year. Both projects have hundreds of mails on
their lists each month. That's not what I would associate with
"mature and stable", which sounds more like "maintenance mode".

So, both projects are actively developed and used. We know how
to vote and cut releases. The projects have prospects of growing
and attracting a larger user base, from which we can hope to
get new committers over time. But at the moment, both depend
on a very small group of developers that provide continuity,
with occasional patches from others coming in. And we don't get
the time to grow organically, with Jakarta disintegrating.

I don't know what Sebastian plans for JMeter. Oleg and I will
push for an HttpComponents TLP later this year. Not because we
feel that the project is ready for that move, but because we
see it is the best option left to us. Either we stay at Jakarta
until we're being asked to leave (or shut down), or we make a
move of our own while we can still choose the time to our
convenience. A fallback option is to move from Jakarta to
WebServices, exchanging one umbrella for another. That wouldn't
move us forward, and the impression we got from Jakarta is that
umbrellas are not in favour at the board.
As a TLP, we will have a fighting chance to grow the project
to the point where it no longer depends on just the two of us.
Until that is achieved, we'll be one of those projects that
Niall referred to, with community issues because they didn't
pass through the Incubator. That's why I thought it was a good
occasion to ask for suggestions. The new HttpComponents as well
as the old HttpClient we maintain are being used by Apache
projects, so coming into the Incubator is not an option for
HttpComponents.

I'm sorry if this is getting off-topic. I'm just trying to
tap into the Incubator's experience in community building.

cheers,
  Roland

[1] http://wiki.apache.org/jakarta/JakartaBoardReport-current


---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org

Re: Jakarta [was: Effects on corporate backing withdrawals [was: Incubator Proposal: Pig]]

[Niclas Hedhman <ni...@hedhman.org>]

On Saturday 29 September 2007 00:03, Roland Weber wrote:
> Staying at Jakarta will buy some time, but won't last forever.
>
> If you have ideas on what to do with these small but active
> projects, please come over to general@jakarta and share your
> thoughts.

Can't Jakarta just be revitalized as a home for small, but mature and stable 
projects??

We must allow for stable, near perfect codebases, to "just exist" without 
further development, i.e. no dedicated community.


Cheers
Niclas

---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org

Jakarta [was: Effects on corporate backing withdrawals [was: Incubator Proposal: Pig]]

[Roland Weber <os...@dubioso.net>]

Niall Pemberton wrote:
> Theres more of an issue IMO with projects that don't come thru the
> incubator, since they don't have to meet the Incubator's stringent
> graduation requirement. As an example - Tapestry was pushed out to a
> TLP from Jakarta,[...]

Jakarta is disintegrating. All big projects have gone TLP,
there are two or three more that might just make it. The rest
is too inactive, because of maturity or disinterest, to stand
on their own. There have been discussions every few months on
how to revive inactive projects, and sending them back into
the incubator was one of the options. Not that it matters much,
I don't remember any that picked up enough interest.

Now the projects that are still active but have only a small
developer community - too small for Incubator standards for
sure - are caught between a rock and a hard place. There are
users out there, and the code has seen many official releases.
Going into the Incubator and making unofficial "incubating"
releases from there is not a preferred option.
Going TLP with just enough PMCs to collect three binding votes
during holiday season creates TLPs with a very high dependency
on very few people. Projects which are an issue.
Staying at Jakarta will buy some time, but won't last forever.

If you have ideas on what to do with these small but active
projects, please come over to general@jakarta and share your
thoughts. The two examples I have in mind are HttpComponents
and JMeter, but there may be others:

http://jakarta.apache.org/httpcomponents/index.html
http://jakarta.apache.org/jmeter/index.html

cheers,
  Roland


---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org

Re: Effects on corporate backing withdrawals [was: Incubator Proposal: Pig]

[Niclas Hedhman <ni...@hedhman.org>]

On Thursday 04 October 2007 04:56, Robert Burrell Donkin wrote:
> > http://agileskills2.org/blog/2007/09/my_thoughts_on_the_differences.html
>
> i have the impression that howard is one of those people who dominates
> by his charisma and energy rather than any abuse of the process

Yes, and I hope he has wisdom enough to mitigate confrontation should it 
arise.

Cheers
-- 
Niclas Hedhman, Software Developer

I  live here; http://tinyurl.com/2qq9er
I  work here; http://tinyurl.com/2ymelc
I relax here; http://tinyurl.com/2cgsug

---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org

Re: Effects on corporate backing withdrawals [was: Incubator Proposal: Pig]

[Robert Burrell Donkin <ro...@gmail.com>]

On 9/26/07, Niall Pemberton <ni...@gmail.com> wrote:
> On 9/25/07, Guillaume Nodet <gn...@gmail.com> wrote:
> > One of the purpose of the incubator is to ensure that there is a
> > sustainable developer community, so I don't see failure of incubating
> > projects as a real problem.
>
> +1
>
> Theres more of an issue IMO with projects that don't come thru the
> incubator, since they don't have to meet the Incubator's stringent
> graduation requirement. As an example - Tapestry was pushed out to a
> TLP from Jakarta, but the following blog from a Tapestry committer
> doesn't make good reading from a community PoV:
>
> http://agileskills2.org/blog/2007/09/my_thoughts_on_the_differences.html

i have the impression that howard is one of those people who dominates
by his charisma and energy rather than any abuse of the process

- robert

---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org

Re: Effects on corporate backing withdrawals [was: Incubator Proposal: Pig]

[Niall Pemberton <ni...@gmail.com>]

On 9/25/07, Guillaume Nodet <gn...@gmail.com> wrote:
> One of the purpose of the incubator is to ensure that there is a
> sustainable developer community, so I don't see failure of incubating
> projects as a real problem.

+1

Theres more of an issue IMO with projects that don't come thru the
incubator, since they don't have to meet the Incubator's stringent
graduation requirement. As an example - Tapestry was pushed out to a
TLP from Jakarta, but the following blog from a Tapestry committer
doesn't make good reading from a community PoV:

http://agileskills2.org/blog/2007/09/my_thoughts_on_the_differences.html

Niall

> On 9/25/07, Noel J. Bergman <no...@devtech.com> wrote:
> > Dims wrote:
> > > Niclas Hedhman asked:
> > > > Do we have any examples where corporate backing has been withdrawn, and
> > how
> > > > the project was affected, whether inside or outside ASF?
> > > TSIK -> Verisign folks lost interest, community did not form, project
> > shelved.
> >
> > Plus Kabuki, Heraldry, and possibly Lokahi.  We still hope to save the
> > latter, as there is consistently a lot of user interest, but the developer
> > input has dwindled, and we need developers!
> >
> >         --- Noel

---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org

RE: Effects on corporate backing withdrawals [was: Incubator Proposal: Pig]

["Noel J. Bergman" <no...@devtech.com>]

> One of the purpose of the incubator is to ensure that there is a
> sustainable developer community, so I don't see failure of
> incubating projects as a real problem.

Neither do I.  It merely underscores the need to make sure that there is
such a sustainable community.  But Niclas did ask for examples.  :-)

	--- Noel



---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org

Re: Effects on corporate backing withdrawals [was: Incubator Proposal: Pig]

[Guillaume Nodet <gn...@gmail.com>]

One of the purpose of the incubator is to ensure that there is a
sustainable developer community, so I don't see failure of incubating
projects as a real problem.

On 9/25/07, Noel J. Bergman <no...@devtech.com> wrote:
> Dims wrote:
> > Niclas Hedhman asked:
> > > Do we have any examples where corporate backing has been withdrawn, and
> how
> > > the project was affected, whether inside or outside ASF?
> > TSIK -> Verisign folks lost interest, community did not form, project
> shelved.
>
> Plus Kabuki, Heraldry, and possibly Lokahi.  We still hope to save the
> latter, as there is consistently a lot of user interest, but the developer
> input has dwindled, and we need developers!
>
>         --- Noel
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
>
>


-- 
Cheers,
Guillaume Nodet
------------------------
Blog: http://gnodet.blogspot.com/

---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org

Re: Effects on corporate backing withdrawals [was: Incubator Proposal: Pig]

[Steve Loughran <st...@apache.org>]

Noel J. Bergman wrote:
> Dims wrote:
>> Niclas Hedhman asked:
>>> Do we have any examples where corporate backing has been withdrawn, and
> how
>>> the project was affected, whether inside or outside ASF?
>> TSIK -> Verisign folks lost interest, community did not form, project
> shelved.
> 
> Plus Kabuki, Heraldry, and possibly Lokahi.  We still hope to save the
> latter, as there is consistently a lot of user interest, but the developer
> input has dwindled, and we need developers!

ahh, now I understand why you've been trying to get me on the mail list :)

The biggest departure I know of was not in the incubator, it was the 
implementation of bits of WS-RF that HP was doing under WS;, what was 
it, Apache Muse? suddenly corporate priorities got changed and all FTEs 
got reassigned to something else. It just sat there for a while before 
IBM took up the challenge with a port to Axis2.

similarly, there was a bit of stutter in Axis1 when the IBM team 
suddenly dropped of the net. There was lots of other active developers, 
but there were whole swathes of things like Java-to-WSDL code that came 
from IBM and which the others suddenly needed to learn, because till now 
that area had been well covered  by the IBM folk, but not oustandingly 
well documented. At least they provided lots of tests, which does make 
it easier for others to take on the maintenance task -it reduces the 
amount of damage done while learning.

It seems to me then, that the problem is more than just in-incubator.

-steve

---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org

RE: Effects on corporate backing withdrawals [was: Incubator Proposal: Pig]

["Noel J. Bergman" <no...@devtech.com>]

Dims wrote:
> Niclas Hedhman asked:
> > Do we have any examples where corporate backing has been withdrawn, and
how
> > the project was affected, whether inside or outside ASF?
> TSIK -> Verisign folks lost interest, community did not form, project
shelved.

Plus Kabuki, Heraldry, and possibly Lokahi.  We still hope to save the
latter, as there is consistently a lot of user interest, but the developer
input has dwindled, and we need developers!

	--- Noel



---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org

Re: Effects on corporate backing withdrawals [was: Incubator Proposal: Pig]

[Davanum Srinivas <da...@gmail.com>]

TSIK -> Verisign folks lost interest, community did not form, project shelved.

-- dims

On 9/25/07, Niclas Hedhman <ni...@hedhman.org> wrote:
> On Tuesday 25 September 2007 04:18, Robert Burrell Donkin wrote:
>
> > this is probably just an indication that it's time to starting thinking...
>
> +1 to all said.
>
>
> Do we have any examples where corporate backing has been withdrawn, and how
> the project was affected, whether inside or outside ASF?
>
> It would be interesting to hear about... Not that this has much to do about
> the current proposal Pig...
>
>
> Cheers
> --
> Niclas Hedhman, Software Developer
>
> I  live here; http://tinyurl.com/2qq9er
> I  work here; http://tinyurl.com/2ymelc
> I relax here; http://tinyurl.com/2cgsug
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
>
>


-- 
Davanum Srinivas :: http://davanum.wordpress.com

---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org

Effects on corporate backing withdrawals [was: Incubator Proposal: Pig]

[Niclas Hedhman <ni...@hedhman.org>]

On Tuesday 25 September 2007 04:18, Robert Burrell Donkin wrote:

> this is probably just an indication that it's time to starting thinking...

+1 to all said.


Do we have any examples where corporate backing has been withdrawn, and how 
the project was affected, whether inside or outside ASF?

It would be interesting to hear about... Not that this has much to do about 
the current proposal Pig...


Cheers
-- 
Niclas Hedhman, Software Developer

I  live here; http://tinyurl.com/2qq9er
I  work here; http://tinyurl.com/2ymelc
I relax here; http://tinyurl.com/2cgsug

---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org

Re: Incubator Proposal: Pig

[Robert Burrell Donkin <ro...@gmail.com>]

On 9/24/07, Doug Cutting <cu...@apache.org> wrote:
> Jim Jagielski wrote:
> > In other words, if the corporate support
> > of the project or podling went away, would they stop
> > developing and working on the codebase because they,
> > after all, had no allegiance in the code at all? Were
> > they, in effect, coders-for-hire?
>
> Yes, this is a known risk, perhaps the largest risk for Pig's
> incubation.  We must develop a diverse developer community so the
> project can survive the departure of any employer or individual.  Yahoo!
> is aware of this, and seeks non-Yahoo! developers.  This is the primary
> motivation to incubate.  If Yahoo! wished to develop Pig alone, then it
> could simply continue to distribute it under a BSD license.

true

any corporation can pick a license, host a public repository, build
brand awareness and create a project where the source is open but the
development is closed. the reason to approach apache is that we've had
a reasonable track record in the difficult task of building healthy
and open communties.

> And, yes, some developers may stop contributing when they change
> employment.  But, in my experience, many others will be hired
> specifically for their experience with an Apache project.  I see more
> careers built around Apache experience than short-term coders-for-hire.
>   Today Yahoo! is hiring folks to work on Pig.  Soon, hopefully, other
> companies will do so as well, if they're not already.  There's no shame
> in being paid to work on Apache projects, is there?

of course not

but there has been a definite shift over the years. committers paid to
work full time on a project (as opposed to being allowed to work on
the project in work time) are now more common. it seems comfortable
when a long time contributor is hired to work full time on a project.
now that so many volunteers are now employed to code open source, it
is perhaps inevitable that corporations will look to hire from outside
this pool. i'm not sure (though) that we've had enough time to digest
this phenomenum to really understand it's long term effects on
community health.

this is probably just an indication that it's time to starting thinking...

- robert

---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org

Re: Incubator Proposal: Pig

[Doug Cutting <cu...@apache.org>]

Jim Jagielski wrote:
> In other words, if the corporate support
> of the project or podling went away, would they stop
> developing and working on the codebase because they,
> after all, had no allegiance in the code at all? Were
> they, in effect, coders-for-hire?

Yes, this is a known risk, perhaps the largest risk for Pig's 
incubation.  We must develop a diverse developer community so the 
project can survive the departure of any employer or individual.  Yahoo! 
is aware of this, and seeks non-Yahoo! developers.  This is the primary 
motivation to incubate.  If Yahoo! wished to develop Pig alone, then it 
could simply continue to distribute it under a BSD license.

And, yes, some developers may stop contributing when they change 
employment.  But, in my experience, many others will be hired 
specifically for their experience with an Apache project.  I see more 
careers built around Apache experience than short-term coders-for-hire. 
  Today Yahoo! is hiring folks to work on Pig.  Soon, hopefully, other 
companies will do so as well, if they're not already.  There's no shame 
in being paid to work on Apache projects, is there?

Doug

---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org

Re: Incubator Proposal: Pig

[Jim Jagielski <ji...@jaguNET.com>]

I am a +1 on the proposal, but I am still unclear, at
this point, how Y! is going to align the open source aspects
of Pig with their hiring push for Pig developers as per:

     http://research.yahoo.com/project/pig

I guess this is more a general concern about the changing dynamics.
First, of course, there were people developing code as volunteers
because it was fun or because the were directly influenced
by the code itself (after all, this is where httpd got its
start). Then we were able to move into the sweet spot where
not only did we have "true" volunteer developers but also
developers who got paid to continue developing... Now we
seem to be getting into the realm where a condition of
their employment is to "code ASF stuff"... The main concern
is whether they are developing because they want to, or
they have to. In other words, if the corporate support
of the project or podling went away, would they stop
developing and working on the codebase because they,
after all, had no allegiance in the code at all? Were
they, in effect, coders-for-hire?

Certainly Pig is not unique in this. There are other Incubator
podlings soo much in line with a major corporate entity
that if the entity decided today that Apache Foo didn't make
corporate sense, that 95% of "their" developers would never be
seen or heard from again...

I, of course, trust the Mentors of projects to work through
these issues, and a condition of graduation after all is that
the community itself is diverse enough and strong enough to
survive such transitions. But I see this becoming harder as
time goes on as well as much, much more common.

---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org

Re: Incubator Proposal: Pig

[Sylvain Wallez <sy...@apache.org>]

Olga Natkovich wrote:
> Hi,
>  
> Yahoo! research and development teams have developed a proposal below. The
> proposal is also available on wiki at
> http://wiki.apache.org/incubator/PigProposal.
> We would like to ask that the ASF consider forming a podling according to
> the proposal.
>   

High-level tools like Pig are definitely needed to ease the adoption
non-traditional storage/database systems like Hadoop, both by the
developer communities and their managers.

I was pretty excited when the first opensource version was released a
few months ago, so a big +1 for this proposal.

I'd be happy to be a mentor too.

Sylvain

-- 
Sylvain Wallez - http://bluxte.net


---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org

Re: Incubator Proposal: Pig

[Doug Cutting <cu...@apache.org>]

Bertrand Delacretaz wrote:
> +1 to the proposal, and I'd be happy to help as a mentor.

Sylvain Wallez wrote:
> I'd be happy to be a mentor too.

Thanks!  Please feel free to add yourself to the proposal.

Doug

---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org

Re: Incubator Proposal: Pig

[Petar Tahchiev <pa...@gmail.com>]

+1 from me, too. It looks very promising. :-)



On 9/21/07, Bertrand Delacretaz <bd...@apache.org> wrote:
>
> On 9/18/07, Olga Natkovich <ol...@yahoo-inc.com> wrote:
>
> > ...We would like to ask that the ASF consider forming a podling
> according to
> > the proposal....
>
> +1 to the proposal, and I'd be happy to help as a mentor.
>
> -Bertrand
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
>
>


-- 
Regards, Petar!
Karlovo, Bulgaria.

EOOXML objections
http://www.grokdoc.net/index.php/EOOXML_objections

Public PGP Key at:
http://keyserver.linux.it/pks/lookup?op=get&search=0x1A15B53B761500F9
Key Fingerprint: AA16 8004 AADD 9C76 EF5B  4210 1A15 B53B 7615 00F9

Re: Incubator Proposal: Pig

[Bertrand Delacretaz <bd...@apache.org>]

On 9/18/07, Olga Natkovich <ol...@yahoo-inc.com> wrote:

> ...We would like to ask that the ASF consider forming a podling according to
> the proposal....

+1 to the proposal, and I'd be happy to help as a mentor.

-Bertrand

---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org

Re: Incubator Proposal: Pig

["William A. Rowe, Jr." <wr...@rowe-clan.net>]

Doug Cutting wrote:
> Niclas Hedhman wrote:
>> a) The name "Pig" is somewhat provocative (not kosher/halal) and I
>> would like to hear the rationale behind the name, and whether there
>> are any willingness to look for another name.
> 
> It is not meant to be provocative.  It is named after the animal and is
> not an acronym.  It also provides a convenient name for the query
> language, "Pig Latin".  I'm sure that the developers would have some
> regret about changing the name, but if it were truly determined to be
> offensive then I believe there would be a willingness to change it.

Not offensive at all, nobody has to eat the source code.

That said, is Igpay available as a project name and would the project
entertain it?  Now that would be a fun name :)

Bill


---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org

Re: Incubator Proposal: Pig

[Carl Trieloff <cc...@redhat.com>]

We could add rules indefinitely  to make just about anyone not usable... 
I don't have any issues
with Pig.

Carl.

Yoav Shapira wrote:
> Hey,
>
> On 9/24/07, Doug Cutting <cu...@apache.org> wrote:
>   
>> Niclas Hedhman wrote:
>>     
>>> a) The name "Pig" is somewhat provocative (not kosher/halal) and I would like
>>> to hear the rationale behind the name, and whether there are any willingness
>>> to look for another name.
>>>       
>> It is not meant to be provocative.  It is named after the animal and is
>> not an acronym.  It also provides a convenient name for the query
>> language, "Pig Latin".  I'm sure that the developers would have some
>> regret about changing the name, but if it were truly determined to be
>> offensive then I believe there would be a willingness to change it.
>>     
>
> As another Jewish person on the list, I don't find it the least bit
> provocative.  It's a freakin' animal, that's all ;)
>
> In fact, I'm a big fan of more fun names around here.  Succubus,
> Imperius, Pig, all are great in my book.  I'm tried of four-letter
> acronyms and packaging that's been through N layers of marketing
> analysis ;)
>
> Let's not do this: http://video.google.com/videoplay?docid=36099539665548298
>
> Yoav
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
>
>   

Re: Incubator Proposal: Pig

[Yoav Shapira <yo...@apache.org>]

Hey,

On 9/24/07, Doug Cutting <cu...@apache.org> wrote:
> Niclas Hedhman wrote:
> > a) The name "Pig" is somewhat provocative (not kosher/halal) and I would like
> > to hear the rationale behind the name, and whether there are any willingness
> > to look for another name.
>
> It is not meant to be provocative.  It is named after the animal and is
> not an acronym.  It also provides a convenient name for the query
> language, "Pig Latin".  I'm sure that the developers would have some
> regret about changing the name, but if it were truly determined to be
> offensive then I believe there would be a willingness to change it.

As another Jewish person on the list, I don't find it the least bit
provocative.  It's a freakin' animal, that's all ;)

In fact, I'm a big fan of more fun names around here.  Succubus,
Imperius, Pig, all are great in my book.  I'm tried of four-letter
acronyms and packaging that's been through N layers of marketing
analysis ;)

Let's not do this: http://video.google.com/videoplay?docid=36099539665548298

Yoav

---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org

Re: Incubator Proposal: Pig

[Benjamin Reed <br...@yahoo-inc.com>]

Actually we would really like to modify Ruby and Python to support Pig Latin 
as part of the language: Ubyray and Ythonpay. We have avoided creating our 
own crippled scripting language, there are just too many in the world, and 
instead hope to take an existing language and and embedded Pig Latin into it 
as a first class part of the language.

ben

On Monday 24 September 2007, Craig L Russell wrote:
> Speaking just for myself, I find the name unusual but not offensive
> or even provocative. The fact that you wouldn't eat an animal doesn't
> mean you deny its existence...
>
> Of course, I thought the language was officially called igpay atinlay.
>
> Craig
>
> On Sep 24, 2007, at 10:15 AM, Doug Cutting wrote:
> > Niclas Hedhman wrote:
> >> a) The name "Pig" is somewhat provocative (not kosher/halal) and I
> >> would like to hear the rationale behind the name, and whether
> >> there are any willingness to look for another name.
> >
> > It is not meant to be provocative.  It is named after the animal
> > and is not an acronym.  It also provides a convenient name for the
> > query language, "Pig Latin".  I'm sure that the developers would
> > have some regret about changing the name, but if it were truly
> > determined to be offensive then I believe there would be a
> > willingness to change it.
> >
> > Doug
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> > For additional commands, e-mail: general-help@incubator.apache.org
>
> Craig Russell
> Architect, Sun Java Enterprise System http://java.sun.com/products/jdo
> 408 276-5638 mailto:Craig.Russell@sun.com
> P.S. A good JDO? O, Gasp!



---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org

Re: Incubator Proposal: Pig

[Eelco Hillenius <ee...@gmail.com>]

On 9/24/07, Craig L Russell <Cr...@sun.com> wrote:
> Speaking just for myself, I find the name unusual but not offensive
> or even provocative. The fact that you wouldn't eat an animal doesn't
> mean you deny its existence...

And they make good pets in many cultures and are generally
acknowledged to be smart. I thought 'pig' sounds a bit weird for an
OSS project, but certainly not offensive.

Eelco

---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org

Re: Incubator Proposal: Pig

[Andrus Adamchik <an...@objectstyle.org>]

On Sep 24, 2007, at 8:22 PM, Craig L Russell wrote:
> Speaking just for myself, I find the name unusual but not offensive  
> or even provocative. The fact that you wouldn't eat an animal  
> doesn't mean you deny its existence...

+1

I find it a fun name, and the one unlikely to infringe on the  
existing software trademarks.

Andrus

---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org

Re: Incubator Proposal: Pig

[Craig L Russell <Cr...@Sun.COM>]

Speaking just for myself, I find the name unusual but not offensive  
or even provocative. The fact that you wouldn't eat an animal doesn't  
mean you deny its existence...

Of course, I thought the language was officially called igpay atinlay.

Craig

On Sep 24, 2007, at 10:15 AM, Doug Cutting wrote:

> Niclas Hedhman wrote:
>> a) The name "Pig" is somewhat provocative (not kosher/halal) and I  
>> would like to hear the rationale behind the name, and whether  
>> there are any willingness to look for another name.
>
> It is not meant to be provocative.  It is named after the animal  
> and is not an acronym.  It also provides a convenient name for the  
> query language, "Pig Latin".  I'm sure that the developers would  
> have some regret about changing the name, but if it were truly  
> determined to be offensive then I believe there would be a  
> willingness to change it.
>
> Doug
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
>

Craig Russell
Architect, Sun Java Enterprise System http://java.sun.com/products/jdo
408 276-5638 mailto:Craig.Russell@sun.com
P.S. A good JDO? O, Gasp!

Re: Incubator Proposal: Pig

[Greg Stein <gs...@gmail.com>]

I don't find the name provocative either, but the connotations are a  
bit weird :-P

On Sep 24, 2007, at 10:39 AM, "Noel J. Bergman" <no...@devtech.com>  
wrote:

> Doug Cutting wrote:
>
>> Niclas Hedhman wrote:
>>> a) The name "Pig" is somewhat provocative (not kosher/halal) and I
>>> would like to hear the rationale behind the name, and whether there
>>> are any willingness to look for another name.
>
>> It is not meant to be provocative.  It is named after the animal  
>> and is
>> not an acronym.
>
> As one of the residents around here who keeps Kosher, I don't find it
> offensive in that regard.  I do, however, consider it an odd choice  
> for a
> name, since refering to some software as a Pig is never intended to  
> be a
> good thing.  :-)  I certainly don't want anyone calling my work  
> produce a
> pig!  :-)
>
> Just pointing it out, not basing a vote on the name.  :-)
>
>    --- Noel
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org

RE: Incubator Proposal: Pig

["Noel J. Bergman" <no...@devtech.com>]

Doug Cutting wrote:

> Niclas Hedhman wrote:
> > a) The name "Pig" is somewhat provocative (not kosher/halal) and I
> > would like to hear the rationale behind the name, and whether there
> > are any willingness to look for another name.

> It is not meant to be provocative.  It is named after the animal and is
> not an acronym.

As one of the residents around here who keeps Kosher, I don't find it
offensive in that regard.  I do, however, consider it an odd choice for a
name, since refering to some software as a Pig is never intended to be a
good thing.  :-)  I certainly don't want anyone calling my work produce a
pig!  :-)

Just pointing it out, not basing a vote on the name.  :-)

	--- Noel



---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org

Re: Incubator Proposal: Pig

[Doug Cutting <cu...@apache.org>]

Niclas Hedhman wrote:
> a) The name "Pig" is somewhat provocative (not kosher/halal) and I would like 
> to hear the rationale behind the name, and whether there are any willingness 
> to look for another name.

It is not meant to be provocative.  It is named after the animal and is 
not an acronym.  It also provides a convenient name for the query 
language, "Pig Latin".  I'm sure that the developers would have some 
regret about changing the name, but if it were truly determined to be 
offensive then I believe there would be a willingness to change it.

Doug

---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org

Re: Incubator Proposal: Pig

[Thilo Goetz <tw...@gmx.de>]

Niclas Hedhman wrote:
[...]
> 
> b) I can't say that I understand the technical merits of the proposal, and 
> just see the headline "analyzing large data sets". And I would like to know 
> the relationship with UIMA's statement "... analyze large volumes of 
> unstructured information..." and hear whether there are overlap, synergies 
> and/or collaboration in view.

Niclas,

I'm not 100% clear on where there could be synergies between
Pig and UIMA.  Map/reduce is a natural distribution
strategy for UIMA, so executing UIMA programs on top of Hadoop
seems natural.  Maybe Pig can help with that and make it easier
somehow.  However, that is not clear to me from the proposal
at this time.

At the same time, I don't really think there is any overlap.
Pig is concerned with computation in a distributed environment,
while UIMA is agnostic in that respect.  On the other hand,
UIMA offers a component model to develop analysis modules and
combine them into processing chains (with an emphasis on reuse).
I do not see from the proposal that Pig is in the business of
defining a component model.

So synergies probably yes, no overlap as far as I can see.

--Thilo


---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org

Re: Incubator Proposal: Pig

[Niclas Hedhman <ni...@hedhman.org>]

On Wednesday 19 September 2007 03:52, Olga Natkovich wrote:
> We would like to ask that the ASF consider forming a podling according to
> the proposal.

+1, but I also got a couple of observations.

a) The name "Pig" is somewhat provocative (not kosher/halal) and I would like 
to hear the rationale behind the name, and whether there are any willingness 
to look for another name.

b) I can't say that I understand the technical merits of the proposal, and 
just see the headline "analyzing large data sets". And I would like to know 
the relationship with UIMA's statement "... analyze large volumes of 
unstructured information..." and hear whether there are overlap, synergies 
and/or collaboration in view.


Cheers
-- 
Niclas Hedhman, Software Developer

I  live here; http://tinyurl.com/2qq9er
I  work here; http://tinyurl.com/2ymelc
I relax here; http://tinyurl.com/2cgsug

---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org

Re: Incubator Proposal: Pig

[Brian McCallister <br...@skife.org>]

+1 -- I'd offer to help as much as I can, but I know how little that  
is right now :-(

Definitely support (and will probably use at least ;-)

-Brian

On Sep 18, 2007, at 12:52 PM, Olga Natkovich wrote:

> Hi,
>
> Yahoo! research and development teams have developed a proposal  
> below. The
> proposal is also available on wiki at
> <http://wiki.apache.org/incubator/PigProposal>
> http://wiki.apache.org/incubator/PigProposal.
> We would like to ask that the ASF consider forming a podling  
> according to
> the proposal.
>
> Thanks,
>
> Olga Natkovich
>  <ma...@yahoo-inc.com> olgan@yahoo-inc.com
>
> ---------------------------------------------------------------------- 
> ------
> ---------
>
> = Pig Open Source Proposal =
>
> == Abstract ==
>
> Pig is a platform for analyzing large data sets.
>
> == Proposal ==
>
> The Pig project consists of high-level languages for expressing data
> analysis programs, coupled with infrastructure for evaluating these
> programs. The salient property of Pig programs is that their  
> structure is
> amenable to substantial parallelization, which in turns enables  
> them to
> handle very large data sets.
>
> At the present time, Pig's infrastructure layer consists of a  
> compiler that
> produces sequences of Map-Reduce programs, for which large-scale  
> parallel
> implementations already exist (e.g., the Hadoop subproject). Pig's  
> language
> layer currently consists of a textual language called Pig Latin,  
> which has
> the following key properties:
>
>  1. ''Ease of programming''. It is trivial to achieve parallel  
> execution of
> simple, "embarrassingly parallel" data analysis tasks. Complex tasks
> comprised of multiple interrelated data transformations are explicitly
> encoded as data flow sequences, making them easy to write,  
> understand, and
> maintain.
>  2. ''Optimization opportunities''. The way in which tasks are encoded
> permits the system to optimize their execution automatically,  
> allowing the
> user to focus on semantics rather than efficiency.
>  3. ''Extensibility''. Users can create their own functions to do
> special-purpose processing.
>
> == Background ==
>
> Pig started as a research project at Yahoo! in May of 2006 to  
> combine ideas
> in parallel databases and distributed computing. The first internal  
> release
> took place in July 2006. The first release was a simple front-end  
> to the
> Hadoop Map/Reduce framework. The following releases added new  
> features and
> evolved the language based on user feedback. In July 2007, pig was  
> taken
> over by a development team and the first production version is due  
> to be
> released on 9/28/07.
>
> Since its inception, we had observed a steady growth of the user  
> community
> within Yahoo!.  In April 2007, Pig was released under a BSD-type  
> license.
> Several external parties are using this version and have expressed  
> interest
> in collaborating on its development.
>
> == Rationale ==
>
> In an information-centric world, innovation is driven by ad-hoc  
> analysis of
> large data sets. For example, search engine companies routinely  
> deploy and
> refine services based on analyzing the recorded behavior of users,
> publishers, and advertisers. The rate of innovation depends on the
> efficiency with which data can be
> analyzed.
>
> To analyze large data sets efficiently, one needs parallelism. The  
> cheapest
> and most scalable form of parallelism is cluster computing.  
> Unfortunately,
> programming for a cluster computing environment is difficult and
> time-consuming. Pig makes it easy to harness the power of cluster  
> computing
> for ad-hoc data analysis.
>
> While other language exist that try to achieve the same goals, we  
> believe
> that Pig provides more flexibility and gives more control to the  
> end user.
>
> SQL typically requires (1) importing data from a user's preferred  
> format
> into a database system's internal format (2) well-structured,  
> normalized
> data with a declared schema, and (3) programs expressed in declarative
> SELECT-FROM-WHERE blocks. In contrast, Pig Latin facilitates (1)
> interoperability, i.e. data may be read/written in a format  
> accepted by
> other applications such as text editors or graph generators (2)  
> flexibility,
> i.e. data may be loosely structured or have structure that is
> defined operationally, and (3) adoption by programmers who find  
> procedural
> programming more natural than declarative programming.
>
> Sawzall is a scripting language used at Google on top of Map-Reduce. A
> sawzall program has a fairly rigid structure consisting of a  
> filtering phase
> (the map step) followed by an aggregation phase (the reduce step).
> Furthermore, only the filtering phase can be written by the user,  
> and only a
> pre-built set of aggregations are available (new ones are non- 
> trivial to
> add). While Pig Latin has similar higher level primitives like  
> filtering and
> aggregation, an arbitrary number of them can be flexibly chained  
> together in
> a Pig Latin program, and all primitives can use user-defined  
> functions with
> equal ease. Further, Pig Latin has additional primitives such as  
> cogrouping,
> that allow operations such as joins (which require multiple  
> programs in
> Sawzall) to be written in a single line in Pig Latin. Further, Pig  
> Latin is
> designed
> to be embedded into other languages, and can use functions written  
> in other
> languages. Thus, in contrast to Sawzall, it directly caters to a large
> community of developers without having to make them learn an  
> entirely new
> programming language.
>
> == Current Status ==
>
> === Meritocracy ===
>
> Pig was started as a project that was developed by Yahoo! research  
> team.
> Recently we have added a development team that works in harmony  
> with the
> research team with both teams actively and successfully  
> contributing to the
> project. We are planning to create the environment that encourages
> meritocracy and is consistent with the meritocracy principles of  
> Apache.
> Within the team we have people actively participating in the Hadoop
> subproject.
>
> === Community ===
>
> Pig has an active user community within Yahoo! that has been steadily
> growing. Pig also attracted external users since its release under a
> BSD-type license.  Several external parties are using the product  
> and have
> expressed interest in collaborating on its development.
>
> Also, since the current version of Pig is built on top of the  
> Hadoop we
> believe that we will be able to quickly extend our community by  
> attracting
> both the Hadoop users and developers to the project.
>
> === Core Developers ===
>
> Our contributors come from both research and development world and  
> most have
> background in database internals and large scale distributed systems.
>
> === Alignment ===
>
> Yahoo! seeks to develop Pig collaboratively with others, not to  
> control and
> maintain it independently.  Apache offers the best legal and social
> framework for such community-based software development.
>
> Also, the current version of Pig runs on top of the Hadoop's Map- 
> Reduce
> infrastructure which is part of Apache. We believe there would be a  
> lot of
> synergy between the projects both in terms of users and developers.
>
> == Known Risks ==
> === Orphaned products ===
>
> All current contributors are part of Yahoo which is a major player  
> in the
> space and is committed to grid computing. Also we expect high  
> degree of
> synergy with Hadoop subproject.
>
> === Inexperience with Open Source ===
>
> Two of the committers have extensive experience with open source  
> and Apache.
> The rest are new to open source and will be guided through the  
> process by
> the team members with experience.
>
> === Homogenous Developers ===
>
> The current list of committers is confined to Yahoo employees. Our  
> plan is
> to recruit more committers once the project gets on the way.
>
> === Reliance on Salaried Developers ===
>
> Currently, all contributors are Yahoo employees. By extending the
> development community we are hoping to mitigate this risk.
>
> === Relationships with Other Apache Products ===
>
> Pig is built on top of Hadoop and we expect deep collaboration with  
> Hadoop
> subproject.
>
> === An Excessive Fascination with the Apache Brand ===
>
> Yahoo already have a strong brand and is not interested in Apache  
> as a way
> to gain visibility. Yahoo! seeks to develop Pig collaboratively  
> with others,
> not to control and maintain it independently.  Apache offers the  
> best legal
> and social framework for such community-based software development.
>
> == Documentation ==
>
> http://research.yahoo.com/project/pig
>
> == Initial Source ==
>
> The initial source will be donated by Yahoo Inc. The donating  
> company will
> contribute the initial code base once the proposal is accepted and  
> necessary
> infrastructure has been set up.
>
> == External Dependencies ==
>
>  1. bzip2:  <http://www.kohsuke.org/bzip2/:Apache>
> http://www.kohsuke.org/bzip2/:Apache license
>  2. javacc:  <https://javacc.dev.java.net/:BSD>
> https://javacc.dev.java.net/:BSD license
>  3. hadoop:  <http://lucene.apache.org/hadoop/:Apache>
> http://lucene.apache.org/hadoop/:Apache license
>  4. log4j:  <http://logging.apache.org/log4j/>
> http://logging.apache.org/log4j/: Apache license
>  5. jsch:  <http://www.jcraft.com/jsch> http://www.jcraft.com/jsch:  
> BSD
> style license:  <http://www.jcraft.com/jsch/LICENSE.txt>
> http://www.jcraft.com/jsch/LICENSE.txt
>
> == Required Resources ==
> == Mailing lists ==
>
> We would need the following mailing lists
>  1. pig-private (with moderated subscriptions)
>  2. pig-dev
>  3. pig-commits
>  4. pig-user
>
> === Subversion Directory ===
>
> https://svn.apache.org/repos/asf/incubator/pig
>
> === Issue Tracking ===
>
> JIRA PIG (PIG)
>
> == Initial Committers ==
>
>  1. Nigel Daley ( <ma...@yahoo-inc.com> ndaley@yahoo-inc.com)
>  2. Alan Gates ( <ma...@yahoo-inc.com> gates@yahoo-inc.com)
>  3. Olga Natkovich ( <ma...@yahoo-inc.com> olgan@yahoo-inc.com)
>  4. Chris Olston ( <ma...@yahoo-inc.com> olston@yahoo-inc.com)
>  5. Owen O'Malley ( <ma...@yahoo-inc.com> oom@yahoo-inc.com)
>  6. Ben Reed ( <ma...@yahoo-inc.com> breed@yahoo-inc.com)
>  7. Utkarsh Srivastava ( <ma...@yahoo-inc.com>
> utkarsh@yahoo-inc.com)
>
> == Affiliation ==
>
> All initial committers are affiliated with Yahoo!
>
> == Sponsors ==
>
> === Champion ===
>
> Doug Cutting
>
> === Nominated Mentors ===
>
> Doug Cutting
>
> === Sponsoring Entity ===
>
> Incubator
>
>


---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org

Re: Incubator Proposal: Pig

[Doug Cutting <cu...@apache.org>]

Olga Natkovich wrote:
> http://wiki.apache.org/incubator/PigProposal.
> We would like to ask that the ASF consider forming a podling according to
> the proposal.

Thanks for all the comments.  I've seen no issues raised that should 
block Pig from entering incubation.  Unless something arises before 
then, I will call a formal vote on the proposal tomorrow.

Doug

---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org

Re: Incubator Proposal: Pig

[Torsten Curdt <tc...@apache.org>]

Done!

On 20.09.2007, at 19:46, Doug Cutting wrote:

> Torsten Curdt wrote:
>> +1
>> Actually I would also be interested in stepping up as a mentor.
>
> Thanks, that'd be great!
>
> Please add yourself to the proposal in the wiki.
>
> Doug
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
>


---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org

Re: Incubator Proposal: Pig

[Doug Cutting <cu...@apache.org>]

Torsten Curdt wrote:
> +1
> 
> Actually I would also be interested in stepping up as a mentor.

Thanks, that'd be great!

Please add yourself to the proposal in the wiki.

Doug

---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org

Re: Incubator Proposal: Pig

[Torsten Curdt <tc...@apache.org>]

On 20.09.2007, at 19:06, Robert Burrell Donkin wrote:

> On 9/20/07, Leo Simons <ma...@leosimons.com> wrote:
>> On Sep 18, 2007, at 9:52 PM, Olga Natkovich wrote:
>>> Yahoo! research and development teams have developed a proposal
>>> below. The
>>> proposal is also available on wiki at
>>> <http://wiki.apache.org/incubator/PigProposal>
>>> http://wiki.apache.org/incubator/PigProposal.
>>> We would like to ask that the ASF consider forming a podling
>>> according to
>>> the proposal.
>>> ...
>>> Pig is a platform for analyzing large data sets.
>>
>> +1, looks cool!
>>
>> ...seems like your biggest challenge here is attracting a diverse
>> developer community, and hopefully the apache incubation process will
>> help you there...
>
> +1

+1

Actually I would also be interested in stepping up as a mentor.

cheers
--
Torsten


---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org

Re: Incubator Proposal: Pig

[Robert Burrell Donkin <ro...@gmail.com>]

On 9/20/07, Leo Simons <ma...@leosimons.com> wrote:
> On Sep 18, 2007, at 9:52 PM, Olga Natkovich wrote:
> > Yahoo! research and development teams have developed a proposal
> > below. The
> > proposal is also available on wiki at
> > <http://wiki.apache.org/incubator/PigProposal>
> > http://wiki.apache.org/incubator/PigProposal.
> > We would like to ask that the ASF consider forming a podling
> > according to
> > the proposal.
> > ...
> > Pig is a platform for analyzing large data sets.
>
> +1, looks cool!
>
> ...seems like your biggest challenge here is attracting a diverse
> developer community, and hopefully the apache incubation process will
> help you there...

+1

it's very important to focus on encouraging new developers in the
neonate period of a project

the energy required to let people know about a new project is often
underestimated. the open source space is now much bigger and more
diffuse than years ago. so it's not as easy for interesting projects
and interested people to find each other any more. stuff like blogging
(www.planetapache.org aggregates many blogs written by apache
committers) and podcasting (www.feathercast.org is an apache podcast)
are useful but tend to reach only people who are already interested in
apache. articles, grassroots meeting and conference talks are also
important.

one of the black arts is trying to ensure that the right level of
exposure is acheived. too much too early before the development
infrastructure is ready leads to disappointment but too little too
late when the project is too finished means that there is less chance
for meaningful contributions to be made.

- robert

---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org

Re: Incubator Proposal: Pig

[Leo Simons <ma...@leosimons.com>]

On Sep 18, 2007, at 9:52 PM, Olga Natkovich wrote:
> Yahoo! research and development teams have developed a proposal  
> below. The
> proposal is also available on wiki at
> <http://wiki.apache.org/incubator/PigProposal>
> http://wiki.apache.org/incubator/PigProposal.
> We would like to ask that the ASF consider forming a podling  
> according to
> the proposal.
> ...
> Pig is a platform for analyzing large data sets.

+1, looks cool!

...seems like your biggest challenge here is attracting a diverse  
developer community, and hopefully the apache incubation process will  
help you there...

cheers,

Leo Simons
--
http://www.leosimons.com/blog/



---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org