[Vote] Merge branch-trunk-win to trunk

[Suresh Srinivas <su...@hortonworks.com>]

I had posted heads up about merging branch-trunk-win to trunk on Feb 8th. I
am happy to announce that we are ready for the merge.

Here is a brief recap on the highlights of the work done:
- Command-line scripts for the Hadoop surface area
- Mapping the HDFS permissions model to Windows
- Abstracted and reconciled mismatches around differences in Path semantics
in Java and Windows
- Native Task Controller for Windows
- Implementation of a Block Placement Policy to support cloud environments,
more specifically Azure.
- Implementation of Hadoop native libraries for Windows (compression
codecs, native I/O)
- Several reliability issues, including race-conditions, intermittent test
failures, resource leaks.
- Several new unit test cases written for the above changes

Please find the details of the work in CHANGES.branch-trunk-win.txt -
Common changes<http://bit.ly/Xe7Ynv>, HDFS changes<http://bit.ly/13QOSo9>,
and YARN and MapReduce changes <http://bit.ly/128zzMt>. This is the work
ported from branch-1-win to a branch based on trunk.

For details of the testing done, please see the thread -
http://bit.ly/WpavJ4. Merge patch for this is available on HADOOP-8562<
https://issues.apache.org/jira/browse/HADOOP-8562>.

This was a large undertaking that involved developing code, testing the
entire Hadoop stack, including scale tests. This is made possible only with
the contribution from many many folks in the community. Following people
contributed to this work: Ivan Mitic, Chuan Liu, Ramya Sunil, Bikas Saha,
Kanna Karanam, John Gordon, Brandon Li, Chris Nauroth, David Lao, Sumadhur
Reddy Bolli, Arpit Agarwal, Ahmed El Baz, Mike Liddell, Jing Zhao, Thejas
Nair, Steve Maine, Ganeshan Iyer, Raja Aluri, Giridharan Kesavan, Ramya
Bharathi Nimmagadda, Daryn Sharp, Arun Murthy, Tsz-Wo Nicholas Sze, Suresh
Srinivas and Sanjay Radia. There are many others who contributed as well
providing feedback and comments on numerous jiras.

The vote will run for seven days and will end on March 5, 6:00PM PST.

Regards,
Suresh




On Thu, Feb 7, 2013 at 6:41 PM, Mahadevan Venkatraman
<ma...@microsoft.com>wrote:

> It is super exciting to look at the prospect of these changes being merged
> to trunk. Having Windows as one of the supported Hadoop platforms is a
> fantastic opportunity both for the Hadoop project and Microsoft customers.
>
> This work began around a year back when a few of us started with a basic
> port of Hadoop on Windows. Ever since, the Hadoop team in Microsoft have
> made significant progress in the following areas:
> (PS: Some of these items are already included in Suresh's email, but
> including again for completeness)
>
> - Command-line scripts for the Hadoop surface area
> - Mapping the HDFS permissions model to Windows
> - Abstracted and reconciled mismatches around differences in Path
> semantics in Java and Windows
> - Native Task Controller for Windows
> - Implementation of a Block Placement Policy to support cloud
> environments, more specifically Azure.
> - Implementation of Hadoop native libraries for Windows (compression
> codecs, native I/O) - Several reliability issues, including
> race-conditions, intermittent test failures, resource leaks.
> - Several new unit test cases written for the above changes
>
> In the process, we have closely engaged with the Apache open source
> community and have got great support and assistance from the community in
> terms of contributing fixes, code review comments and commits.
>
> In addition, the Hadoop team at Microsoft has also made good progress in
> other projects including Hive, Pig, Sqoop, Oozie, HCat and HBase. Many of
> these changes have already been committed to the respective trunks with
> help from various committers and contributors. It is great to see the
> commitment of the community to support multiple platforms, and we look
> forward to the day when a developer/customer is able to successfully deploy
> a complete solution stack based on Apache Hadoop releases.
>
> Next Steps:
>
> All of the above changes are part of the Windows Azure HDInsight and
> HDInsight Server products from Microsoft. We have successfully on-boarded
> several internal customers and have been running production workloads on
> Windows Azure HDInsight. Our vision is to create a big data platform based
> on Hadoop, and we are committed to helping make Hadoop a world-class
> solution that anyone can use to solve their biggest data challenges.
>
> As an immediate next step, we would like to have a discussion around how
> we can ensure that the quality of the mainline Hadoop branches on Windows
> is maintained. To this end, we would like to get to the state where we have
> pre-checkin validation gates and nightly test runs enabled on Windows. If
> you have any suggestions around this, please do send an email.  We are
> committed to helping sustain the long-term quality of Hadoop on both Linux
> and Windows.
>
> We sincerely thank the community for their contribution and support so
> far. And hope to continue having a close engagement in the future.
>
> -Microsoft HDInsight Team
>
>
> -----Original Message-----
> From: Suresh Srinivas [mailto:suresh@hortonworks.com]
> Sent: Thursday, February 7, 2013 5:42 PM
> To: common-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org;
> hdfs-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
> Subject: Heads up - merge branch-trunk-win to trunk
>
> The support for Hadoop on Windows was proposed in HADOOP-8079<
> https://issues.apache.org/jira/browse/HADOOP-8079> almost a year ago. The
> goal was to make Hadoop natively integrated, full-featured, and performance
> and scalability tuned on Windows Server or Windows Azure.
> We are happy to announce that a lot of progress has been made in this
> regard.
>
> Initial work started in a feature branch, branch-1-win, based on branch-1.
> The details related to the work done in the branch can be seen in
> CHANGES.txt<
> http://svn.apache.org/viewvc/hadoop/common/branches/branch-1-win/CHANGES.branch-1-win.txt?view=markup
> >.
> This work has been ported to a branch, branch-trunk-win, based on trunk.
> Merge patch for this is available on
> HADOOP-8562<https://issues.apache.org/jira/browse/HADOOP-8562>
> .
>
> Highlights of the work done so far:
> 1. Necessary changes in Hadoop to run natively on Windows. These changes
> handle differences in platforms related to path names, process/task
> management etc.
> 2. Addition of winutils tools for managing file permissions and ownership,
> user group mapping, hardlinks, symbolic links, chmod, disk utilization, and
> process/task management.
> 3. Added cmd scripts equivalent to existing shell scripts
> hadoop-daemon.sh, start and stop scripts.
> 4. Addition of block placement policy implemnation to support cloud
> enviroment, more specifically Azure.
>
> We are very close to wrapping up the work in branch-trunk-win and getting
> ready for a merge. Currently the merge patch is passing close to 100% of
> unit tests on Linux. Soon I will call for a vote to merge this branch into
> trunk.
>
> Next steps:
> 1. Call for vote to merge branch-trunk-win to trunk, when the work
> completes and precommit build is clean.
> 2. Start a discussion on adding Jenkins precommit builds on windows and
> how to integrate that with the existing commit process.
>
> Let me know if you have any questions.
>
> Regards,
> Suresh
>
>


-- 
http://hortonworks.com/download/

Re: [Vote] Merge branch-trunk-win to trunk

[Ramya Sunil <ra...@hortonworks.com>]

+1 for the merge.

As someone who has been testing the code for many months now, both on
singlenode and multinode clusters, I am very confident about the stability
and the quality of the code. I have run several regression tests to verify
distributed cache, streaming, compression, capacity scheduler, job history
and many more features in HDFS and MR.

- Ramya

On Thu, Feb 28, 2013 at 3:08 PM, sanjay Radia <sa...@hortonworks.com>wrote:

> +1
> Java has done the bulk of the work in making Hadoop multi-platform.
> Windows specific code is a tiny percentage of the code.
> Jeninks support for windows is going help us keep the platform portable
> going forward.
> I expect that the vast majority of new commits have  no problems. I
> propose that we start by fixing problems that Jenkins raises but not block
> new commits for too long if the author does not have a windows box or if a
> volunteer does not step up.
>
> sanjay
>
>
>
>

Re: [Vote] Merge branch-trunk-win to trunk

[Konstantin Boudnik <co...@apache.org>]

On Thu, Feb 28, 2013 at 03:08PM, sanjay Radia wrote:
> +1
> Java has done the bulk of the work in making Hadoop multi-platform.
> Windows specific code is a tiny percentage of the code.
> Jeninks support for windows is going help us keep the platform portable going forward.
> I expect that the vast majority of new commits have  no problems. I propose
> that we start by fixing problems that Jenkins raises but not block new
> commits for too long if the author does not have a windows box or if a
> volunteer does not step up.

Considering a typical set of software most of the people here work with it
would be completely inappropriate to block commits for failing Windows
specific features. After all, Microsoft never did bother to check what
features or compatibilty matters they have broke in Java and elsewhere, so why
should we?

I believe this kind of rules have to be set and discussed before the merge is
done.

Cheers,
  Cos

Re: [Vote] Merge branch-trunk-win to trunk

[Ramya Sunil <ra...@hortonworks.com>]

+1 for the merge.

As someone who has been testing the code for many months now, both on
singlenode and multinode clusters, I am very confident about the stability
and the quality of the code. I have run several regression tests to verify
distributed cache, streaming, compression, capacity scheduler, job history
and many more features in HDFS and MR.

- Ramya

On Thu, Feb 28, 2013 at 3:08 PM, sanjay Radia <sa...@hortonworks.com>wrote:

> +1
> Java has done the bulk of the work in making Hadoop multi-platform.
> Windows specific code is a tiny percentage of the code.
> Jeninks support for windows is going help us keep the platform portable
> going forward.
> I expect that the vast majority of new commits have  no problems. I
> propose that we start by fixing problems that Jenkins raises but not block
> new commits for too long if the author does not have a windows box or if a
> volunteer does not step up.
>
> sanjay
>
>
>
>

Re: [Vote] Merge branch-trunk-win to trunk

[Konstantin Boudnik <co...@apache.org>]

On Thu, Feb 28, 2013 at 03:08PM, sanjay Radia wrote:
> +1
> Java has done the bulk of the work in making Hadoop multi-platform.
> Windows specific code is a tiny percentage of the code.
> Jeninks support for windows is going help us keep the platform portable going forward.
> I expect that the vast majority of new commits have  no problems. I propose
> that we start by fixing problems that Jenkins raises but not block new
> commits for too long if the author does not have a windows box or if a
> volunteer does not step up.

Considering a typical set of software most of the people here work with it
would be completely inappropriate to block commits for failing Windows
specific features. After all, Microsoft never did bother to check what
features or compatibilty matters they have broke in Java and elsewhere, so why
should we?

I believe this kind of rules have to be set and discussed before the merge is
done.

Cheers,
  Cos

Re: [Vote] Merge branch-trunk-win to trunk

[Konstantin Boudnik <co...@apache.org>]

On Thu, Feb 28, 2013 at 03:08PM, sanjay Radia wrote:
> +1
> Java has done the bulk of the work in making Hadoop multi-platform.
> Windows specific code is a tiny percentage of the code.
> Jeninks support for windows is going help us keep the platform portable going forward.
> I expect that the vast majority of new commits have  no problems. I propose
> that we start by fixing problems that Jenkins raises but not block new
> commits for too long if the author does not have a windows box or if a
> volunteer does not step up.

Considering a typical set of software most of the people here work with it
would be completely inappropriate to block commits for failing Windows
specific features. After all, Microsoft never did bother to check what
features or compatibilty matters they have broke in Java and elsewhere, so why
should we?

I believe this kind of rules have to be set and discussed before the merge is
done.

Cheers,
  Cos

Re: [Vote] Merge branch-trunk-win to trunk

[Konstantin Boudnik <co...@apache.org>]

On Thu, Feb 28, 2013 at 03:08PM, sanjay Radia wrote:
> +1
> Java has done the bulk of the work in making Hadoop multi-platform.
> Windows specific code is a tiny percentage of the code.
> Jeninks support for windows is going help us keep the platform portable going forward.
> I expect that the vast majority of new commits have  no problems. I propose
> that we start by fixing problems that Jenkins raises but not block new
> commits for too long if the author does not have a windows box or if a
> volunteer does not step up.

Considering a typical set of software most of the people here work with it
would be completely inappropriate to block commits for failing Windows
specific features. After all, Microsoft never did bother to check what
features or compatibilty matters they have broke in Java and elsewhere, so why
should we?

I believe this kind of rules have to be set and discussed before the merge is
done.

Cheers,
  Cos

Re: [Vote] Merge branch-trunk-win to trunk

[Ramya Sunil <ra...@hortonworks.com>]

+1 for the merge.

As someone who has been testing the code for many months now, both on
singlenode and multinode clusters, I am very confident about the stability
and the quality of the code. I have run several regression tests to verify
distributed cache, streaming, compression, capacity scheduler, job history
and many more features in HDFS and MR.

- Ramya

On Thu, Feb 28, 2013 at 3:08 PM, sanjay Radia <sa...@hortonworks.com>wrote:

> +1
> Java has done the bulk of the work in making Hadoop multi-platform.
> Windows specific code is a tiny percentage of the code.
> Jeninks support for windows is going help us keep the platform portable
> going forward.
> I expect that the vast majority of new commits have  no problems. I
> propose that we start by fixing problems that Jenkins raises but not block
> new commits for too long if the author does not have a windows box or if a
> volunteer does not step up.
>
> sanjay
>
>
>
>

Re: [Vote] Merge branch-trunk-win to trunk

[Ramya Sunil <ra...@hortonworks.com>]

+1 for the merge.

As someone who has been testing the code for many months now, both on
singlenode and multinode clusters, I am very confident about the stability
and the quality of the code. I have run several regression tests to verify
distributed cache, streaming, compression, capacity scheduler, job history
and many more features in HDFS and MR.

- Ramya

On Thu, Feb 28, 2013 at 3:08 PM, sanjay Radia <sa...@hortonworks.com>wrote:

> +1
> Java has done the bulk of the work in making Hadoop multi-platform.
> Windows specific code is a tiny percentage of the code.
> Jeninks support for windows is going help us keep the platform portable
> going forward.
> I expect that the vast majority of new commits have  no problems. I
> propose that we start by fixing problems that Jenkins raises but not block
> new commits for too long if the author does not have a windows box or if a
> volunteer does not step up.
>
> sanjay
>
>
>
>

Re: [Vote] Merge branch-trunk-win to trunk

[sanjay Radia <sa...@hortonworks.com>]

+1
Java has done the bulk of the work in making Hadoop multi-platform.
Windows specific code is a tiny percentage of the code.
Jeninks support for windows is going help us keep the platform portable going forward.
I expect that the vast majority of new commits have  no problems. I propose that we start by fixing problems that Jenkins raises but not block new commits for too long if the author does not have a windows box or if a volunteer does not step up.

sanjay

RE: [Vote] Merge branch-trunk-win to trunk

[Bikas Saha <bi...@hortonworks.com>]

That's sounds like a reasonable approach. Like you say below, we need to
ensure is that the Java side of OS specific optimizations is generic and
wont need drastic surgery when that optimization is ported to another
platform.

Bikas

-----Original Message-----
From: Uma Maheswara Rao G [mailto:maheswara@huawei.com]
Sent: Thursday, February 28, 2013 9:20 PM
To: hdfs-dev@hadoop.apache.org; common-dev@hadoop.apache.org
Cc: yarn-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
Subject: RE: [Vote] Merge branch-trunk-win to trunk

+1 (non-binding)

 Thanks a lot for the work done by Suresh and team of community!
I don't think there will be much problems because of platform dependancy
as our development is with Java. If we have some native code portings,
then dev members has to take care of them.
One question regarding to it:
Ex: if one contributor is giving the patch for some native code porting
for some performance improvements and he is interested only in Linux. Then
I hope some other contributors will help in getting the windows patch if
possible.
If others busy to get that done with in time lines, then I think we can
commit Linux support patch and leave one JIRA open for Windows support? [
make sure that porting options configurable and platform check and give
release note about platform support..etc?]

Regards,
Uma

________________________________________
From: Suresh Srinivas [suresh@hortonworks.com]
Sent: Wednesday, February 27, 2013 4:25 AM
To: common-dev@hadoop.apache.org
Cc: yarn-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org;
mapreduce-dev@hadoop.apache.org
Subject: [Vote] Merge branch-trunk-win to trunk

I had posted heads up about merging branch-trunk-win to trunk on Feb 8th.
I am happy to announce that we are ready for the merge.

Here is a brief recap on the highlights of the work done:
- Command-line scripts for the Hadoop surface area
- Mapping the HDFS permissions model to Windows
- Abstracted and reconciled mismatches around differences in Path
semantics in Java and Windows
- Native Task Controller for Windows
- Implementation of a Block Placement Policy to support cloud
environments, more specifically Azure.
- Implementation of Hadoop native libraries for Windows (compression
codecs, native I/O)
- Several reliability issues, including race-conditions, intermittent test
failures, resource leaks.
- Several new unit test cases written for the above changes

Please find the details of the work in CHANGES.branch-trunk-win.txt -
Common changes<http://bit.ly/Xe7Ynv>, HDFS changes<http://bit.ly/13QOSo9>,
and YARN and MapReduce changes <http://bit.ly/128zzMt>. This is the work
ported from branch-1-win to a branch based on trunk.

For details of the testing done, please see the thread -
http://bit.ly/WpavJ4. Merge patch for this is available on HADOOP-8562<
https://issues.apache.org/jira/browse/HADOOP-8562>.

This was a large undertaking that involved developing code, testing the
entire Hadoop stack, including scale tests. This is made possible only
with the contribution from many many folks in the community. Following
people contributed to this work: Ivan Mitic, Chuan Liu, Ramya Sunil, Bikas
Saha, Kanna Karanam, John Gordon, Brandon Li, Chris Nauroth, David Lao,
Sumadhur Reddy Bolli, Arpit Agarwal, Ahmed El Baz, Mike Liddell, Jing
Zhao, Thejas Nair, Steve Maine, Ganeshan Iyer, Raja Aluri, Giridharan
Kesavan, Ramya Bharathi Nimmagadda, Daryn Sharp, Arun Murthy, Tsz-Wo
Nicholas Sze, Suresh Srinivas and Sanjay Radia. There are many others who
contributed as well providing feedback and comments on numerous jiras.

The vote will run for seven days and will end on March 5, 6:00PM PST.

Regards,
Suresh




On Thu, Feb 7, 2013 at 6:41 PM, Mahadevan Venkatraman
<ma...@microsoft.com>wrote:

> It is super exciting to look at the prospect of these changes being
> merged to trunk. Having Windows as one of the supported Hadoop
> platforms is a fantastic opportunity both for the Hadoop project and
Microsoft customers.
>
> This work began around a year back when a few of us started with a
> basic port of Hadoop on Windows. Ever since, the Hadoop team in
> Microsoft have made significant progress in the following areas:
> (PS: Some of these items are already included in Suresh's email, but
> including again for completeness)
>
> - Command-line scripts for the Hadoop surface area
> - Mapping the HDFS permissions model to Windows
> - Abstracted and reconciled mismatches around differences in Path
> semantics in Java and Windows
> - Native Task Controller for Windows
> - Implementation of a Block Placement Policy to support cloud
> environments, more specifically Azure.
> - Implementation of Hadoop native libraries for Windows (compression
> codecs, native I/O) - Several reliability issues, including
> race-conditions, intermittent test failures, resource leaks.
> - Several new unit test cases written for the above changes
>
> In the process, we have closely engaged with the Apache open source
> community and have got great support and assistance from the community
> in terms of contributing fixes, code review comments and commits.
>
> In addition, the Hadoop team at Microsoft has also made good progress
> in other projects including Hive, Pig, Sqoop, Oozie, HCat and HBase.
> Many of these changes have already been committed to the respective
> trunks with help from various committers and contributors. It is great
> to see the commitment of the community to support multiple platforms,
> and we look forward to the day when a developer/customer is able to
> successfully deploy a complete solution stack based on Apache Hadoop
releases.
>
> Next Steps:
>
> All of the above changes are part of the Windows Azure HDInsight and
> HDInsight Server products from Microsoft. We have successfully
> on-boarded several internal customers and have been running production
> workloads on Windows Azure HDInsight. Our vision is to create a big
> data platform based on Hadoop, and we are committed to helping make
> Hadoop a world-class solution that anyone can use to solve their biggest
data challenges.
>
> As an immediate next step, we would like to have a discussion around
> how we can ensure that the quality of the mainline Hadoop branches on
> Windows is maintained. To this end, we would like to get to the state
> where we have pre-checkin validation gates and nightly test runs
> enabled on Windows. If you have any suggestions around this, please do
> send an email.  We are committed to helping sustain the long-term
> quality of Hadoop on both Linux and Windows.
>
> We sincerely thank the community for their contribution and support so
> far. And hope to continue having a close engagement in the future.
>
> -Microsoft HDInsight Team
>
>
> -----Original Message-----
> From: Suresh Srinivas [mailto:suresh@hortonworks.com]
> Sent: Thursday, February 7, 2013 5:42 PM
> To: common-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org;
> hdfs-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
> Subject: Heads up - merge branch-trunk-win to trunk
>
> The support for Hadoop on Windows was proposed in HADOOP-8079<
> https://issues.apache.org/jira/browse/HADOOP-8079> almost a year ago.
> The goal was to make Hadoop natively integrated, full-featured, and
> performance and scalability tuned on Windows Server or Windows Azure.
> We are happy to announce that a lot of progress has been made in this
> regard.
>
> Initial work started in a feature branch, branch-1-win, based on
branch-1.
> The details related to the work done in the branch can be seen in
> CHANGES.txt<
> http://svn.apache.org/viewvc/hadoop/common/branches/branch-1-win/CHANG
> ES.branch-1-win.txt?view=markup
> >.
> This work has been ported to a branch, branch-trunk-win, based on trunk.
> Merge patch for this is available on
> HADOOP-8562<https://issues.apache.org/jira/browse/HADOOP-8562>
> .
>
> Highlights of the work done so far:
> 1. Necessary changes in Hadoop to run natively on Windows. These
> changes handle differences in platforms related to path names,
> process/task management etc.
> 2. Addition of winutils tools for managing file permissions and
> ownership, user group mapping, hardlinks, symbolic links, chmod, disk
> utilization, and process/task management.
> 3. Added cmd scripts equivalent to existing shell scripts
> hadoop-daemon.sh, start and stop scripts.
> 4. Addition of block placement policy implemnation to support cloud
> enviroment, more specifically Azure.
>
> We are very close to wrapping up the work in branch-trunk-win and
> getting ready for a merge. Currently the merge patch is passing close
> to 100% of unit tests on Linux. Soon I will call for a vote to merge
> this branch into trunk.
>
> Next steps:
> 1. Call for vote to merge branch-trunk-win to trunk, when the work
> completes and precommit build is clean.
> 2. Start a discussion on adding Jenkins precommit builds on windows
> and how to integrate that with the existing commit process.
>
> Let me know if you have any questions.
>
> Regards,
> Suresh
>
>


--
http://hortonworks.com/download/

RE: [Vote] Merge branch-trunk-win to trunk

[Bikas Saha <bi...@hortonworks.com>]

That's sounds like a reasonable approach. Like you say below, we need to
ensure is that the Java side of OS specific optimizations is generic and
wont need drastic surgery when that optimization is ported to another
platform.

Bikas

-----Original Message-----
From: Uma Maheswara Rao G [mailto:maheswara@huawei.com]
Sent: Thursday, February 28, 2013 9:20 PM
To: hdfs-dev@hadoop.apache.org; common-dev@hadoop.apache.org
Cc: yarn-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
Subject: RE: [Vote] Merge branch-trunk-win to trunk

+1 (non-binding)

 Thanks a lot for the work done by Suresh and team of community!
I don't think there will be much problems because of platform dependancy
as our development is with Java. If we have some native code portings,
then dev members has to take care of them.
One question regarding to it:
Ex: if one contributor is giving the patch for some native code porting
for some performance improvements and he is interested only in Linux. Then
I hope some other contributors will help in getting the windows patch if
possible.
If others busy to get that done with in time lines, then I think we can
commit Linux support patch and leave one JIRA open for Windows support? [
make sure that porting options configurable and platform check and give
release note about platform support..etc?]

Regards,
Uma

________________________________________
From: Suresh Srinivas [suresh@hortonworks.com]
Sent: Wednesday, February 27, 2013 4:25 AM
To: common-dev@hadoop.apache.org
Cc: yarn-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org;
mapreduce-dev@hadoop.apache.org
Subject: [Vote] Merge branch-trunk-win to trunk

I had posted heads up about merging branch-trunk-win to trunk on Feb 8th.
I am happy to announce that we are ready for the merge.

Here is a brief recap on the highlights of the work done:
- Command-line scripts for the Hadoop surface area
- Mapping the HDFS permissions model to Windows
- Abstracted and reconciled mismatches around differences in Path
semantics in Java and Windows
- Native Task Controller for Windows
- Implementation of a Block Placement Policy to support cloud
environments, more specifically Azure.
- Implementation of Hadoop native libraries for Windows (compression
codecs, native I/O)
- Several reliability issues, including race-conditions, intermittent test
failures, resource leaks.
- Several new unit test cases written for the above changes

Please find the details of the work in CHANGES.branch-trunk-win.txt -
Common changes<http://bit.ly/Xe7Ynv>, HDFS changes<http://bit.ly/13QOSo9>,
and YARN and MapReduce changes <http://bit.ly/128zzMt>. This is the work
ported from branch-1-win to a branch based on trunk.

For details of the testing done, please see the thread -
http://bit.ly/WpavJ4. Merge patch for this is available on HADOOP-8562<
https://issues.apache.org/jira/browse/HADOOP-8562>.

This was a large undertaking that involved developing code, testing the
entire Hadoop stack, including scale tests. This is made possible only
with the contribution from many many folks in the community. Following
people contributed to this work: Ivan Mitic, Chuan Liu, Ramya Sunil, Bikas
Saha, Kanna Karanam, John Gordon, Brandon Li, Chris Nauroth, David Lao,
Sumadhur Reddy Bolli, Arpit Agarwal, Ahmed El Baz, Mike Liddell, Jing
Zhao, Thejas Nair, Steve Maine, Ganeshan Iyer, Raja Aluri, Giridharan
Kesavan, Ramya Bharathi Nimmagadda, Daryn Sharp, Arun Murthy, Tsz-Wo
Nicholas Sze, Suresh Srinivas and Sanjay Radia. There are many others who
contributed as well providing feedback and comments on numerous jiras.

The vote will run for seven days and will end on March 5, 6:00PM PST.

Regards,
Suresh




On Thu, Feb 7, 2013 at 6:41 PM, Mahadevan Venkatraman
<ma...@microsoft.com>wrote:

> It is super exciting to look at the prospect of these changes being
> merged to trunk. Having Windows as one of the supported Hadoop
> platforms is a fantastic opportunity both for the Hadoop project and
Microsoft customers.
>
> This work began around a year back when a few of us started with a
> basic port of Hadoop on Windows. Ever since, the Hadoop team in
> Microsoft have made significant progress in the following areas:
> (PS: Some of these items are already included in Suresh's email, but
> including again for completeness)
>
> - Command-line scripts for the Hadoop surface area
> - Mapping the HDFS permissions model to Windows
> - Abstracted and reconciled mismatches around differences in Path
> semantics in Java and Windows
> - Native Task Controller for Windows
> - Implementation of a Block Placement Policy to support cloud
> environments, more specifically Azure.
> - Implementation of Hadoop native libraries for Windows (compression
> codecs, native I/O) - Several reliability issues, including
> race-conditions, intermittent test failures, resource leaks.
> - Several new unit test cases written for the above changes
>
> In the process, we have closely engaged with the Apache open source
> community and have got great support and assistance from the community
> in terms of contributing fixes, code review comments and commits.
>
> In addition, the Hadoop team at Microsoft has also made good progress
> in other projects including Hive, Pig, Sqoop, Oozie, HCat and HBase.
> Many of these changes have already been committed to the respective
> trunks with help from various committers and contributors. It is great
> to see the commitment of the community to support multiple platforms,
> and we look forward to the day when a developer/customer is able to
> successfully deploy a complete solution stack based on Apache Hadoop
releases.
>
> Next Steps:
>
> All of the above changes are part of the Windows Azure HDInsight and
> HDInsight Server products from Microsoft. We have successfully
> on-boarded several internal customers and have been running production
> workloads on Windows Azure HDInsight. Our vision is to create a big
> data platform based on Hadoop, and we are committed to helping make
> Hadoop a world-class solution that anyone can use to solve their biggest
data challenges.
>
> As an immediate next step, we would like to have a discussion around
> how we can ensure that the quality of the mainline Hadoop branches on
> Windows is maintained. To this end, we would like to get to the state
> where we have pre-checkin validation gates and nightly test runs
> enabled on Windows. If you have any suggestions around this, please do
> send an email.  We are committed to helping sustain the long-term
> quality of Hadoop on both Linux and Windows.
>
> We sincerely thank the community for their contribution and support so
> far. And hope to continue having a close engagement in the future.
>
> -Microsoft HDInsight Team
>
>
> -----Original Message-----
> From: Suresh Srinivas [mailto:suresh@hortonworks.com]
> Sent: Thursday, February 7, 2013 5:42 PM
> To: common-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org;
> hdfs-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
> Subject: Heads up - merge branch-trunk-win to trunk
>
> The support for Hadoop on Windows was proposed in HADOOP-8079<
> https://issues.apache.org/jira/browse/HADOOP-8079> almost a year ago.
> The goal was to make Hadoop natively integrated, full-featured, and
> performance and scalability tuned on Windows Server or Windows Azure.
> We are happy to announce that a lot of progress has been made in this
> regard.
>
> Initial work started in a feature branch, branch-1-win, based on
branch-1.
> The details related to the work done in the branch can be seen in
> CHANGES.txt<
> http://svn.apache.org/viewvc/hadoop/common/branches/branch-1-win/CHANG
> ES.branch-1-win.txt?view=markup
> >.
> This work has been ported to a branch, branch-trunk-win, based on trunk.
> Merge patch for this is available on
> HADOOP-8562<https://issues.apache.org/jira/browse/HADOOP-8562>
> .
>
> Highlights of the work done so far:
> 1. Necessary changes in Hadoop to run natively on Windows. These
> changes handle differences in platforms related to path names,
> process/task management etc.
> 2. Addition of winutils tools for managing file permissions and
> ownership, user group mapping, hardlinks, symbolic links, chmod, disk
> utilization, and process/task management.
> 3. Added cmd scripts equivalent to existing shell scripts
> hadoop-daemon.sh, start and stop scripts.
> 4. Addition of block placement policy implemnation to support cloud
> enviroment, more specifically Azure.
>
> We are very close to wrapping up the work in branch-trunk-win and
> getting ready for a merge. Currently the merge patch is passing close
> to 100% of unit tests on Linux. Soon I will call for a vote to merge
> this branch into trunk.
>
> Next steps:
> 1. Call for vote to merge branch-trunk-win to trunk, when the work
> completes and precommit build is clean.
> 2. Start a discussion on adding Jenkins precommit builds on windows
> and how to integrate that with the existing commit process.
>
> Let me know if you have any questions.
>
> Regards,
> Suresh
>
>


--
http://hortonworks.com/download/

RE: [Vote] Merge branch-trunk-win to trunk

[Bikas Saha <bi...@hortonworks.com>]

That's sounds like a reasonable approach. Like you say below, we need to
ensure is that the Java side of OS specific optimizations is generic and
wont need drastic surgery when that optimization is ported to another
platform.

Bikas

-----Original Message-----
From: Uma Maheswara Rao G [mailto:maheswara@huawei.com]
Sent: Thursday, February 28, 2013 9:20 PM
To: hdfs-dev@hadoop.apache.org; common-dev@hadoop.apache.org
Cc: yarn-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
Subject: RE: [Vote] Merge branch-trunk-win to trunk

+1 (non-binding)

 Thanks a lot for the work done by Suresh and team of community!
I don't think there will be much problems because of platform dependancy
as our development is with Java. If we have some native code portings,
then dev members has to take care of them.
One question regarding to it:
Ex: if one contributor is giving the patch for some native code porting
for some performance improvements and he is interested only in Linux. Then
I hope some other contributors will help in getting the windows patch if
possible.
If others busy to get that done with in time lines, then I think we can
commit Linux support patch and leave one JIRA open for Windows support? [
make sure that porting options configurable and platform check and give
release note about platform support..etc?]

Regards,
Uma

________________________________________
From: Suresh Srinivas [suresh@hortonworks.com]
Sent: Wednesday, February 27, 2013 4:25 AM
To: common-dev@hadoop.apache.org
Cc: yarn-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org;
mapreduce-dev@hadoop.apache.org
Subject: [Vote] Merge branch-trunk-win to trunk

I had posted heads up about merging branch-trunk-win to trunk on Feb 8th.
I am happy to announce that we are ready for the merge.

Here is a brief recap on the highlights of the work done:
- Command-line scripts for the Hadoop surface area
- Mapping the HDFS permissions model to Windows
- Abstracted and reconciled mismatches around differences in Path
semantics in Java and Windows
- Native Task Controller for Windows
- Implementation of a Block Placement Policy to support cloud
environments, more specifically Azure.
- Implementation of Hadoop native libraries for Windows (compression
codecs, native I/O)
- Several reliability issues, including race-conditions, intermittent test
failures, resource leaks.
- Several new unit test cases written for the above changes

Please find the details of the work in CHANGES.branch-trunk-win.txt -
Common changes<http://bit.ly/Xe7Ynv>, HDFS changes<http://bit.ly/13QOSo9>,
and YARN and MapReduce changes <http://bit.ly/128zzMt>. This is the work
ported from branch-1-win to a branch based on trunk.

For details of the testing done, please see the thread -
http://bit.ly/WpavJ4. Merge patch for this is available on HADOOP-8562<
https://issues.apache.org/jira/browse/HADOOP-8562>.

This was a large undertaking that involved developing code, testing the
entire Hadoop stack, including scale tests. This is made possible only
with the contribution from many many folks in the community. Following
people contributed to this work: Ivan Mitic, Chuan Liu, Ramya Sunil, Bikas
Saha, Kanna Karanam, John Gordon, Brandon Li, Chris Nauroth, David Lao,
Sumadhur Reddy Bolli, Arpit Agarwal, Ahmed El Baz, Mike Liddell, Jing
Zhao, Thejas Nair, Steve Maine, Ganeshan Iyer, Raja Aluri, Giridharan
Kesavan, Ramya Bharathi Nimmagadda, Daryn Sharp, Arun Murthy, Tsz-Wo
Nicholas Sze, Suresh Srinivas and Sanjay Radia. There are many others who
contributed as well providing feedback and comments on numerous jiras.

The vote will run for seven days and will end on March 5, 6:00PM PST.

Regards,
Suresh




On Thu, Feb 7, 2013 at 6:41 PM, Mahadevan Venkatraman
<ma...@microsoft.com>wrote:

> It is super exciting to look at the prospect of these changes being
> merged to trunk. Having Windows as one of the supported Hadoop
> platforms is a fantastic opportunity both for the Hadoop project and
Microsoft customers.
>
> This work began around a year back when a few of us started with a
> basic port of Hadoop on Windows. Ever since, the Hadoop team in
> Microsoft have made significant progress in the following areas:
> (PS: Some of these items are already included in Suresh's email, but
> including again for completeness)
>
> - Command-line scripts for the Hadoop surface area
> - Mapping the HDFS permissions model to Windows
> - Abstracted and reconciled mismatches around differences in Path
> semantics in Java and Windows
> - Native Task Controller for Windows
> - Implementation of a Block Placement Policy to support cloud
> environments, more specifically Azure.
> - Implementation of Hadoop native libraries for Windows (compression
> codecs, native I/O) - Several reliability issues, including
> race-conditions, intermittent test failures, resource leaks.
> - Several new unit test cases written for the above changes
>
> In the process, we have closely engaged with the Apache open source
> community and have got great support and assistance from the community
> in terms of contributing fixes, code review comments and commits.
>
> In addition, the Hadoop team at Microsoft has also made good progress
> in other projects including Hive, Pig, Sqoop, Oozie, HCat and HBase.
> Many of these changes have already been committed to the respective
> trunks with help from various committers and contributors. It is great
> to see the commitment of the community to support multiple platforms,
> and we look forward to the day when a developer/customer is able to
> successfully deploy a complete solution stack based on Apache Hadoop
releases.
>
> Next Steps:
>
> All of the above changes are part of the Windows Azure HDInsight and
> HDInsight Server products from Microsoft. We have successfully
> on-boarded several internal customers and have been running production
> workloads on Windows Azure HDInsight. Our vision is to create a big
> data platform based on Hadoop, and we are committed to helping make
> Hadoop a world-class solution that anyone can use to solve their biggest
data challenges.
>
> As an immediate next step, we would like to have a discussion around
> how we can ensure that the quality of the mainline Hadoop branches on
> Windows is maintained. To this end, we would like to get to the state
> where we have pre-checkin validation gates and nightly test runs
> enabled on Windows. If you have any suggestions around this, please do
> send an email.  We are committed to helping sustain the long-term
> quality of Hadoop on both Linux and Windows.
>
> We sincerely thank the community for their contribution and support so
> far. And hope to continue having a close engagement in the future.
>
> -Microsoft HDInsight Team
>
>
> -----Original Message-----
> From: Suresh Srinivas [mailto:suresh@hortonworks.com]
> Sent: Thursday, February 7, 2013 5:42 PM
> To: common-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org;
> hdfs-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
> Subject: Heads up - merge branch-trunk-win to trunk
>
> The support for Hadoop on Windows was proposed in HADOOP-8079<
> https://issues.apache.org/jira/browse/HADOOP-8079> almost a year ago.
> The goal was to make Hadoop natively integrated, full-featured, and
> performance and scalability tuned on Windows Server or Windows Azure.
> We are happy to announce that a lot of progress has been made in this
> regard.
>
> Initial work started in a feature branch, branch-1-win, based on
branch-1.
> The details related to the work done in the branch can be seen in
> CHANGES.txt<
> http://svn.apache.org/viewvc/hadoop/common/branches/branch-1-win/CHANG
> ES.branch-1-win.txt?view=markup
> >.
> This work has been ported to a branch, branch-trunk-win, based on trunk.
> Merge patch for this is available on
> HADOOP-8562<https://issues.apache.org/jira/browse/HADOOP-8562>
> .
>
> Highlights of the work done so far:
> 1. Necessary changes in Hadoop to run natively on Windows. These
> changes handle differences in platforms related to path names,
> process/task management etc.
> 2. Addition of winutils tools for managing file permissions and
> ownership, user group mapping, hardlinks, symbolic links, chmod, disk
> utilization, and process/task management.
> 3. Added cmd scripts equivalent to existing shell scripts
> hadoop-daemon.sh, start and stop scripts.
> 4. Addition of block placement policy implemnation to support cloud
> enviroment, more specifically Azure.
>
> We are very close to wrapping up the work in branch-trunk-win and
> getting ready for a merge. Currently the merge patch is passing close
> to 100% of unit tests on Linux. Soon I will call for a vote to merge
> this branch into trunk.
>
> Next steps:
> 1. Call for vote to merge branch-trunk-win to trunk, when the work
> completes and precommit build is clean.
> 2. Start a discussion on adding Jenkins precommit builds on windows
> and how to integrate that with the existing commit process.
>
> Let me know if you have any questions.
>
> Regards,
> Suresh
>
>


--
http://hortonworks.com/download/

RE: [Vote] Merge branch-trunk-win to trunk

[Bikas Saha <bi...@hortonworks.com>]

That's sounds like a reasonable approach. Like you say below, we need to
ensure is that the Java side of OS specific optimizations is generic and
wont need drastic surgery when that optimization is ported to another
platform.

Bikas

-----Original Message-----
From: Uma Maheswara Rao G [mailto:maheswara@huawei.com]
Sent: Thursday, February 28, 2013 9:20 PM
To: hdfs-dev@hadoop.apache.org; common-dev@hadoop.apache.org
Cc: yarn-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
Subject: RE: [Vote] Merge branch-trunk-win to trunk

+1 (non-binding)

 Thanks a lot for the work done by Suresh and team of community!
I don't think there will be much problems because of platform dependancy
as our development is with Java. If we have some native code portings,
then dev members has to take care of them.
One question regarding to it:
Ex: if one contributor is giving the patch for some native code porting
for some performance improvements and he is interested only in Linux. Then
I hope some other contributors will help in getting the windows patch if
possible.
If others busy to get that done with in time lines, then I think we can
commit Linux support patch and leave one JIRA open for Windows support? [
make sure that porting options configurable and platform check and give
release note about platform support..etc?]

Regards,
Uma

________________________________________
From: Suresh Srinivas [suresh@hortonworks.com]
Sent: Wednesday, February 27, 2013 4:25 AM
To: common-dev@hadoop.apache.org
Cc: yarn-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org;
mapreduce-dev@hadoop.apache.org
Subject: [Vote] Merge branch-trunk-win to trunk

I had posted heads up about merging branch-trunk-win to trunk on Feb 8th.
I am happy to announce that we are ready for the merge.

Here is a brief recap on the highlights of the work done:
- Command-line scripts for the Hadoop surface area
- Mapping the HDFS permissions model to Windows
- Abstracted and reconciled mismatches around differences in Path
semantics in Java and Windows
- Native Task Controller for Windows
- Implementation of a Block Placement Policy to support cloud
environments, more specifically Azure.
- Implementation of Hadoop native libraries for Windows (compression
codecs, native I/O)
- Several reliability issues, including race-conditions, intermittent test
failures, resource leaks.
- Several new unit test cases written for the above changes

Please find the details of the work in CHANGES.branch-trunk-win.txt -
Common changes<http://bit.ly/Xe7Ynv>, HDFS changes<http://bit.ly/13QOSo9>,
and YARN and MapReduce changes <http://bit.ly/128zzMt>. This is the work
ported from branch-1-win to a branch based on trunk.

For details of the testing done, please see the thread -
http://bit.ly/WpavJ4. Merge patch for this is available on HADOOP-8562<
https://issues.apache.org/jira/browse/HADOOP-8562>.

This was a large undertaking that involved developing code, testing the
entire Hadoop stack, including scale tests. This is made possible only
with the contribution from many many folks in the community. Following
people contributed to this work: Ivan Mitic, Chuan Liu, Ramya Sunil, Bikas
Saha, Kanna Karanam, John Gordon, Brandon Li, Chris Nauroth, David Lao,
Sumadhur Reddy Bolli, Arpit Agarwal, Ahmed El Baz, Mike Liddell, Jing
Zhao, Thejas Nair, Steve Maine, Ganeshan Iyer, Raja Aluri, Giridharan
Kesavan, Ramya Bharathi Nimmagadda, Daryn Sharp, Arun Murthy, Tsz-Wo
Nicholas Sze, Suresh Srinivas and Sanjay Radia. There are many others who
contributed as well providing feedback and comments on numerous jiras.

The vote will run for seven days and will end on March 5, 6:00PM PST.

Regards,
Suresh




On Thu, Feb 7, 2013 at 6:41 PM, Mahadevan Venkatraman
<ma...@microsoft.com>wrote:

> It is super exciting to look at the prospect of these changes being
> merged to trunk. Having Windows as one of the supported Hadoop
> platforms is a fantastic opportunity both for the Hadoop project and
Microsoft customers.
>
> This work began around a year back when a few of us started with a
> basic port of Hadoop on Windows. Ever since, the Hadoop team in
> Microsoft have made significant progress in the following areas:
> (PS: Some of these items are already included in Suresh's email, but
> including again for completeness)
>
> - Command-line scripts for the Hadoop surface area
> - Mapping the HDFS permissions model to Windows
> - Abstracted and reconciled mismatches around differences in Path
> semantics in Java and Windows
> - Native Task Controller for Windows
> - Implementation of a Block Placement Policy to support cloud
> environments, more specifically Azure.
> - Implementation of Hadoop native libraries for Windows (compression
> codecs, native I/O) - Several reliability issues, including
> race-conditions, intermittent test failures, resource leaks.
> - Several new unit test cases written for the above changes
>
> In the process, we have closely engaged with the Apache open source
> community and have got great support and assistance from the community
> in terms of contributing fixes, code review comments and commits.
>
> In addition, the Hadoop team at Microsoft has also made good progress
> in other projects including Hive, Pig, Sqoop, Oozie, HCat and HBase.
> Many of these changes have already been committed to the respective
> trunks with help from various committers and contributors. It is great
> to see the commitment of the community to support multiple platforms,
> and we look forward to the day when a developer/customer is able to
> successfully deploy a complete solution stack based on Apache Hadoop
releases.
>
> Next Steps:
>
> All of the above changes are part of the Windows Azure HDInsight and
> HDInsight Server products from Microsoft. We have successfully
> on-boarded several internal customers and have been running production
> workloads on Windows Azure HDInsight. Our vision is to create a big
> data platform based on Hadoop, and we are committed to helping make
> Hadoop a world-class solution that anyone can use to solve their biggest
data challenges.
>
> As an immediate next step, we would like to have a discussion around
> how we can ensure that the quality of the mainline Hadoop branches on
> Windows is maintained. To this end, we would like to get to the state
> where we have pre-checkin validation gates and nightly test runs
> enabled on Windows. If you have any suggestions around this, please do
> send an email.  We are committed to helping sustain the long-term
> quality of Hadoop on both Linux and Windows.
>
> We sincerely thank the community for their contribution and support so
> far. And hope to continue having a close engagement in the future.
>
> -Microsoft HDInsight Team
>
>
> -----Original Message-----
> From: Suresh Srinivas [mailto:suresh@hortonworks.com]
> Sent: Thursday, February 7, 2013 5:42 PM
> To: common-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org;
> hdfs-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
> Subject: Heads up - merge branch-trunk-win to trunk
>
> The support for Hadoop on Windows was proposed in HADOOP-8079<
> https://issues.apache.org/jira/browse/HADOOP-8079> almost a year ago.
> The goal was to make Hadoop natively integrated, full-featured, and
> performance and scalability tuned on Windows Server or Windows Azure.
> We are happy to announce that a lot of progress has been made in this
> regard.
>
> Initial work started in a feature branch, branch-1-win, based on
branch-1.
> The details related to the work done in the branch can be seen in
> CHANGES.txt<
> http://svn.apache.org/viewvc/hadoop/common/branches/branch-1-win/CHANG
> ES.branch-1-win.txt?view=markup
> >.
> This work has been ported to a branch, branch-trunk-win, based on trunk.
> Merge patch for this is available on
> HADOOP-8562<https://issues.apache.org/jira/browse/HADOOP-8562>
> .
>
> Highlights of the work done so far:
> 1. Necessary changes in Hadoop to run natively on Windows. These
> changes handle differences in platforms related to path names,
> process/task management etc.
> 2. Addition of winutils tools for managing file permissions and
> ownership, user group mapping, hardlinks, symbolic links, chmod, disk
> utilization, and process/task management.
> 3. Added cmd scripts equivalent to existing shell scripts
> hadoop-daemon.sh, start and stop scripts.
> 4. Addition of block placement policy implemnation to support cloud
> enviroment, more specifically Azure.
>
> We are very close to wrapping up the work in branch-trunk-win and
> getting ready for a merge. Currently the merge patch is passing close
> to 100% of unit tests on Linux. Soon I will call for a vote to merge
> this branch into trunk.
>
> Next steps:
> 1. Call for vote to merge branch-trunk-win to trunk, when the work
> completes and precommit build is clean.
> 2. Start a discussion on adding Jenkins precommit builds on windows
> and how to integrate that with the existing commit process.
>
> Let me know if you have any questions.
>
> Regards,
> Suresh
>
>


--
http://hortonworks.com/download/

RE: [Vote] Merge branch-trunk-win to trunk

[Uma Maheswara Rao G <ma...@huawei.com>]

+1 (non-binding)

 Thanks a lot for the work done by Suresh and team of community!
I don't think there will be much problems because of platform dependancy as our development is with Java. If we have some native code portings, then dev members has to take care of them.
One question regarding to it:
Ex: if one contributor is giving the patch for some native code porting for some performance improvements and he is interested only in Linux. Then I hope some other contributors will help in getting the windows patch if possible.
If others busy to get that done with in time lines, then I think we can commit Linux support patch and leave one JIRA open for Windows support? [ make sure that porting options configurable and platform check and give release note about platform support..etc?]

Regards,
Uma

________________________________________
From: Suresh Srinivas [suresh@hortonworks.com]
Sent: Wednesday, February 27, 2013 4:25 AM
To: common-dev@hadoop.apache.org
Cc: yarn-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
Subject: [Vote] Merge branch-trunk-win to trunk

I had posted heads up about merging branch-trunk-win to trunk on Feb 8th. I
am happy to announce that we are ready for the merge.

Here is a brief recap on the highlights of the work done:
- Command-line scripts for the Hadoop surface area
- Mapping the HDFS permissions model to Windows
- Abstracted and reconciled mismatches around differences in Path semantics
in Java and Windows
- Native Task Controller for Windows
- Implementation of a Block Placement Policy to support cloud environments,
more specifically Azure.
- Implementation of Hadoop native libraries for Windows (compression
codecs, native I/O)
- Several reliability issues, including race-conditions, intermittent test
failures, resource leaks.
- Several new unit test cases written for the above changes

Please find the details of the work in CHANGES.branch-trunk-win.txt -
Common changes<http://bit.ly/Xe7Ynv>, HDFS changes<http://bit.ly/13QOSo9>,
and YARN and MapReduce changes <http://bit.ly/128zzMt>. This is the work
ported from branch-1-win to a branch based on trunk.

For details of the testing done, please see the thread -
http://bit.ly/WpavJ4. Merge patch for this is available on HADOOP-8562<
https://issues.apache.org/jira/browse/HADOOP-8562>.

This was a large undertaking that involved developing code, testing the
entire Hadoop stack, including scale tests. This is made possible only with
the contribution from many many folks in the community. Following people
contributed to this work: Ivan Mitic, Chuan Liu, Ramya Sunil, Bikas Saha,
Kanna Karanam, John Gordon, Brandon Li, Chris Nauroth, David Lao, Sumadhur
Reddy Bolli, Arpit Agarwal, Ahmed El Baz, Mike Liddell, Jing Zhao, Thejas
Nair, Steve Maine, Ganeshan Iyer, Raja Aluri, Giridharan Kesavan, Ramya
Bharathi Nimmagadda, Daryn Sharp, Arun Murthy, Tsz-Wo Nicholas Sze, Suresh
Srinivas and Sanjay Radia. There are many others who contributed as well
providing feedback and comments on numerous jiras.

The vote will run for seven days and will end on March 5, 6:00PM PST.

Regards,
Suresh




On Thu, Feb 7, 2013 at 6:41 PM, Mahadevan Venkatraman
<ma...@microsoft.com>wrote:

> It is super exciting to look at the prospect of these changes being merged
> to trunk. Having Windows as one of the supported Hadoop platforms is a
> fantastic opportunity both for the Hadoop project and Microsoft customers.
>
> This work began around a year back when a few of us started with a basic
> port of Hadoop on Windows. Ever since, the Hadoop team in Microsoft have
> made significant progress in the following areas:
> (PS: Some of these items are already included in Suresh's email, but
> including again for completeness)
>
> - Command-line scripts for the Hadoop surface area
> - Mapping the HDFS permissions model to Windows
> - Abstracted and reconciled mismatches around differences in Path
> semantics in Java and Windows
> - Native Task Controller for Windows
> - Implementation of a Block Placement Policy to support cloud
> environments, more specifically Azure.
> - Implementation of Hadoop native libraries for Windows (compression
> codecs, native I/O) - Several reliability issues, including
> race-conditions, intermittent test failures, resource leaks.
> - Several new unit test cases written for the above changes
>
> In the process, we have closely engaged with the Apache open source
> community and have got great support and assistance from the community in
> terms of contributing fixes, code review comments and commits.
>
> In addition, the Hadoop team at Microsoft has also made good progress in
> other projects including Hive, Pig, Sqoop, Oozie, HCat and HBase. Many of
> these changes have already been committed to the respective trunks with
> help from various committers and contributors. It is great to see the
> commitment of the community to support multiple platforms, and we look
> forward to the day when a developer/customer is able to successfully deploy
> a complete solution stack based on Apache Hadoop releases.
>
> Next Steps:
>
> All of the above changes are part of the Windows Azure HDInsight and
> HDInsight Server products from Microsoft. We have successfully on-boarded
> several internal customers and have been running production workloads on
> Windows Azure HDInsight. Our vision is to create a big data platform based
> on Hadoop, and we are committed to helping make Hadoop a world-class
> solution that anyone can use to solve their biggest data challenges.
>
> As an immediate next step, we would like to have a discussion around how
> we can ensure that the quality of the mainline Hadoop branches on Windows
> is maintained. To this end, we would like to get to the state where we have
> pre-checkin validation gates and nightly test runs enabled on Windows. If
> you have any suggestions around this, please do send an email.  We are
> committed to helping sustain the long-term quality of Hadoop on both Linux
> and Windows.
>
> We sincerely thank the community for their contribution and support so
> far. And hope to continue having a close engagement in the future.
>
> -Microsoft HDInsight Team
>
>
> -----Original Message-----
> From: Suresh Srinivas [mailto:suresh@hortonworks.com]
> Sent: Thursday, February 7, 2013 5:42 PM
> To: common-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org;
> hdfs-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
> Subject: Heads up - merge branch-trunk-win to trunk
>
> The support for Hadoop on Windows was proposed in HADOOP-8079<
> https://issues.apache.org/jira/browse/HADOOP-8079> almost a year ago. The
> goal was to make Hadoop natively integrated, full-featured, and performance
> and scalability tuned on Windows Server or Windows Azure.
> We are happy to announce that a lot of progress has been made in this
> regard.
>
> Initial work started in a feature branch, branch-1-win, based on branch-1.
> The details related to the work done in the branch can be seen in
> CHANGES.txt<
> http://svn.apache.org/viewvc/hadoop/common/branches/branch-1-win/CHANGES.branch-1-win.txt?view=markup
> >.
> This work has been ported to a branch, branch-trunk-win, based on trunk.
> Merge patch for this is available on
> HADOOP-8562<https://issues.apache.org/jira/browse/HADOOP-8562>
> .
>
> Highlights of the work done so far:
> 1. Necessary changes in Hadoop to run natively on Windows. These changes
> handle differences in platforms related to path names, process/task
> management etc.
> 2. Addition of winutils tools for managing file permissions and ownership,
> user group mapping, hardlinks, symbolic links, chmod, disk utilization, and
> process/task management.
> 3. Added cmd scripts equivalent to existing shell scripts
> hadoop-daemon.sh, start and stop scripts.
> 4. Addition of block placement policy implemnation to support cloud
> enviroment, more specifically Azure.
>
> We are very close to wrapping up the work in branch-trunk-win and getting
> ready for a merge. Currently the merge patch is passing close to 100% of
> unit tests on Linux. Soon I will call for a vote to merge this branch into
> trunk.
>
> Next steps:
> 1. Call for vote to merge branch-trunk-win to trunk, when the work
> completes and precommit build is clean.
> 2. Start a discussion on adding Jenkins precommit builds on windows and
> how to integrate that with the existing commit process.
>
> Let me know if you have any questions.
>
> Regards,
> Suresh
>
>


--
http://hortonworks.com/download/

Re: [Vote] Merge branch-trunk-win to trunk

[Arun C Murthy <ac...@hortonworks.com>]

+1 (binding)

Very exciting to see this effort come to fruition!

Arun

On Feb 26, 2013, at 2:55 PM, Suresh Srinivas wrote:

> I had posted heads up about merging branch-trunk-win to trunk on Feb 8th. I
> am happy to announce that we are ready for the merge.
> 
> Here is a brief recap on the highlights of the work done:
> - Command-line scripts for the Hadoop surface area
> - Mapping the HDFS permissions model to Windows
> - Abstracted and reconciled mismatches around differences in Path semantics
> in Java and Windows
> - Native Task Controller for Windows
> - Implementation of a Block Placement Policy to support cloud environments,
> more specifically Azure.
> - Implementation of Hadoop native libraries for Windows (compression
> codecs, native I/O)
> - Several reliability issues, including race-conditions, intermittent test
> failures, resource leaks.
> - Several new unit test cases written for the above changes
> 
> Please find the details of the work in CHANGES.branch-trunk-win.txt -
> Common changes<http://bit.ly/Xe7Ynv>, HDFS changes<http://bit.ly/13QOSo9>,
> and YARN and MapReduce changes <http://bit.ly/128zzMt>. This is the work
> ported from branch-1-win to a branch based on trunk.
> 
> For details of the testing done, please see the thread -
> http://bit.ly/WpavJ4. Merge patch for this is available on HADOOP-8562<
> https://issues.apache.org/jira/browse/HADOOP-8562>.
> 
> This was a large undertaking that involved developing code, testing the
> entire Hadoop stack, including scale tests. This is made possible only with
> the contribution from many many folks in the community. Following people
> contributed to this work: Ivan Mitic, Chuan Liu, Ramya Sunil, Bikas Saha,
> Kanna Karanam, John Gordon, Brandon Li, Chris Nauroth, David Lao, Sumadhur
> Reddy Bolli, Arpit Agarwal, Ahmed El Baz, Mike Liddell, Jing Zhao, Thejas
> Nair, Steve Maine, Ganeshan Iyer, Raja Aluri, Giridharan Kesavan, Ramya
> Bharathi Nimmagadda, Daryn Sharp, Arun Murthy, Tsz-Wo Nicholas Sze, Suresh
> Srinivas and Sanjay Radia. There are many others who contributed as well
> providing feedback and comments on numerous jiras.
> 
> The vote will run for seven days and will end on March 5, 6:00PM PST.
> 
> Regards,
> Suresh
> 
> 
> 
> 
> On Thu, Feb 7, 2013 at 6:41 PM, Mahadevan Venkatraman
> <ma...@microsoft.com>wrote:
> 
>> It is super exciting to look at the prospect of these changes being merged
>> to trunk. Having Windows as one of the supported Hadoop platforms is a
>> fantastic opportunity both for the Hadoop project and Microsoft customers.
>> 
>> This work began around a year back when a few of us started with a basic
>> port of Hadoop on Windows. Ever since, the Hadoop team in Microsoft have
>> made significant progress in the following areas:
>> (PS: Some of these items are already included in Suresh's email, but
>> including again for completeness)
>> 
>> - Command-line scripts for the Hadoop surface area
>> - Mapping the HDFS permissions model to Windows
>> - Abstracted and reconciled mismatches around differences in Path
>> semantics in Java and Windows
>> - Native Task Controller for Windows
>> - Implementation of a Block Placement Policy to support cloud
>> environments, more specifically Azure.
>> - Implementation of Hadoop native libraries for Windows (compression
>> codecs, native I/O) - Several reliability issues, including
>> race-conditions, intermittent test failures, resource leaks.
>> - Several new unit test cases written for the above changes
>> 
>> In the process, we have closely engaged with the Apache open source
>> community and have got great support and assistance from the community in
>> terms of contributing fixes, code review comments and commits.
>> 
>> In addition, the Hadoop team at Microsoft has also made good progress in
>> other projects including Hive, Pig, Sqoop, Oozie, HCat and HBase. Many of
>> these changes have already been committed to the respective trunks with
>> help from various committers and contributors. It is great to see the
>> commitment of the community to support multiple platforms, and we look
>> forward to the day when a developer/customer is able to successfully deploy
>> a complete solution stack based on Apache Hadoop releases.
>> 
>> Next Steps:
>> 
>> All of the above changes are part of the Windows Azure HDInsight and
>> HDInsight Server products from Microsoft. We have successfully on-boarded
>> several internal customers and have been running production workloads on
>> Windows Azure HDInsight. Our vision is to create a big data platform based
>> on Hadoop, and we are committed to helping make Hadoop a world-class
>> solution that anyone can use to solve their biggest data challenges.
>> 
>> As an immediate next step, we would like to have a discussion around how
>> we can ensure that the quality of the mainline Hadoop branches on Windows
>> is maintained. To this end, we would like to get to the state where we have
>> pre-checkin validation gates and nightly test runs enabled on Windows. If
>> you have any suggestions around this, please do send an email.  We are
>> committed to helping sustain the long-term quality of Hadoop on both Linux
>> and Windows.
>> 
>> We sincerely thank the community for their contribution and support so
>> far. And hope to continue having a close engagement in the future.
>> 
>> -Microsoft HDInsight Team
>> 
>> 
>> -----Original Message-----
>> From: Suresh Srinivas [mailto:suresh@hortonworks.com]
>> Sent: Thursday, February 7, 2013 5:42 PM
>> To: common-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org;
>> hdfs-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
>> Subject: Heads up - merge branch-trunk-win to trunk
>> 
>> The support for Hadoop on Windows was proposed in HADOOP-8079<
>> https://issues.apache.org/jira/browse/HADOOP-8079> almost a year ago. The
>> goal was to make Hadoop natively integrated, full-featured, and performance
>> and scalability tuned on Windows Server or Windows Azure.
>> We are happy to announce that a lot of progress has been made in this
>> regard.
>> 
>> Initial work started in a feature branch, branch-1-win, based on branch-1.
>> The details related to the work done in the branch can be seen in
>> CHANGES.txt<
>> http://svn.apache.org/viewvc/hadoop/common/branches/branch-1-win/CHANGES.branch-1-win.txt?view=markup
>>> .
>> This work has been ported to a branch, branch-trunk-win, based on trunk.
>> Merge patch for this is available on
>> HADOOP-8562<https://issues.apache.org/jira/browse/HADOOP-8562>
>> .
>> 
>> Highlights of the work done so far:
>> 1. Necessary changes in Hadoop to run natively on Windows. These changes
>> handle differences in platforms related to path names, process/task
>> management etc.
>> 2. Addition of winutils tools for managing file permissions and ownership,
>> user group mapping, hardlinks, symbolic links, chmod, disk utilization, and
>> process/task management.
>> 3. Added cmd scripts equivalent to existing shell scripts
>> hadoop-daemon.sh, start and stop scripts.
>> 4. Addition of block placement policy implemnation to support cloud
>> enviroment, more specifically Azure.
>> 
>> We are very close to wrapping up the work in branch-trunk-win and getting
>> ready for a merge. Currently the merge patch is passing close to 100% of
>> unit tests on Linux. Soon I will call for a vote to merge this branch into
>> trunk.
>> 
>> Next steps:
>> 1. Call for vote to merge branch-trunk-win to trunk, when the work
>> completes and precommit build is clean.
>> 2. Start a discussion on adding Jenkins precommit builds on windows and
>> how to integrate that with the existing commit process.
>> 
>> Let me know if you have any questions.
>> 
>> Regards,
>> Suresh
>> 
>> 
> 
> 
> -- 
> http://hortonworks.com/download/

--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/

Re: [Vote] Merge branch-trunk-win to trunk

[sanjay Radia <sa...@hortonworks.com>]

+1
Java has done the bulk of the work in making Hadoop multi-platform.
Windows specific code is a tiny percentage of the code.
Jeninks support for windows is going help us keep the platform portable going forward.
I expect that the vast majority of new commits have  no problems. I propose that we start by fixing problems that Jenkins raises but not block new commits for too long if the author does not have a windows box or if a volunteer does not step up.

sanjay

RE: [Vote] Merge branch-trunk-win to trunk

[Uma Maheswara Rao G <ma...@huawei.com>]

+1 (non-binding)

 Thanks a lot for the work done by Suresh and team of community!
I don't think there will be much problems because of platform dependancy as our development is with Java. If we have some native code portings, then dev members has to take care of them.
One question regarding to it:
Ex: if one contributor is giving the patch for some native code porting for some performance improvements and he is interested only in Linux. Then I hope some other contributors will help in getting the windows patch if possible.
If others busy to get that done with in time lines, then I think we can commit Linux support patch and leave one JIRA open for Windows support? [ make sure that porting options configurable and platform check and give release note about platform support..etc?]

Regards,
Uma

________________________________________
From: Suresh Srinivas [suresh@hortonworks.com]
Sent: Wednesday, February 27, 2013 4:25 AM
To: common-dev@hadoop.apache.org
Cc: yarn-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
Subject: [Vote] Merge branch-trunk-win to trunk

I had posted heads up about merging branch-trunk-win to trunk on Feb 8th. I
am happy to announce that we are ready for the merge.

Here is a brief recap on the highlights of the work done:
- Command-line scripts for the Hadoop surface area
- Mapping the HDFS permissions model to Windows
- Abstracted and reconciled mismatches around differences in Path semantics
in Java and Windows
- Native Task Controller for Windows
- Implementation of a Block Placement Policy to support cloud environments,
more specifically Azure.
- Implementation of Hadoop native libraries for Windows (compression
codecs, native I/O)
- Several reliability issues, including race-conditions, intermittent test
failures, resource leaks.
- Several new unit test cases written for the above changes

Please find the details of the work in CHANGES.branch-trunk-win.txt -
Common changes<http://bit.ly/Xe7Ynv>, HDFS changes<http://bit.ly/13QOSo9>,
and YARN and MapReduce changes <http://bit.ly/128zzMt>. This is the work
ported from branch-1-win to a branch based on trunk.

For details of the testing done, please see the thread -
http://bit.ly/WpavJ4. Merge patch for this is available on HADOOP-8562<
https://issues.apache.org/jira/browse/HADOOP-8562>.

This was a large undertaking that involved developing code, testing the
entire Hadoop stack, including scale tests. This is made possible only with
the contribution from many many folks in the community. Following people
contributed to this work: Ivan Mitic, Chuan Liu, Ramya Sunil, Bikas Saha,
Kanna Karanam, John Gordon, Brandon Li, Chris Nauroth, David Lao, Sumadhur
Reddy Bolli, Arpit Agarwal, Ahmed El Baz, Mike Liddell, Jing Zhao, Thejas
Nair, Steve Maine, Ganeshan Iyer, Raja Aluri, Giridharan Kesavan, Ramya
Bharathi Nimmagadda, Daryn Sharp, Arun Murthy, Tsz-Wo Nicholas Sze, Suresh
Srinivas and Sanjay Radia. There are many others who contributed as well
providing feedback and comments on numerous jiras.

The vote will run for seven days and will end on March 5, 6:00PM PST.

Regards,
Suresh




On Thu, Feb 7, 2013 at 6:41 PM, Mahadevan Venkatraman
<ma...@microsoft.com>wrote:

> It is super exciting to look at the prospect of these changes being merged
> to trunk. Having Windows as one of the supported Hadoop platforms is a
> fantastic opportunity both for the Hadoop project and Microsoft customers.
>
> This work began around a year back when a few of us started with a basic
> port of Hadoop on Windows. Ever since, the Hadoop team in Microsoft have
> made significant progress in the following areas:
> (PS: Some of these items are already included in Suresh's email, but
> including again for completeness)
>
> - Command-line scripts for the Hadoop surface area
> - Mapping the HDFS permissions model to Windows
> - Abstracted and reconciled mismatches around differences in Path
> semantics in Java and Windows
> - Native Task Controller for Windows
> - Implementation of a Block Placement Policy to support cloud
> environments, more specifically Azure.
> - Implementation of Hadoop native libraries for Windows (compression
> codecs, native I/O) - Several reliability issues, including
> race-conditions, intermittent test failures, resource leaks.
> - Several new unit test cases written for the above changes
>
> In the process, we have closely engaged with the Apache open source
> community and have got great support and assistance from the community in
> terms of contributing fixes, code review comments and commits.
>
> In addition, the Hadoop team at Microsoft has also made good progress in
> other projects including Hive, Pig, Sqoop, Oozie, HCat and HBase. Many of
> these changes have already been committed to the respective trunks with
> help from various committers and contributors. It is great to see the
> commitment of the community to support multiple platforms, and we look
> forward to the day when a developer/customer is able to successfully deploy
> a complete solution stack based on Apache Hadoop releases.
>
> Next Steps:
>
> All of the above changes are part of the Windows Azure HDInsight and
> HDInsight Server products from Microsoft. We have successfully on-boarded
> several internal customers and have been running production workloads on
> Windows Azure HDInsight. Our vision is to create a big data platform based
> on Hadoop, and we are committed to helping make Hadoop a world-class
> solution that anyone can use to solve their biggest data challenges.
>
> As an immediate next step, we would like to have a discussion around how
> we can ensure that the quality of the mainline Hadoop branches on Windows
> is maintained. To this end, we would like to get to the state where we have
> pre-checkin validation gates and nightly test runs enabled on Windows. If
> you have any suggestions around this, please do send an email.  We are
> committed to helping sustain the long-term quality of Hadoop on both Linux
> and Windows.
>
> We sincerely thank the community for their contribution and support so
> far. And hope to continue having a close engagement in the future.
>
> -Microsoft HDInsight Team
>
>
> -----Original Message-----
> From: Suresh Srinivas [mailto:suresh@hortonworks.com]
> Sent: Thursday, February 7, 2013 5:42 PM
> To: common-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org;
> hdfs-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
> Subject: Heads up - merge branch-trunk-win to trunk
>
> The support for Hadoop on Windows was proposed in HADOOP-8079<
> https://issues.apache.org/jira/browse/HADOOP-8079> almost a year ago. The
> goal was to make Hadoop natively integrated, full-featured, and performance
> and scalability tuned on Windows Server or Windows Azure.
> We are happy to announce that a lot of progress has been made in this
> regard.
>
> Initial work started in a feature branch, branch-1-win, based on branch-1.
> The details related to the work done in the branch can be seen in
> CHANGES.txt<
> http://svn.apache.org/viewvc/hadoop/common/branches/branch-1-win/CHANGES.branch-1-win.txt?view=markup
> >.
> This work has been ported to a branch, branch-trunk-win, based on trunk.
> Merge patch for this is available on
> HADOOP-8562<https://issues.apache.org/jira/browse/HADOOP-8562>
> .
>
> Highlights of the work done so far:
> 1. Necessary changes in Hadoop to run natively on Windows. These changes
> handle differences in platforms related to path names, process/task
> management etc.
> 2. Addition of winutils tools for managing file permissions and ownership,
> user group mapping, hardlinks, symbolic links, chmod, disk utilization, and
> process/task management.
> 3. Added cmd scripts equivalent to existing shell scripts
> hadoop-daemon.sh, start and stop scripts.
> 4. Addition of block placement policy implemnation to support cloud
> enviroment, more specifically Azure.
>
> We are very close to wrapping up the work in branch-trunk-win and getting
> ready for a merge. Currently the merge patch is passing close to 100% of
> unit tests on Linux. Soon I will call for a vote to merge this branch into
> trunk.
>
> Next steps:
> 1. Call for vote to merge branch-trunk-win to trunk, when the work
> completes and precommit build is clean.
> 2. Start a discussion on adding Jenkins precommit builds on windows and
> how to integrate that with the existing commit process.
>
> Let me know if you have any questions.
>
> Regards,
> Suresh
>
>


--
http://hortonworks.com/download/

Re: [Vote] Merge branch-trunk-win to trunk

[sanjay Radia <sa...@hortonworks.com>]

+1
Java has done the bulk of the work in making Hadoop multi-platform.
Windows specific code is a tiny percentage of the code.
Jeninks support for windows is going help us keep the platform portable going forward.
I expect that the vast majority of new commits have  no problems. I propose that we start by fixing problems that Jenkins raises but not block new commits for too long if the author does not have a windows box or if a volunteer does not step up.

sanjay

Re: [Vote] Merge branch-trunk-win to trunk

[Chris Nauroth <cn...@hortonworks.com>]

+1 (non-binding)

I've been testing and patching this branch for the past several months, and
I believe we've reached stability for a merge to trunk.  I want to point
out once again that the branch has been tested on Linux to build confidence
that regressions were not introduced on existing platforms.  I also want to
second the comments from Bikas about how wonderful the collaboration has
been!

Thank you,
--Chris


On Tue, Feb 26, 2013 at 4:30 PM, Bikas Saha <bi...@hortonworks.com> wrote:

> +1
>
> As someone who has been part of this effort from inception, I am glad that
> we have reached this stable state in the project on both branches of
> Hadoop.
> It has been a great collaboration across teams and engineers and opens up
> Hadoop to a whole new set of deployments and developers!
>
> Bikas
>
> -----Original Message-----
> From: Suresh Srinivas [mailto:suresh@hortonworks.com]
> Sent: Tuesday, February 26, 2013 2:56 PM
> To: common-dev@hadoop.apache.org
> Cc: yarn-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org;
> mapreduce-dev@hadoop.apache.org
> Subject: [Vote] Merge branch-trunk-win to trunk
>
> I had posted heads up about merging branch-trunk-win to trunk on Feb 8th.
> I am happy to announce that we are ready for the merge.
>
> Here is a brief recap on the highlights of the work done:
> - Command-line scripts for the Hadoop surface area
> - Mapping the HDFS permissions model to Windows
> - Abstracted and reconciled mismatches around differences in Path
> semantics in Java and Windows
> - Native Task Controller for Windows
> - Implementation of a Block Placement Policy to support cloud
> environments, more specifically Azure.
> - Implementation of Hadoop native libraries for Windows (compression
> codecs, native I/O)
> - Several reliability issues, including race-conditions, intermittent test
> failures, resource leaks.
> - Several new unit test cases written for the above changes
>
> Please find the details of the work in CHANGES.branch-trunk-win.txt -
> Common changes<http://bit.ly/Xe7Ynv>, HDFS changes<http://bit.ly/13QOSo9>,
> and YARN and MapReduce changes <http://bit.ly/128zzMt>. This is the work
> ported from branch-1-win to a branch based on trunk.
>
> For details of the testing done, please see the thread -
> http://bit.ly/WpavJ4. Merge patch for this is available on HADOOP-8562<
> https://issues.apache.org/jira/browse/HADOOP-8562>.
>
> This was a large undertaking that involved developing code, testing the
> entire Hadoop stack, including scale tests. This is made possible only
> with the contribution from many many folks in the community. Following
> people contributed to this work: Ivan Mitic, Chuan Liu, Ramya Sunil, Bikas
> Saha, Kanna Karanam, John Gordon, Brandon Li, Chris Nauroth, David Lao,
> Sumadhur Reddy Bolli, Arpit Agarwal, Ahmed El Baz, Mike Liddell, Jing
> Zhao, Thejas Nair, Steve Maine, Ganeshan Iyer, Raja Aluri, Giridharan
> Kesavan, Ramya Bharathi Nimmagadda, Daryn Sharp, Arun Murthy, Tsz-Wo
> Nicholas Sze, Suresh Srinivas and Sanjay Radia. There are many others who
> contributed as well providing feedback and comments on numerous jiras.
>
> The vote will run for seven days and will end on March 5, 6:00PM PST.
>
> Regards,
> Suresh
>
>
>
>
> On Thu, Feb 7, 2013 at 6:41 PM, Mahadevan Venkatraman
> <ma...@microsoft.com>wrote:
>
> > It is super exciting to look at the prospect of these changes being
> > merged to trunk. Having Windows as one of the supported Hadoop
> > platforms is a fantastic opportunity both for the Hadoop project and
> Microsoft customers.
> >
> > This work began around a year back when a few of us started with a
> > basic port of Hadoop on Windows. Ever since, the Hadoop team in
> > Microsoft have made significant progress in the following areas:
> > (PS: Some of these items are already included in Suresh's email, but
> > including again for completeness)
> >
> > - Command-line scripts for the Hadoop surface area
> > - Mapping the HDFS permissions model to Windows
> > - Abstracted and reconciled mismatches around differences in Path
> > semantics in Java and Windows
> > - Native Task Controller for Windows
> > - Implementation of a Block Placement Policy to support cloud
> > environments, more specifically Azure.
> > - Implementation of Hadoop native libraries for Windows (compression
> > codecs, native I/O) - Several reliability issues, including
> > race-conditions, intermittent test failures, resource leaks.
> > - Several new unit test cases written for the above changes
> >
> > In the process, we have closely engaged with the Apache open source
> > community and have got great support and assistance from the community
> > in terms of contributing fixes, code review comments and commits.
> >
> > In addition, the Hadoop team at Microsoft has also made good progress
> > in other projects including Hive, Pig, Sqoop, Oozie, HCat and HBase.
> > Many of these changes have already been committed to the respective
> > trunks with help from various committers and contributors. It is great
> > to see the commitment of the community to support multiple platforms,
> > and we look forward to the day when a developer/customer is able to
> > successfully deploy a complete solution stack based on Apache Hadoop
> releases.
> >
> > Next Steps:
> >
> > All of the above changes are part of the Windows Azure HDInsight and
> > HDInsight Server products from Microsoft. We have successfully
> > on-boarded several internal customers and have been running production
> > workloads on Windows Azure HDInsight. Our vision is to create a big
> > data platform based on Hadoop, and we are committed to helping make
> > Hadoop a world-class solution that anyone can use to solve their biggest
> data challenges.
> >
> > As an immediate next step, we would like to have a discussion around
> > how we can ensure that the quality of the mainline Hadoop branches on
> > Windows is maintained. To this end, we would like to get to the state
> > where we have pre-checkin validation gates and nightly test runs
> > enabled on Windows. If you have any suggestions around this, please do
> > send an email.  We are committed to helping sustain the long-term
> > quality of Hadoop on both Linux and Windows.
> >
> > We sincerely thank the community for their contribution and support so
> > far. And hope to continue having a close engagement in the future.
> >
> > -Microsoft HDInsight Team
> >
> >
> > -----Original Message-----
> > From: Suresh Srinivas [mailto:suresh@hortonworks.com]
> > Sent: Thursday, February 7, 2013 5:42 PM
> > To: common-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org;
> > hdfs-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
> > Subject: Heads up - merge branch-trunk-win to trunk
> >
> > The support for Hadoop on Windows was proposed in HADOOP-8079<
> > https://issues.apache.org/jira/browse/HADOOP-8079> almost a year ago.
> > The goal was to make Hadoop natively integrated, full-featured, and
> > performance and scalability tuned on Windows Server or Windows Azure.
> > We are happy to announce that a lot of progress has been made in this
> > regard.
> >
> > Initial work started in a feature branch, branch-1-win, based on
> branch-1.
> > The details related to the work done in the branch can be seen in
> > CHANGES.txt<
> > http://svn.apache.org/viewvc/hadoop/common/branches/branch-1-win/CHANG
> > ES.branch-1-win.txt?view=markup
> > >.
> > This work has been ported to a branch, branch-trunk-win, based on trunk.
> > Merge patch for this is available on
> > HADOOP-8562<https://issues.apache.org/jira/browse/HADOOP-8562>
> > .
> >
> > Highlights of the work done so far:
> > 1. Necessary changes in Hadoop to run natively on Windows. These
> > changes handle differences in platforms related to path names,
> > process/task management etc.
> > 2. Addition of winutils tools for managing file permissions and
> > ownership, user group mapping, hardlinks, symbolic links, chmod, disk
> > utilization, and process/task management.
> > 3. Added cmd scripts equivalent to existing shell scripts
> > hadoop-daemon.sh, start and stop scripts.
> > 4. Addition of block placement policy implemnation to support cloud
> > enviroment, more specifically Azure.
> >
> > We are very close to wrapping up the work in branch-trunk-win and
> > getting ready for a merge. Currently the merge patch is passing close
> > to 100% of unit tests on Linux. Soon I will call for a vote to merge
> > this branch into trunk.
> >
> > Next steps:
> > 1. Call for vote to merge branch-trunk-win to trunk, when the work
> > completes and precommit build is clean.
> > 2. Start a discussion on adding Jenkins precommit builds on windows
> > and how to integrate that with the existing commit process.
> >
> > Let me know if you have any questions.
> >
> > Regards,
> > Suresh
> >
> >
>
>
> --
> http://hortonworks.com/download/
>

RE: [Vote] Merge branch-trunk-win to trunk

[Bikas Saha <bi...@hortonworks.com>]

+1

As someone who has been part of this effort from inception, I am glad that
we have reached this stable state in the project on both branches of
Hadoop.
It has been a great collaboration across teams and engineers and opens up
Hadoop to a whole new set of deployments and developers!

Bikas

-----Original Message-----
From: Suresh Srinivas [mailto:suresh@hortonworks.com]
Sent: Tuesday, February 26, 2013 2:56 PM
To: common-dev@hadoop.apache.org
Cc: yarn-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org;
mapreduce-dev@hadoop.apache.org
Subject: [Vote] Merge branch-trunk-win to trunk

I had posted heads up about merging branch-trunk-win to trunk on Feb 8th.
I am happy to announce that we are ready for the merge.

Here is a brief recap on the highlights of the work done:
- Command-line scripts for the Hadoop surface area
- Mapping the HDFS permissions model to Windows
- Abstracted and reconciled mismatches around differences in Path
semantics in Java and Windows
- Native Task Controller for Windows
- Implementation of a Block Placement Policy to support cloud
environments, more specifically Azure.
- Implementation of Hadoop native libraries for Windows (compression
codecs, native I/O)
- Several reliability issues, including race-conditions, intermittent test
failures, resource leaks.
- Several new unit test cases written for the above changes

Please find the details of the work in CHANGES.branch-trunk-win.txt -
Common changes<http://bit.ly/Xe7Ynv>, HDFS changes<http://bit.ly/13QOSo9>,
and YARN and MapReduce changes <http://bit.ly/128zzMt>. This is the work
ported from branch-1-win to a branch based on trunk.

For details of the testing done, please see the thread -
http://bit.ly/WpavJ4. Merge patch for this is available on HADOOP-8562<
https://issues.apache.org/jira/browse/HADOOP-8562>.

This was a large undertaking that involved developing code, testing the
entire Hadoop stack, including scale tests. This is made possible only
with the contribution from many many folks in the community. Following
people contributed to this work: Ivan Mitic, Chuan Liu, Ramya Sunil, Bikas
Saha, Kanna Karanam, John Gordon, Brandon Li, Chris Nauroth, David Lao,
Sumadhur Reddy Bolli, Arpit Agarwal, Ahmed El Baz, Mike Liddell, Jing
Zhao, Thejas Nair, Steve Maine, Ganeshan Iyer, Raja Aluri, Giridharan
Kesavan, Ramya Bharathi Nimmagadda, Daryn Sharp, Arun Murthy, Tsz-Wo
Nicholas Sze, Suresh Srinivas and Sanjay Radia. There are many others who
contributed as well providing feedback and comments on numerous jiras.

The vote will run for seven days and will end on March 5, 6:00PM PST.

Regards,
Suresh




On Thu, Feb 7, 2013 at 6:41 PM, Mahadevan Venkatraman
<ma...@microsoft.com>wrote:

> It is super exciting to look at the prospect of these changes being
> merged to trunk. Having Windows as one of the supported Hadoop
> platforms is a fantastic opportunity both for the Hadoop project and
Microsoft customers.
>
> This work began around a year back when a few of us started with a
> basic port of Hadoop on Windows. Ever since, the Hadoop team in
> Microsoft have made significant progress in the following areas:
> (PS: Some of these items are already included in Suresh's email, but
> including again for completeness)
>
> - Command-line scripts for the Hadoop surface area
> - Mapping the HDFS permissions model to Windows
> - Abstracted and reconciled mismatches around differences in Path
> semantics in Java and Windows
> - Native Task Controller for Windows
> - Implementation of a Block Placement Policy to support cloud
> environments, more specifically Azure.
> - Implementation of Hadoop native libraries for Windows (compression
> codecs, native I/O) - Several reliability issues, including
> race-conditions, intermittent test failures, resource leaks.
> - Several new unit test cases written for the above changes
>
> In the process, we have closely engaged with the Apache open source
> community and have got great support and assistance from the community
> in terms of contributing fixes, code review comments and commits.
>
> In addition, the Hadoop team at Microsoft has also made good progress
> in other projects including Hive, Pig, Sqoop, Oozie, HCat and HBase.
> Many of these changes have already been committed to the respective
> trunks with help from various committers and contributors. It is great
> to see the commitment of the community to support multiple platforms,
> and we look forward to the day when a developer/customer is able to
> successfully deploy a complete solution stack based on Apache Hadoop
releases.
>
> Next Steps:
>
> All of the above changes are part of the Windows Azure HDInsight and
> HDInsight Server products from Microsoft. We have successfully
> on-boarded several internal customers and have been running production
> workloads on Windows Azure HDInsight. Our vision is to create a big
> data platform based on Hadoop, and we are committed to helping make
> Hadoop a world-class solution that anyone can use to solve their biggest
data challenges.
>
> As an immediate next step, we would like to have a discussion around
> how we can ensure that the quality of the mainline Hadoop branches on
> Windows is maintained. To this end, we would like to get to the state
> where we have pre-checkin validation gates and nightly test runs
> enabled on Windows. If you have any suggestions around this, please do
> send an email.  We are committed to helping sustain the long-term
> quality of Hadoop on both Linux and Windows.
>
> We sincerely thank the community for their contribution and support so
> far. And hope to continue having a close engagement in the future.
>
> -Microsoft HDInsight Team
>
>
> -----Original Message-----
> From: Suresh Srinivas [mailto:suresh@hortonworks.com]
> Sent: Thursday, February 7, 2013 5:42 PM
> To: common-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org;
> hdfs-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
> Subject: Heads up - merge branch-trunk-win to trunk
>
> The support for Hadoop on Windows was proposed in HADOOP-8079<
> https://issues.apache.org/jira/browse/HADOOP-8079> almost a year ago.
> The goal was to make Hadoop natively integrated, full-featured, and
> performance and scalability tuned on Windows Server or Windows Azure.
> We are happy to announce that a lot of progress has been made in this
> regard.
>
> Initial work started in a feature branch, branch-1-win, based on
branch-1.
> The details related to the work done in the branch can be seen in
> CHANGES.txt<
> http://svn.apache.org/viewvc/hadoop/common/branches/branch-1-win/CHANG
> ES.branch-1-win.txt?view=markup
> >.
> This work has been ported to a branch, branch-trunk-win, based on trunk.
> Merge patch for this is available on
> HADOOP-8562<https://issues.apache.org/jira/browse/HADOOP-8562>
> .
>
> Highlights of the work done so far:
> 1. Necessary changes in Hadoop to run natively on Windows. These
> changes handle differences in platforms related to path names,
> process/task management etc.
> 2. Addition of winutils tools for managing file permissions and
> ownership, user group mapping, hardlinks, symbolic links, chmod, disk
> utilization, and process/task management.
> 3. Added cmd scripts equivalent to existing shell scripts
> hadoop-daemon.sh, start and stop scripts.
> 4. Addition of block placement policy implemnation to support cloud
> enviroment, more specifically Azure.
>
> We are very close to wrapping up the work in branch-trunk-win and
> getting ready for a merge. Currently the merge patch is passing close
> to 100% of unit tests on Linux. Soon I will call for a vote to merge
> this branch into trunk.
>
> Next steps:
> 1. Call for vote to merge branch-trunk-win to trunk, when the work
> completes and precommit build is clean.
> 2. Start a discussion on adding Jenkins precommit builds on windows
> and how to integrate that with the existing commit process.
>
> Let me know if you have any questions.
>
> Regards,
> Suresh
>
>


--
http://hortonworks.com/download/

RE: [Vote] Merge branch-trunk-win to trunk

[Kanna Karanam <ka...@microsoft.com>]

+1 non-binding

I am playing with it for several months in a multi-node Windows cluster environment and found it is very stable. I am sure that it can help us to bring more developers like me (JAVA & Windows Developers) to contribute more and help the Hadoop customer & developer communities.

Thanks,
Kanna

-----Original Message-----
From: Raja Aluri [mailto:raja@cmbasics.com] 
Sent: Thursday, February 28, 2013 11:17 AM
To: common-dev@hadoop.apache.org
Cc: yarn-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
Subject: Re: [Vote] Merge branch-trunk-win to trunk

+1 non-binding
Nice to see that this work is going to trunk.

Raja  Aluri


On Tue, Feb 26, 2013 at 2:55 PM, Suresh Srinivas <su...@hortonworks.com>wrote:

> I had posted heads up about merging branch-trunk-win to trunk on Feb 
> 8th. I am happy to announce that we are ready for the merge.
>
> Here is a brief recap on the highlights of the work done:
> - Command-line scripts for the Hadoop surface area
> - Mapping the HDFS permissions model to Windows
> - Abstracted and reconciled mismatches around differences in Path 
> semantics in Java and Windows
> - Native Task Controller for Windows
> - Implementation of a Block Placement Policy to support cloud 
> environments, more specifically Azure.
> - Implementation of Hadoop native libraries for Windows (compression 
> codecs, native I/O)
> - Several reliability issues, including race-conditions, intermittent 
> test failures, resource leaks.
> - Several new unit test cases written for the above changes
>
> Please find the details of the work in CHANGES.branch-trunk-win.txt - 
> Common changes<http://bit.ly/Xe7Ynv>, HDFS 
> changes<http://bit.ly/13QOSo9>, and YARN and MapReduce changes 
> <http://bit.ly/128zzMt>. This is the work ported from branch-1-win to a branch based on trunk.
>
> For details of the testing done, please see the thread - 
> http://bit.ly/WpavJ4. Merge patch for this is available on 
> HADOOP-8562< https://issues.apache.org/jira/browse/HADOOP-8562>.
>
> This was a large undertaking that involved developing code, testing 
> the entire Hadoop stack, including scale tests. This is made possible 
> only with the contribution from many many folks in the community. 
> Following people contributed to this work: Ivan Mitic, Chuan Liu, 
> Ramya Sunil, Bikas Saha, Kanna Karanam, John Gordon, Brandon Li, Chris 
> Nauroth, David Lao, Sumadhur Reddy Bolli, Arpit Agarwal, Ahmed El Baz, 
> Mike Liddell, Jing Zhao, Thejas Nair, Steve Maine, Ganeshan Iyer, Raja 
> Aluri, Giridharan Kesavan, Ramya Bharathi Nimmagadda, Daryn Sharp, 
> Arun Murthy, Tsz-Wo Nicholas Sze, Suresh Srinivas and Sanjay Radia. 
> There are many others who contributed as well providing feedback and comments on numerous jiras.
>
> The vote will run for seven days and will end on March 5, 6:00PM PST.
>
> Regards,
> Suresh
>
>
>
>
> On Thu, Feb 7, 2013 at 6:41 PM, Mahadevan Venkatraman
> <ma...@microsoft.com>wrote:
>
> > It is super exciting to look at the prospect of these changes being
> merged
> > to trunk. Having Windows as one of the supported Hadoop platforms is 
> > a fantastic opportunity both for the Hadoop project and Microsoft
> customers.
> >
> > This work began around a year back when a few of us started with a 
> > basic port of Hadoop on Windows. Ever since, the Hadoop team in 
> > Microsoft have made significant progress in the following areas:
> > (PS: Some of these items are already included in Suresh's email, but 
> > including again for completeness)
> >
> > - Command-line scripts for the Hadoop surface area
> > - Mapping the HDFS permissions model to Windows
> > - Abstracted and reconciled mismatches around differences in Path 
> > semantics in Java and Windows
> > - Native Task Controller for Windows
> > - Implementation of a Block Placement Policy to support cloud 
> > environments, more specifically Azure.
> > - Implementation of Hadoop native libraries for Windows (compression 
> > codecs, native I/O) - Several reliability issues, including 
> > race-conditions, intermittent test failures, resource leaks.
> > - Several new unit test cases written for the above changes
> >
> > In the process, we have closely engaged with the Apache open source 
> > community and have got great support and assistance from the 
> > community in terms of contributing fixes, code review comments and commits.
> >
> > In addition, the Hadoop team at Microsoft has also made good 
> > progress in other projects including Hive, Pig, Sqoop, Oozie, HCat 
> > and HBase. Many of these changes have already been committed to the 
> > respective trunks with help from various committers and 
> > contributors. It is great to see the commitment of the community to 
> > support multiple platforms, and we look forward to the day when a 
> > developer/customer is able to successfully
> deploy
> > a complete solution stack based on Apache Hadoop releases.
> >
> > Next Steps:
> >
> > All of the above changes are part of the Windows Azure HDInsight and 
> > HDInsight Server products from Microsoft. We have successfully 
> > on-boarded several internal customers and have been running 
> > production workloads on Windows Azure HDInsight. Our vision is to 
> > create a big data platform
> based
> > on Hadoop, and we are committed to helping make Hadoop a world-class 
> > solution that anyone can use to solve their biggest data challenges.
> >
> > As an immediate next step, we would like to have a discussion around 
> > how we can ensure that the quality of the mainline Hadoop branches 
> > on Windows is maintained. To this end, we would like to get to the 
> > state where we
> have
> > pre-checkin validation gates and nightly test runs enabled on 
> > Windows. If you have any suggestions around this, please do send an 
> > email.  We are committed to helping sustain the long-term quality of 
> > Hadoop on both
> Linux
> > and Windows.
> >
> > We sincerely thank the community for their contribution and support 
> > so far. And hope to continue having a close engagement in the future.
> >
> > -Microsoft HDInsight Team
> >
> >
> > -----Original Message-----
> > From: Suresh Srinivas [mailto:suresh@hortonworks.com]
> > Sent: Thursday, February 7, 2013 5:42 PM
> > To: common-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org; 
> > hdfs-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
> > Subject: Heads up - merge branch-trunk-win to trunk
> >
> > The support for Hadoop on Windows was proposed in HADOOP-8079< 
> > https://issues.apache.org/jira/browse/HADOOP-8079> almost a year ago.
> The
> > goal was to make Hadoop natively integrated, full-featured, and
> performance
> > and scalability tuned on Windows Server or Windows Azure.
> > We are happy to announce that a lot of progress has been made in 
> > this regard.
> >
> > Initial work started in a feature branch, branch-1-win, based on
> branch-1.
> > The details related to the work done in the branch can be seen in 
> > CHANGES.txt<
> >
> http://svn.apache.org/viewvc/hadoop/common/branches/branch-1-win/CHANG
> ES.branch-1-win.txt?view=markup
> > >.
> > This work has been ported to a branch, branch-trunk-win, based on trunk.
> > Merge patch for this is available on 
> > HADOOP-8562<https://issues.apache.org/jira/browse/HADOOP-8562>
> > .
> >
> > Highlights of the work done so far:
> > 1. Necessary changes in Hadoop to run natively on Windows. These 
> > changes handle differences in platforms related to path names, 
> > process/task management etc.
> > 2. Addition of winutils tools for managing file permissions and
> ownership,
> > user group mapping, hardlinks, symbolic links, chmod, disk 
> > utilization,
> and
> > process/task management.
> > 3. Added cmd scripts equivalent to existing shell scripts 
> > hadoop-daemon.sh, start and stop scripts.
> > 4. Addition of block placement policy implemnation to support cloud 
> > enviroment, more specifically Azure.
> >
> > We are very close to wrapping up the work in branch-trunk-win and 
> > getting ready for a merge. Currently the merge patch is passing 
> > close to 100% of unit tests on Linux. Soon I will call for a vote to 
> > merge this branch
> into
> > trunk.
> >
> > Next steps:
> > 1. Call for vote to merge branch-trunk-win to trunk, when the work 
> > completes and precommit build is clean.
> > 2. Start a discussion on adding Jenkins precommit builds on windows 
> > and how to integrate that with the existing commit process.
> >
> > Let me know if you have any questions.
> >
> > Regards,
> > Suresh
> >
> >
>
>
> --
> http://hortonworks.com/download/
>

RE: [Vote] Merge branch-trunk-win to trunk

[Kanna Karanam <ka...@microsoft.com>]

+1 non-binding

I am playing with it for several months in a multi-node Windows cluster environment and found it is very stable. I am sure that it can help us to bring more developers like me (JAVA & Windows Developers) to contribute more and help the Hadoop customer & developer communities.

Thanks,
Kanna

-----Original Message-----
From: Raja Aluri [mailto:raja@cmbasics.com] 
Sent: Thursday, February 28, 2013 11:17 AM
To: common-dev@hadoop.apache.org
Cc: yarn-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
Subject: Re: [Vote] Merge branch-trunk-win to trunk

+1 non-binding
Nice to see that this work is going to trunk.

Raja  Aluri


On Tue, Feb 26, 2013 at 2:55 PM, Suresh Srinivas <su...@hortonworks.com>wrote:

> I had posted heads up about merging branch-trunk-win to trunk on Feb 
> 8th. I am happy to announce that we are ready for the merge.
>
> Here is a brief recap on the highlights of the work done:
> - Command-line scripts for the Hadoop surface area
> - Mapping the HDFS permissions model to Windows
> - Abstracted and reconciled mismatches around differences in Path 
> semantics in Java and Windows
> - Native Task Controller for Windows
> - Implementation of a Block Placement Policy to support cloud 
> environments, more specifically Azure.
> - Implementation of Hadoop native libraries for Windows (compression 
> codecs, native I/O)
> - Several reliability issues, including race-conditions, intermittent 
> test failures, resource leaks.
> - Several new unit test cases written for the above changes
>
> Please find the details of the work in CHANGES.branch-trunk-win.txt - 
> Common changes<http://bit.ly/Xe7Ynv>, HDFS 
> changes<http://bit.ly/13QOSo9>, and YARN and MapReduce changes 
> <http://bit.ly/128zzMt>. This is the work ported from branch-1-win to a branch based on trunk.
>
> For details of the testing done, please see the thread - 
> http://bit.ly/WpavJ4. Merge patch for this is available on 
> HADOOP-8562< https://issues.apache.org/jira/browse/HADOOP-8562>.
>
> This was a large undertaking that involved developing code, testing 
> the entire Hadoop stack, including scale tests. This is made possible 
> only with the contribution from many many folks in the community. 
> Following people contributed to this work: Ivan Mitic, Chuan Liu, 
> Ramya Sunil, Bikas Saha, Kanna Karanam, John Gordon, Brandon Li, Chris 
> Nauroth, David Lao, Sumadhur Reddy Bolli, Arpit Agarwal, Ahmed El Baz, 
> Mike Liddell, Jing Zhao, Thejas Nair, Steve Maine, Ganeshan Iyer, Raja 
> Aluri, Giridharan Kesavan, Ramya Bharathi Nimmagadda, Daryn Sharp, 
> Arun Murthy, Tsz-Wo Nicholas Sze, Suresh Srinivas and Sanjay Radia. 
> There are many others who contributed as well providing feedback and comments on numerous jiras.
>
> The vote will run for seven days and will end on March 5, 6:00PM PST.
>
> Regards,
> Suresh
>
>
>
>
> On Thu, Feb 7, 2013 at 6:41 PM, Mahadevan Venkatraman
> <ma...@microsoft.com>wrote:
>
> > It is super exciting to look at the prospect of these changes being
> merged
> > to trunk. Having Windows as one of the supported Hadoop platforms is 
> > a fantastic opportunity both for the Hadoop project and Microsoft
> customers.
> >
> > This work began around a year back when a few of us started with a 
> > basic port of Hadoop on Windows. Ever since, the Hadoop team in 
> > Microsoft have made significant progress in the following areas:
> > (PS: Some of these items are already included in Suresh's email, but 
> > including again for completeness)
> >
> > - Command-line scripts for the Hadoop surface area
> > - Mapping the HDFS permissions model to Windows
> > - Abstracted and reconciled mismatches around differences in Path 
> > semantics in Java and Windows
> > - Native Task Controller for Windows
> > - Implementation of a Block Placement Policy to support cloud 
> > environments, more specifically Azure.
> > - Implementation of Hadoop native libraries for Windows (compression 
> > codecs, native I/O) - Several reliability issues, including 
> > race-conditions, intermittent test failures, resource leaks.
> > - Several new unit test cases written for the above changes
> >
> > In the process, we have closely engaged with the Apache open source 
> > community and have got great support and assistance from the 
> > community in terms of contributing fixes, code review comments and commits.
> >
> > In addition, the Hadoop team at Microsoft has also made good 
> > progress in other projects including Hive, Pig, Sqoop, Oozie, HCat 
> > and HBase. Many of these changes have already been committed to the 
> > respective trunks with help from various committers and 
> > contributors. It is great to see the commitment of the community to 
> > support multiple platforms, and we look forward to the day when a 
> > developer/customer is able to successfully
> deploy
> > a complete solution stack based on Apache Hadoop releases.
> >
> > Next Steps:
> >
> > All of the above changes are part of the Windows Azure HDInsight and 
> > HDInsight Server products from Microsoft. We have successfully 
> > on-boarded several internal customers and have been running 
> > production workloads on Windows Azure HDInsight. Our vision is to 
> > create a big data platform
> based
> > on Hadoop, and we are committed to helping make Hadoop a world-class 
> > solution that anyone can use to solve their biggest data challenges.
> >
> > As an immediate next step, we would like to have a discussion around 
> > how we can ensure that the quality of the mainline Hadoop branches 
> > on Windows is maintained. To this end, we would like to get to the 
> > state where we
> have
> > pre-checkin validation gates and nightly test runs enabled on 
> > Windows. If you have any suggestions around this, please do send an 
> > email.  We are committed to helping sustain the long-term quality of 
> > Hadoop on both
> Linux
> > and Windows.
> >
> > We sincerely thank the community for their contribution and support 
> > so far. And hope to continue having a close engagement in the future.
> >
> > -Microsoft HDInsight Team
> >
> >
> > -----Original Message-----
> > From: Suresh Srinivas [mailto:suresh@hortonworks.com]
> > Sent: Thursday, February 7, 2013 5:42 PM
> > To: common-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org; 
> > hdfs-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
> > Subject: Heads up - merge branch-trunk-win to trunk
> >
> > The support for Hadoop on Windows was proposed in HADOOP-8079< 
> > https://issues.apache.org/jira/browse/HADOOP-8079> almost a year ago.
> The
> > goal was to make Hadoop natively integrated, full-featured, and
> performance
> > and scalability tuned on Windows Server or Windows Azure.
> > We are happy to announce that a lot of progress has been made in 
> > this regard.
> >
> > Initial work started in a feature branch, branch-1-win, based on
> branch-1.
> > The details related to the work done in the branch can be seen in 
> > CHANGES.txt<
> >
> http://svn.apache.org/viewvc/hadoop/common/branches/branch-1-win/CHANG
> ES.branch-1-win.txt?view=markup
> > >.
> > This work has been ported to a branch, branch-trunk-win, based on trunk.
> > Merge patch for this is available on 
> > HADOOP-8562<https://issues.apache.org/jira/browse/HADOOP-8562>
> > .
> >
> > Highlights of the work done so far:
> > 1. Necessary changes in Hadoop to run natively on Windows. These 
> > changes handle differences in platforms related to path names, 
> > process/task management etc.
> > 2. Addition of winutils tools for managing file permissions and
> ownership,
> > user group mapping, hardlinks, symbolic links, chmod, disk 
> > utilization,
> and
> > process/task management.
> > 3. Added cmd scripts equivalent to existing shell scripts 
> > hadoop-daemon.sh, start and stop scripts.
> > 4. Addition of block placement policy implemnation to support cloud 
> > enviroment, more specifically Azure.
> >
> > We are very close to wrapping up the work in branch-trunk-win and 
> > getting ready for a merge. Currently the merge patch is passing 
> > close to 100% of unit tests on Linux. Soon I will call for a vote to 
> > merge this branch
> into
> > trunk.
> >
> > Next steps:
> > 1. Call for vote to merge branch-trunk-win to trunk, when the work 
> > completes and precommit build is clean.
> > 2. Start a discussion on adding Jenkins precommit builds on windows 
> > and how to integrate that with the existing commit process.
> >
> > Let me know if you have any questions.
> >
> > Regards,
> > Suresh
> >
> >
>
>
> --
> http://hortonworks.com/download/
>

Re: [Vote] Merge branch-trunk-win to trunk

[Raja Aluri <ra...@cmbasics.com>]

+1 non-binding
Nice to see that this work is going to trunk.

Raja  Aluri


On Tue, Feb 26, 2013 at 2:55 PM, Suresh Srinivas <su...@hortonworks.com>wrote:

> I had posted heads up about merging branch-trunk-win to trunk on Feb 8th. I
> am happy to announce that we are ready for the merge.
>
> Here is a brief recap on the highlights of the work done:
> - Command-line scripts for the Hadoop surface area
> - Mapping the HDFS permissions model to Windows
> - Abstracted and reconciled mismatches around differences in Path semantics
> in Java and Windows
> - Native Task Controller for Windows
> - Implementation of a Block Placement Policy to support cloud environments,
> more specifically Azure.
> - Implementation of Hadoop native libraries for Windows (compression
> codecs, native I/O)
> - Several reliability issues, including race-conditions, intermittent test
> failures, resource leaks.
> - Several new unit test cases written for the above changes
>
> Please find the details of the work in CHANGES.branch-trunk-win.txt -
> Common changes<http://bit.ly/Xe7Ynv>, HDFS changes<http://bit.ly/13QOSo9>,
> and YARN and MapReduce changes <http://bit.ly/128zzMt>. This is the work
> ported from branch-1-win to a branch based on trunk.
>
> For details of the testing done, please see the thread -
> http://bit.ly/WpavJ4. Merge patch for this is available on HADOOP-8562<
> https://issues.apache.org/jira/browse/HADOOP-8562>.
>
> This was a large undertaking that involved developing code, testing the
> entire Hadoop stack, including scale tests. This is made possible only with
> the contribution from many many folks in the community. Following people
> contributed to this work: Ivan Mitic, Chuan Liu, Ramya Sunil, Bikas Saha,
> Kanna Karanam, John Gordon, Brandon Li, Chris Nauroth, David Lao, Sumadhur
> Reddy Bolli, Arpit Agarwal, Ahmed El Baz, Mike Liddell, Jing Zhao, Thejas
> Nair, Steve Maine, Ganeshan Iyer, Raja Aluri, Giridharan Kesavan, Ramya
> Bharathi Nimmagadda, Daryn Sharp, Arun Murthy, Tsz-Wo Nicholas Sze, Suresh
> Srinivas and Sanjay Radia. There are many others who contributed as well
> providing feedback and comments on numerous jiras.
>
> The vote will run for seven days and will end on March 5, 6:00PM PST.
>
> Regards,
> Suresh
>
>
>
>
> On Thu, Feb 7, 2013 at 6:41 PM, Mahadevan Venkatraman
> <ma...@microsoft.com>wrote:
>
> > It is super exciting to look at the prospect of these changes being
> merged
> > to trunk. Having Windows as one of the supported Hadoop platforms is a
> > fantastic opportunity both for the Hadoop project and Microsoft
> customers.
> >
> > This work began around a year back when a few of us started with a basic
> > port of Hadoop on Windows. Ever since, the Hadoop team in Microsoft have
> > made significant progress in the following areas:
> > (PS: Some of these items are already included in Suresh's email, but
> > including again for completeness)
> >
> > - Command-line scripts for the Hadoop surface area
> > - Mapping the HDFS permissions model to Windows
> > - Abstracted and reconciled mismatches around differences in Path
> > semantics in Java and Windows
> > - Native Task Controller for Windows
> > - Implementation of a Block Placement Policy to support cloud
> > environments, more specifically Azure.
> > - Implementation of Hadoop native libraries for Windows (compression
> > codecs, native I/O) - Several reliability issues, including
> > race-conditions, intermittent test failures, resource leaks.
> > - Several new unit test cases written for the above changes
> >
> > In the process, we have closely engaged with the Apache open source
> > community and have got great support and assistance from the community in
> > terms of contributing fixes, code review comments and commits.
> >
> > In addition, the Hadoop team at Microsoft has also made good progress in
> > other projects including Hive, Pig, Sqoop, Oozie, HCat and HBase. Many of
> > these changes have already been committed to the respective trunks with
> > help from various committers and contributors. It is great to see the
> > commitment of the community to support multiple platforms, and we look
> > forward to the day when a developer/customer is able to successfully
> deploy
> > a complete solution stack based on Apache Hadoop releases.
> >
> > Next Steps:
> >
> > All of the above changes are part of the Windows Azure HDInsight and
> > HDInsight Server products from Microsoft. We have successfully on-boarded
> > several internal customers and have been running production workloads on
> > Windows Azure HDInsight. Our vision is to create a big data platform
> based
> > on Hadoop, and we are committed to helping make Hadoop a world-class
> > solution that anyone can use to solve their biggest data challenges.
> >
> > As an immediate next step, we would like to have a discussion around how
> > we can ensure that the quality of the mainline Hadoop branches on Windows
> > is maintained. To this end, we would like to get to the state where we
> have
> > pre-checkin validation gates and nightly test runs enabled on Windows. If
> > you have any suggestions around this, please do send an email.  We are
> > committed to helping sustain the long-term quality of Hadoop on both
> Linux
> > and Windows.
> >
> > We sincerely thank the community for their contribution and support so
> > far. And hope to continue having a close engagement in the future.
> >
> > -Microsoft HDInsight Team
> >
> >
> > -----Original Message-----
> > From: Suresh Srinivas [mailto:suresh@hortonworks.com]
> > Sent: Thursday, February 7, 2013 5:42 PM
> > To: common-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org;
> > hdfs-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
> > Subject: Heads up - merge branch-trunk-win to trunk
> >
> > The support for Hadoop on Windows was proposed in HADOOP-8079<
> > https://issues.apache.org/jira/browse/HADOOP-8079> almost a year ago.
> The
> > goal was to make Hadoop natively integrated, full-featured, and
> performance
> > and scalability tuned on Windows Server or Windows Azure.
> > We are happy to announce that a lot of progress has been made in this
> > regard.
> >
> > Initial work started in a feature branch, branch-1-win, based on
> branch-1.
> > The details related to the work done in the branch can be seen in
> > CHANGES.txt<
> >
> http://svn.apache.org/viewvc/hadoop/common/branches/branch-1-win/CHANGES.branch-1-win.txt?view=markup
> > >.
> > This work has been ported to a branch, branch-trunk-win, based on trunk.
> > Merge patch for this is available on
> > HADOOP-8562<https://issues.apache.org/jira/browse/HADOOP-8562>
> > .
> >
> > Highlights of the work done so far:
> > 1. Necessary changes in Hadoop to run natively on Windows. These changes
> > handle differences in platforms related to path names, process/task
> > management etc.
> > 2. Addition of winutils tools for managing file permissions and
> ownership,
> > user group mapping, hardlinks, symbolic links, chmod, disk utilization,
> and
> > process/task management.
> > 3. Added cmd scripts equivalent to existing shell scripts
> > hadoop-daemon.sh, start and stop scripts.
> > 4. Addition of block placement policy implemnation to support cloud
> > enviroment, more specifically Azure.
> >
> > We are very close to wrapping up the work in branch-trunk-win and getting
> > ready for a merge. Currently the merge patch is passing close to 100% of
> > unit tests on Linux. Soon I will call for a vote to merge this branch
> into
> > trunk.
> >
> > Next steps:
> > 1. Call for vote to merge branch-trunk-win to trunk, when the work
> > completes and precommit build is clean.
> > 2. Start a discussion on adding Jenkins precommit builds on windows and
> > how to integrate that with the existing commit process.
> >
> > Let me know if you have any questions.
> >
> > Regards,
> > Suresh
> >
> >
>
>
> --
> http://hortonworks.com/download/
>

RE: [Vote] Merge branch-trunk-win to trunk

[Uma Maheswara Rao G <ma...@huawei.com>]

+1 (non-binding)

 Thanks a lot for the work done by Suresh and team of community!
I don't think there will be much problems because of platform dependancy as our development is with Java. If we have some native code portings, then dev members has to take care of them.
One question regarding to it:
Ex: if one contributor is giving the patch for some native code porting for some performance improvements and he is interested only in Linux. Then I hope some other contributors will help in getting the windows patch if possible.
If others busy to get that done with in time lines, then I think we can commit Linux support patch and leave one JIRA open for Windows support? [ make sure that porting options configurable and platform check and give release note about platform support..etc?]

Regards,
Uma

________________________________________
From: Suresh Srinivas [suresh@hortonworks.com]
Sent: Wednesday, February 27, 2013 4:25 AM
To: common-dev@hadoop.apache.org
Cc: yarn-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
Subject: [Vote] Merge branch-trunk-win to trunk

I had posted heads up about merging branch-trunk-win to trunk on Feb 8th. I
am happy to announce that we are ready for the merge.

Here is a brief recap on the highlights of the work done:
- Command-line scripts for the Hadoop surface area
- Mapping the HDFS permissions model to Windows
- Abstracted and reconciled mismatches around differences in Path semantics
in Java and Windows
- Native Task Controller for Windows
- Implementation of a Block Placement Policy to support cloud environments,
more specifically Azure.
- Implementation of Hadoop native libraries for Windows (compression
codecs, native I/O)
- Several reliability issues, including race-conditions, intermittent test
failures, resource leaks.
- Several new unit test cases written for the above changes

Please find the details of the work in CHANGES.branch-trunk-win.txt -
Common changes<http://bit.ly/Xe7Ynv>, HDFS changes<http://bit.ly/13QOSo9>,
and YARN and MapReduce changes <http://bit.ly/128zzMt>. This is the work
ported from branch-1-win to a branch based on trunk.

For details of the testing done, please see the thread -
http://bit.ly/WpavJ4. Merge patch for this is available on HADOOP-8562<
https://issues.apache.org/jira/browse/HADOOP-8562>.

This was a large undertaking that involved developing code, testing the
entire Hadoop stack, including scale tests. This is made possible only with
the contribution from many many folks in the community. Following people
contributed to this work: Ivan Mitic, Chuan Liu, Ramya Sunil, Bikas Saha,
Kanna Karanam, John Gordon, Brandon Li, Chris Nauroth, David Lao, Sumadhur
Reddy Bolli, Arpit Agarwal, Ahmed El Baz, Mike Liddell, Jing Zhao, Thejas
Nair, Steve Maine, Ganeshan Iyer, Raja Aluri, Giridharan Kesavan, Ramya
Bharathi Nimmagadda, Daryn Sharp, Arun Murthy, Tsz-Wo Nicholas Sze, Suresh
Srinivas and Sanjay Radia. There are many others who contributed as well
providing feedback and comments on numerous jiras.

The vote will run for seven days and will end on March 5, 6:00PM PST.

Regards,
Suresh




On Thu, Feb 7, 2013 at 6:41 PM, Mahadevan Venkatraman
<ma...@microsoft.com>wrote:

> It is super exciting to look at the prospect of these changes being merged
> to trunk. Having Windows as one of the supported Hadoop platforms is a
> fantastic opportunity both for the Hadoop project and Microsoft customers.
>
> This work began around a year back when a few of us started with a basic
> port of Hadoop on Windows. Ever since, the Hadoop team in Microsoft have
> made significant progress in the following areas:
> (PS: Some of these items are already included in Suresh's email, but
> including again for completeness)
>
> - Command-line scripts for the Hadoop surface area
> - Mapping the HDFS permissions model to Windows
> - Abstracted and reconciled mismatches around differences in Path
> semantics in Java and Windows
> - Native Task Controller for Windows
> - Implementation of a Block Placement Policy to support cloud
> environments, more specifically Azure.
> - Implementation of Hadoop native libraries for Windows (compression
> codecs, native I/O) - Several reliability issues, including
> race-conditions, intermittent test failures, resource leaks.
> - Several new unit test cases written for the above changes
>
> In the process, we have closely engaged with the Apache open source
> community and have got great support and assistance from the community in
> terms of contributing fixes, code review comments and commits.
>
> In addition, the Hadoop team at Microsoft has also made good progress in
> other projects including Hive, Pig, Sqoop, Oozie, HCat and HBase. Many of
> these changes have already been committed to the respective trunks with
> help from various committers and contributors. It is great to see the
> commitment of the community to support multiple platforms, and we look
> forward to the day when a developer/customer is able to successfully deploy
> a complete solution stack based on Apache Hadoop releases.
>
> Next Steps:
>
> All of the above changes are part of the Windows Azure HDInsight and
> HDInsight Server products from Microsoft. We have successfully on-boarded
> several internal customers and have been running production workloads on
> Windows Azure HDInsight. Our vision is to create a big data platform based
> on Hadoop, and we are committed to helping make Hadoop a world-class
> solution that anyone can use to solve their biggest data challenges.
>
> As an immediate next step, we would like to have a discussion around how
> we can ensure that the quality of the mainline Hadoop branches on Windows
> is maintained. To this end, we would like to get to the state where we have
> pre-checkin validation gates and nightly test runs enabled on Windows. If
> you have any suggestions around this, please do send an email.  We are
> committed to helping sustain the long-term quality of Hadoop on both Linux
> and Windows.
>
> We sincerely thank the community for their contribution and support so
> far. And hope to continue having a close engagement in the future.
>
> -Microsoft HDInsight Team
>
>
> -----Original Message-----
> From: Suresh Srinivas [mailto:suresh@hortonworks.com]
> Sent: Thursday, February 7, 2013 5:42 PM
> To: common-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org;
> hdfs-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
> Subject: Heads up - merge branch-trunk-win to trunk
>
> The support for Hadoop on Windows was proposed in HADOOP-8079<
> https://issues.apache.org/jira/browse/HADOOP-8079> almost a year ago. The
> goal was to make Hadoop natively integrated, full-featured, and performance
> and scalability tuned on Windows Server or Windows Azure.
> We are happy to announce that a lot of progress has been made in this
> regard.
>
> Initial work started in a feature branch, branch-1-win, based on branch-1.
> The details related to the work done in the branch can be seen in
> CHANGES.txt<
> http://svn.apache.org/viewvc/hadoop/common/branches/branch-1-win/CHANGES.branch-1-win.txt?view=markup
> >.
> This work has been ported to a branch, branch-trunk-win, based on trunk.
> Merge patch for this is available on
> HADOOP-8562<https://issues.apache.org/jira/browse/HADOOP-8562>
> .
>
> Highlights of the work done so far:
> 1. Necessary changes in Hadoop to run natively on Windows. These changes
> handle differences in platforms related to path names, process/task
> management etc.
> 2. Addition of winutils tools for managing file permissions and ownership,
> user group mapping, hardlinks, symbolic links, chmod, disk utilization, and
> process/task management.
> 3. Added cmd scripts equivalent to existing shell scripts
> hadoop-daemon.sh, start and stop scripts.
> 4. Addition of block placement policy implemnation to support cloud
> enviroment, more specifically Azure.
>
> We are very close to wrapping up the work in branch-trunk-win and getting
> ready for a merge. Currently the merge patch is passing close to 100% of
> unit tests on Linux. Soon I will call for a vote to merge this branch into
> trunk.
>
> Next steps:
> 1. Call for vote to merge branch-trunk-win to trunk, when the work
> completes and precommit build is clean.
> 2. Start a discussion on adding Jenkins precommit builds on windows and
> how to integrate that with the existing commit process.
>
> Let me know if you have any questions.
>
> Regards,
> Suresh
>
>


--
http://hortonworks.com/download/

Re: [Vote] Merge branch-trunk-win to trunk

[Raja Aluri <ra...@cmbasics.com>]

+1 non-binding
Nice to see that this work is going to trunk.

Raja  Aluri


On Tue, Feb 26, 2013 at 2:55 PM, Suresh Srinivas <su...@hortonworks.com>wrote:

> I had posted heads up about merging branch-trunk-win to trunk on Feb 8th. I
> am happy to announce that we are ready for the merge.
>
> Here is a brief recap on the highlights of the work done:
> - Command-line scripts for the Hadoop surface area
> - Mapping the HDFS permissions model to Windows
> - Abstracted and reconciled mismatches around differences in Path semantics
> in Java and Windows
> - Native Task Controller for Windows
> - Implementation of a Block Placement Policy to support cloud environments,
> more specifically Azure.
> - Implementation of Hadoop native libraries for Windows (compression
> codecs, native I/O)
> - Several reliability issues, including race-conditions, intermittent test
> failures, resource leaks.
> - Several new unit test cases written for the above changes
>
> Please find the details of the work in CHANGES.branch-trunk-win.txt -
> Common changes<http://bit.ly/Xe7Ynv>, HDFS changes<http://bit.ly/13QOSo9>,
> and YARN and MapReduce changes <http://bit.ly/128zzMt>. This is the work
> ported from branch-1-win to a branch based on trunk.
>
> For details of the testing done, please see the thread -
> http://bit.ly/WpavJ4. Merge patch for this is available on HADOOP-8562<
> https://issues.apache.org/jira/browse/HADOOP-8562>.
>
> This was a large undertaking that involved developing code, testing the
> entire Hadoop stack, including scale tests. This is made possible only with
> the contribution from many many folks in the community. Following people
> contributed to this work: Ivan Mitic, Chuan Liu, Ramya Sunil, Bikas Saha,
> Kanna Karanam, John Gordon, Brandon Li, Chris Nauroth, David Lao, Sumadhur
> Reddy Bolli, Arpit Agarwal, Ahmed El Baz, Mike Liddell, Jing Zhao, Thejas
> Nair, Steve Maine, Ganeshan Iyer, Raja Aluri, Giridharan Kesavan, Ramya
> Bharathi Nimmagadda, Daryn Sharp, Arun Murthy, Tsz-Wo Nicholas Sze, Suresh
> Srinivas and Sanjay Radia. There are many others who contributed as well
> providing feedback and comments on numerous jiras.
>
> The vote will run for seven days and will end on March 5, 6:00PM PST.
>
> Regards,
> Suresh
>
>
>
>
> On Thu, Feb 7, 2013 at 6:41 PM, Mahadevan Venkatraman
> <ma...@microsoft.com>wrote:
>
> > It is super exciting to look at the prospect of these changes being
> merged
> > to trunk. Having Windows as one of the supported Hadoop platforms is a
> > fantastic opportunity both for the Hadoop project and Microsoft
> customers.
> >
> > This work began around a year back when a few of us started with a basic
> > port of Hadoop on Windows. Ever since, the Hadoop team in Microsoft have
> > made significant progress in the following areas:
> > (PS: Some of these items are already included in Suresh's email, but
> > including again for completeness)
> >
> > - Command-line scripts for the Hadoop surface area
> > - Mapping the HDFS permissions model to Windows
> > - Abstracted and reconciled mismatches around differences in Path
> > semantics in Java and Windows
> > - Native Task Controller for Windows
> > - Implementation of a Block Placement Policy to support cloud
> > environments, more specifically Azure.
> > - Implementation of Hadoop native libraries for Windows (compression
> > codecs, native I/O) - Several reliability issues, including
> > race-conditions, intermittent test failures, resource leaks.
> > - Several new unit test cases written for the above changes
> >
> > In the process, we have closely engaged with the Apache open source
> > community and have got great support and assistance from the community in
> > terms of contributing fixes, code review comments and commits.
> >
> > In addition, the Hadoop team at Microsoft has also made good progress in
> > other projects including Hive, Pig, Sqoop, Oozie, HCat and HBase. Many of
> > these changes have already been committed to the respective trunks with
> > help from various committers and contributors. It is great to see the
> > commitment of the community to support multiple platforms, and we look
> > forward to the day when a developer/customer is able to successfully
> deploy
> > a complete solution stack based on Apache Hadoop releases.
> >
> > Next Steps:
> >
> > All of the above changes are part of the Windows Azure HDInsight and
> > HDInsight Server products from Microsoft. We have successfully on-boarded
> > several internal customers and have been running production workloads on
> > Windows Azure HDInsight. Our vision is to create a big data platform
> based
> > on Hadoop, and we are committed to helping make Hadoop a world-class
> > solution that anyone can use to solve their biggest data challenges.
> >
> > As an immediate next step, we would like to have a discussion around how
> > we can ensure that the quality of the mainline Hadoop branches on Windows
> > is maintained. To this end, we would like to get to the state where we
> have
> > pre-checkin validation gates and nightly test runs enabled on Windows. If
> > you have any suggestions around this, please do send an email.  We are
> > committed to helping sustain the long-term quality of Hadoop on both
> Linux
> > and Windows.
> >
> > We sincerely thank the community for their contribution and support so
> > far. And hope to continue having a close engagement in the future.
> >
> > -Microsoft HDInsight Team
> >
> >
> > -----Original Message-----
> > From: Suresh Srinivas [mailto:suresh@hortonworks.com]
> > Sent: Thursday, February 7, 2013 5:42 PM
> > To: common-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org;
> > hdfs-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
> > Subject: Heads up - merge branch-trunk-win to trunk
> >
> > The support for Hadoop on Windows was proposed in HADOOP-8079<
> > https://issues.apache.org/jira/browse/HADOOP-8079> almost a year ago.
> The
> > goal was to make Hadoop natively integrated, full-featured, and
> performance
> > and scalability tuned on Windows Server or Windows Azure.
> > We are happy to announce that a lot of progress has been made in this
> > regard.
> >
> > Initial work started in a feature branch, branch-1-win, based on
> branch-1.
> > The details related to the work done in the branch can be seen in
> > CHANGES.txt<
> >
> http://svn.apache.org/viewvc/hadoop/common/branches/branch-1-win/CHANGES.branch-1-win.txt?view=markup
> > >.
> > This work has been ported to a branch, branch-trunk-win, based on trunk.
> > Merge patch for this is available on
> > HADOOP-8562<https://issues.apache.org/jira/browse/HADOOP-8562>
> > .
> >
> > Highlights of the work done so far:
> > 1. Necessary changes in Hadoop to run natively on Windows. These changes
> > handle differences in platforms related to path names, process/task
> > management etc.
> > 2. Addition of winutils tools for managing file permissions and
> ownership,
> > user group mapping, hardlinks, symbolic links, chmod, disk utilization,
> and
> > process/task management.
> > 3. Added cmd scripts equivalent to existing shell scripts
> > hadoop-daemon.sh, start and stop scripts.
> > 4. Addition of block placement policy implemnation to support cloud
> > enviroment, more specifically Azure.
> >
> > We are very close to wrapping up the work in branch-trunk-win and getting
> > ready for a merge. Currently the merge patch is passing close to 100% of
> > unit tests on Linux. Soon I will call for a vote to merge this branch
> into
> > trunk.
> >
> > Next steps:
> > 1. Call for vote to merge branch-trunk-win to trunk, when the work
> > completes and precommit build is clean.
> > 2. Start a discussion on adding Jenkins precommit builds on windows and
> > how to integrate that with the existing commit process.
> >
> > Let me know if you have any questions.
> >
> > Regards,
> > Suresh
> >
> >
>
>
> --
> http://hortonworks.com/download/
>

Re: [Vote] Merge branch-trunk-win to trunk

[Steve Loughran <st...@hortonworks.com>]

On 2 March 2013 03:33, Konstantin Boudnik <co...@apache.org> wrote:

>
> Windows is so different from _any_ Unix or pseudo-Unix flavors, including
> Windows with Cygwin - that even multi-platform Java has hard hard time
> dealing with it. This is enough, IMO, to warrant a separate checkpoint.
>
>
Cygwin is the worst of both worlds: not Unix, not windows. Dropping it for
proper windows is much better. Even dropping it altogether would be better.
We hate cygwin problems in Ant as users have unrealistic expectations about
the filesystem and how programs run.

CI-wise, it'd be good to have nightly builds and a preflight check per
JIRA. It sounds like the consensus that is evolving is (in RFC 2119 terms)

   1. CI test runs on Windows SHALL be provided (Matt has promised this)
   2. A patch with Pass(Linux) && Fail(windows) tests MAY be committed
   3. A patch with Pass(Linux) && Fail(windows) tests SHALL be fixed -but
   not necessarily by the author of the original patch
   4. A patch with Pass(windows) && Fail(linux) tests MUST NOT be committed
   5. * It is assumed that if it works on Linux, it SHOULD work on other
   Unix
   6. A patch with Pass(other-unix) && Fail(linux) tests MUST NOT be
   committed (this has never arisen that I know of). This could be merged with
   (3) to state that: all patches must Pass(Linux).
   7. The unix-side operation MAY BE optimised for Linux at the expense of
   other Unices (I remember that for exec()/fork() a way, way back).
   8. The unix-side operation MAY contain features that only work on Linux
   (YARN-3 cgroups are an example of this)
   9. A patch that is optimised for Linux SHOULD have a Windows alternative
   (c.f. local sockets). The alternative MAY be java code that substitutes for
   native C/C++/assembler

That mention of native code raises another question: CPU support.

Now that there's some ARM boxes running Hadoop on Jenkins, perhaps as AMD
ramp up their ARM story and IBM continue with Power, and some new x86 ASM
code is coming from Intel, we ought to have a policy there, something like

   1. Hadoop MAY contain code that works best on x86 systems.
   2. Hadoop MAY contain code that works best on ARM systems.
   3. Hadoop MUST NOT contain code that only works on on a single CPU family
   4. Hadoop SHOULD NOT contain code that  works best on on a single CPU
   family AND makes performance worse on other CPU families. E.G we shouldn't
   mandate CRC, compression and encryption algorithms that speed up on one CPU
   family yet are significantly worse on other CPU families than a
   platform-neutral algorithm.
   5. Hadoop MUST NOT contain code that has assumptions about the CPU
   memory model other than the java 5+ memory model [
   http://www.cs.umd.edu/~pugh/java/memoryModel/] . This is automatic for
   Java code, but needs to be included in native code, as volatile is not a
   barrier operation in C/C++, some CPUs implement different barrier op
   6. Hadoop MUST NOT contain code that has hard-code assumptions about
   cache other than CPUs implement coherency across cores. No hard coded
   assumptions about cache line sizes, write-through vs. write back. RAM could
   be NUMA, but MUST be consistent w.r.t. the causal model of the
   happens-before semantics of the java 5+ memory model. (if you didn't
   understand that, read up on memory models)

That could be teased out for a separate discussion and vote, along with
-maybe- a policy w.r.t non-HFDS filesystems (which could be SHOULD not
break other FSs, MAY reduce performance, patches MAY break (without tests,
who knows?). People implementing alternative filesystems and asserting
compatibility with Hadoop SHOULD run their own CI tests.

Re: [Vote] Merge branch-trunk-win to trunk

[Steve Loughran <st...@hortonworks.com>]

On 2 March 2013 03:33, Konstantin Boudnik <co...@apache.org> wrote:

>
> Windows is so different from _any_ Unix or pseudo-Unix flavors, including
> Windows with Cygwin - that even multi-platform Java has hard hard time
> dealing with it. This is enough, IMO, to warrant a separate checkpoint.
>
>
Cygwin is the worst of both worlds: not Unix, not windows. Dropping it for
proper windows is much better. Even dropping it altogether would be better.
We hate cygwin problems in Ant as users have unrealistic expectations about
the filesystem and how programs run.

CI-wise, it'd be good to have nightly builds and a preflight check per
JIRA. It sounds like the consensus that is evolving is (in RFC 2119 terms)

   1. CI test runs on Windows SHALL be provided (Matt has promised this)
   2. A patch with Pass(Linux) && Fail(windows) tests MAY be committed
   3. A patch with Pass(Linux) && Fail(windows) tests SHALL be fixed -but
   not necessarily by the author of the original patch
   4. A patch with Pass(windows) && Fail(linux) tests MUST NOT be committed
   5. * It is assumed that if it works on Linux, it SHOULD work on other
   Unix
   6. A patch with Pass(other-unix) && Fail(linux) tests MUST NOT be
   committed (this has never arisen that I know of). This could be merged with
   (3) to state that: all patches must Pass(Linux).
   7. The unix-side operation MAY BE optimised for Linux at the expense of
   other Unices (I remember that for exec()/fork() a way, way back).
   8. The unix-side operation MAY contain features that only work on Linux
   (YARN-3 cgroups are an example of this)
   9. A patch that is optimised for Linux SHOULD have a Windows alternative
   (c.f. local sockets). The alternative MAY be java code that substitutes for
   native C/C++/assembler

That mention of native code raises another question: CPU support.

Now that there's some ARM boxes running Hadoop on Jenkins, perhaps as AMD
ramp up their ARM story and IBM continue with Power, and some new x86 ASM
code is coming from Intel, we ought to have a policy there, something like

   1. Hadoop MAY contain code that works best on x86 systems.
   2. Hadoop MAY contain code that works best on ARM systems.
   3. Hadoop MUST NOT contain code that only works on on a single CPU family
   4. Hadoop SHOULD NOT contain code that  works best on on a single CPU
   family AND makes performance worse on other CPU families. E.G we shouldn't
   mandate CRC, compression and encryption algorithms that speed up on one CPU
   family yet are significantly worse on other CPU families than a
   platform-neutral algorithm.
   5. Hadoop MUST NOT contain code that has assumptions about the CPU
   memory model other than the java 5+ memory model [
   http://www.cs.umd.edu/~pugh/java/memoryModel/] . This is automatic for
   Java code, but needs to be included in native code, as volatile is not a
   barrier operation in C/C++, some CPUs implement different barrier op
   6. Hadoop MUST NOT contain code that has hard-code assumptions about
   cache other than CPUs implement coherency across cores. No hard coded
   assumptions about cache line sizes, write-through vs. write back. RAM could
   be NUMA, but MUST be consistent w.r.t. the causal model of the
   happens-before semantics of the java 5+ memory model. (if you didn't
   understand that, read up on memory models)

That could be teased out for a separate discussion and vote, along with
-maybe- a policy w.r.t non-HFDS filesystems (which could be SHOULD not
break other FSs, MAY reduce performance, patches MAY break (without tests,
who knows?). People implementing alternative filesystems and asserting
compatibility with Hadoop SHOULD run their own CI tests.

Re: [Vote] Merge branch-trunk-win to trunk

[Steve Loughran <st...@hortonworks.com>]

On 2 March 2013 03:33, Konstantin Boudnik <co...@apache.org> wrote:

>
> Windows is so different from _any_ Unix or pseudo-Unix flavors, including
> Windows with Cygwin - that even multi-platform Java has hard hard time
> dealing with it. This is enough, IMO, to warrant a separate checkpoint.
>
>
Cygwin is the worst of both worlds: not Unix, not windows. Dropping it for
proper windows is much better. Even dropping it altogether would be better.
We hate cygwin problems in Ant as users have unrealistic expectations about
the filesystem and how programs run.

CI-wise, it'd be good to have nightly builds and a preflight check per
JIRA. It sounds like the consensus that is evolving is (in RFC 2119 terms)

   1. CI test runs on Windows SHALL be provided (Matt has promised this)
   2. A patch with Pass(Linux) && Fail(windows) tests MAY be committed
   3. A patch with Pass(Linux) && Fail(windows) tests SHALL be fixed -but
   not necessarily by the author of the original patch
   4. A patch with Pass(windows) && Fail(linux) tests MUST NOT be committed
   5. * It is assumed that if it works on Linux, it SHOULD work on other
   Unix
   6. A patch with Pass(other-unix) && Fail(linux) tests MUST NOT be
   committed (this has never arisen that I know of). This could be merged with
   (3) to state that: all patches must Pass(Linux).
   7. The unix-side operation MAY BE optimised for Linux at the expense of
   other Unices (I remember that for exec()/fork() a way, way back).
   8. The unix-side operation MAY contain features that only work on Linux
   (YARN-3 cgroups are an example of this)
   9. A patch that is optimised for Linux SHOULD have a Windows alternative
   (c.f. local sockets). The alternative MAY be java code that substitutes for
   native C/C++/assembler

That mention of native code raises another question: CPU support.

Now that there's some ARM boxes running Hadoop on Jenkins, perhaps as AMD
ramp up their ARM story and IBM continue with Power, and some new x86 ASM
code is coming from Intel, we ought to have a policy there, something like

   1. Hadoop MAY contain code that works best on x86 systems.
   2. Hadoop MAY contain code that works best on ARM systems.
   3. Hadoop MUST NOT contain code that only works on on a single CPU family
   4. Hadoop SHOULD NOT contain code that  works best on on a single CPU
   family AND makes performance worse on other CPU families. E.G we shouldn't
   mandate CRC, compression and encryption algorithms that speed up on one CPU
   family yet are significantly worse on other CPU families than a
   platform-neutral algorithm.
   5. Hadoop MUST NOT contain code that has assumptions about the CPU
   memory model other than the java 5+ memory model [
   http://www.cs.umd.edu/~pugh/java/memoryModel/] . This is automatic for
   Java code, but needs to be included in native code, as volatile is not a
   barrier operation in C/C++, some CPUs implement different barrier op
   6. Hadoop MUST NOT contain code that has hard-code assumptions about
   cache other than CPUs implement coherency across cores. No hard coded
   assumptions about cache line sizes, write-through vs. write back. RAM could
   be NUMA, but MUST be consistent w.r.t. the causal model of the
   happens-before semantics of the java 5+ memory model. (if you didn't
   understand that, read up on memory models)

That could be teased out for a separate discussion and vote, along with
-maybe- a policy w.r.t non-HFDS filesystems (which could be SHOULD not
break other FSs, MAY reduce performance, patches MAY break (without tests,
who knows?). People implementing alternative filesystems and asserting
compatibility with Hadoop SHOULD run their own CI tests.

Re: [Vote] Merge branch-trunk-win to trunk

[Steve Loughran <st...@hortonworks.com>]

On 2 March 2013 03:33, Konstantin Boudnik <co...@apache.org> wrote:

>
> Windows is so different from _any_ Unix or pseudo-Unix flavors, including
> Windows with Cygwin - that even multi-platform Java has hard hard time
> dealing with it. This is enough, IMO, to warrant a separate checkpoint.
>
>
Cygwin is the worst of both worlds: not Unix, not windows. Dropping it for
proper windows is much better. Even dropping it altogether would be better.
We hate cygwin problems in Ant as users have unrealistic expectations about
the filesystem and how programs run.

CI-wise, it'd be good to have nightly builds and a preflight check per
JIRA. It sounds like the consensus that is evolving is (in RFC 2119 terms)

   1. CI test runs on Windows SHALL be provided (Matt has promised this)
   2. A patch with Pass(Linux) && Fail(windows) tests MAY be committed
   3. A patch with Pass(Linux) && Fail(windows) tests SHALL be fixed -but
   not necessarily by the author of the original patch
   4. A patch with Pass(windows) && Fail(linux) tests MUST NOT be committed
   5. * It is assumed that if it works on Linux, it SHOULD work on other
   Unix
   6. A patch with Pass(other-unix) && Fail(linux) tests MUST NOT be
   committed (this has never arisen that I know of). This could be merged with
   (3) to state that: all patches must Pass(Linux).
   7. The unix-side operation MAY BE optimised for Linux at the expense of
   other Unices (I remember that for exec()/fork() a way, way back).
   8. The unix-side operation MAY contain features that only work on Linux
   (YARN-3 cgroups are an example of this)
   9. A patch that is optimised for Linux SHOULD have a Windows alternative
   (c.f. local sockets). The alternative MAY be java code that substitutes for
   native C/C++/assembler

That mention of native code raises another question: CPU support.

Now that there's some ARM boxes running Hadoop on Jenkins, perhaps as AMD
ramp up their ARM story and IBM continue with Power, and some new x86 ASM
code is coming from Intel, we ought to have a policy there, something like

   1. Hadoop MAY contain code that works best on x86 systems.
   2. Hadoop MAY contain code that works best on ARM systems.
   3. Hadoop MUST NOT contain code that only works on on a single CPU family
   4. Hadoop SHOULD NOT contain code that  works best on on a single CPU
   family AND makes performance worse on other CPU families. E.G we shouldn't
   mandate CRC, compression and encryption algorithms that speed up on one CPU
   family yet are significantly worse on other CPU families than a
   platform-neutral algorithm.
   5. Hadoop MUST NOT contain code that has assumptions about the CPU
   memory model other than the java 5+ memory model [
   http://www.cs.umd.edu/~pugh/java/memoryModel/] . This is automatic for
   Java code, but needs to be included in native code, as volatile is not a
   barrier operation in C/C++, some CPUs implement different barrier op
   6. Hadoop MUST NOT contain code that has hard-code assumptions about
   cache other than CPUs implement coherency across cores. No hard coded
   assumptions about cache line sizes, write-through vs. write back. RAM could
   be NUMA, but MUST be consistent w.r.t. the causal model of the
   happens-before semantics of the java 5+ memory model. (if you didn't
   understand that, read up on memory models)

That could be teased out for a separate discussion and vote, along with
-maybe- a policy w.r.t non-HFDS filesystems (which could be SHOULD not
break other FSs, MAY reduce performance, patches MAY break (without tests,
who knows?). People implementing alternative filesystems and asserting
compatibility with Hadoop SHOULD run their own CI tests.

Re: [Vote] Merge branch-trunk-win to trunk

[Konstantin Boudnik <co...@apache.org>]

Suresh, I appreciate all the troubles you're going through wrt syncing up the
huge patch for a long time - I really do.

I am not asking to have full test-patch process in place. But I think it is a
real good idea to have a way to run the full test suite once in a while - or
as Konstantin proposing - for a given patch, to make sure that codebase
doesn't bitrot at the edges.

"Official" support has nothing to do with the issue - you're trying to build a
straw man argument around this. What is relevant, on the other hand, is that
Windows is so different from _any_ Unix or pseudo-Unix flavors, including
Windows with Cygwin - that even multi-platform Java has hard hard time
dealing with it. This is enough, IMO, to warrant a separate checkpoint.

I hope I have explained myself better.
  Cos

On Fri, Mar 01, 2013 at 05:55PM, Suresh Srinivas wrote:
> > It seems that with the HW in place, the matter of setting at least nightly
> > build is trivial for anyone with up to date Windows knowledge. I wish I
> > could
> > help. Going without a validation is a recipe for a disaster IMO.
> >
> > -1 until some reasonable solution is implemented.
> >   Cos
> 
> 
> Cos, I have hard time understanding your veto?
> 
> Here is my rationale for merge:
> Currently with all the cross platform support, the merge patch has +1 from
> Jenkins on Linux. So no regression has been introduced in Hadoop on Linux.
> 
> As regards to Windows support I want to make two points:
> 1. Until Jenkins is setup and we are passing all the tests, I am okay, as
> Konstntin proposed, if we do not officially declare Windows as supported. I
> do not want to tie the patch merge with setting up Windows Jenkins. I have
> been maintaining this branch for a long time and keeping it in sync with
> trunk is non-trivial.
> 2. After Jenkins is setup, there is a concern in the community about -1
> from Windows hindering patch commits. As others have already suggested in
> the thread, I am okay committing new patches even if -1 is posted by
> Jenkins on Windows. The team that worked on Windows will fix the issues. I
> do not see many such issues cropping up.

Re: [Vote] Merge branch-trunk-win to trunk

[Konstantin Boudnik <co...@apache.org>]

Suresh, I appreciate all the troubles you're going through wrt syncing up the
huge patch for a long time - I really do.

I am not asking to have full test-patch process in place. But I think it is a
real good idea to have a way to run the full test suite once in a while - or
as Konstantin proposing - for a given patch, to make sure that codebase
doesn't bitrot at the edges.

"Official" support has nothing to do with the issue - you're trying to build a
straw man argument around this. What is relevant, on the other hand, is that
Windows is so different from _any_ Unix or pseudo-Unix flavors, including
Windows with Cygwin - that even multi-platform Java has hard hard time
dealing with it. This is enough, IMO, to warrant a separate checkpoint.

I hope I have explained myself better.
  Cos

On Fri, Mar 01, 2013 at 05:55PM, Suresh Srinivas wrote:
> > It seems that with the HW in place, the matter of setting at least nightly
> > build is trivial for anyone with up to date Windows knowledge. I wish I
> > could
> > help. Going without a validation is a recipe for a disaster IMO.
> >
> > -1 until some reasonable solution is implemented.
> >   Cos
> 
> 
> Cos, I have hard time understanding your veto?
> 
> Here is my rationale for merge:
> Currently with all the cross platform support, the merge patch has +1 from
> Jenkins on Linux. So no regression has been introduced in Hadoop on Linux.
> 
> As regards to Windows support I want to make two points:
> 1. Until Jenkins is setup and we are passing all the tests, I am okay, as
> Konstntin proposed, if we do not officially declare Windows as supported. I
> do not want to tie the patch merge with setting up Windows Jenkins. I have
> been maintaining this branch for a long time and keeping it in sync with
> trunk is non-trivial.
> 2. After Jenkins is setup, there is a concern in the community about -1
> from Windows hindering patch commits. As others have already suggested in
> the thread, I am okay committing new patches even if -1 is posted by
> Jenkins on Windows. The team that worked on Windows will fix the issues. I
> do not see many such issues cropping up.

Re: [Vote] Merge branch-trunk-win to trunk

[Konstantin Boudnik <co...@apache.org>]

Suresh, I appreciate all the troubles you're going through wrt syncing up the
huge patch for a long time - I really do.

I am not asking to have full test-patch process in place. But I think it is a
real good idea to have a way to run the full test suite once in a while - or
as Konstantin proposing - for a given patch, to make sure that codebase
doesn't bitrot at the edges.

"Official" support has nothing to do with the issue - you're trying to build a
straw man argument around this. What is relevant, on the other hand, is that
Windows is so different from _any_ Unix or pseudo-Unix flavors, including
Windows with Cygwin - that even multi-platform Java has hard hard time
dealing with it. This is enough, IMO, to warrant a separate checkpoint.

I hope I have explained myself better.
  Cos

On Fri, Mar 01, 2013 at 05:55PM, Suresh Srinivas wrote:
> > It seems that with the HW in place, the matter of setting at least nightly
> > build is trivial for anyone with up to date Windows knowledge. I wish I
> > could
> > help. Going without a validation is a recipe for a disaster IMO.
> >
> > -1 until some reasonable solution is implemented.
> >   Cos
> 
> 
> Cos, I have hard time understanding your veto?
> 
> Here is my rationale for merge:
> Currently with all the cross platform support, the merge patch has +1 from
> Jenkins on Linux. So no regression has been introduced in Hadoop on Linux.
> 
> As regards to Windows support I want to make two points:
> 1. Until Jenkins is setup and we are passing all the tests, I am okay, as
> Konstntin proposed, if we do not officially declare Windows as supported. I
> do not want to tie the patch merge with setting up Windows Jenkins. I have
> been maintaining this branch for a long time and keeping it in sync with
> trunk is non-trivial.
> 2. After Jenkins is setup, there is a concern in the community about -1
> from Windows hindering patch commits. As others have already suggested in
> the thread, I am okay committing new patches even if -1 is posted by
> Jenkins on Windows. The team that worked on Windows will fix the issues. I
> do not see many such issues cropping up.

Re: [Vote] Merge branch-trunk-win to trunk

[Suresh Srinivas <su...@hortonworks.com>]

> It seems that with the HW in place, the matter of setting at least nightly
> build is trivial for anyone with up to date Windows knowledge. I wish I
> could
> help. Going without a validation is a recipe for a disaster IMO.
>
> -1 until some reasonable solution is implemented.
>   Cos


Cos, I have hard time understanding your veto?

Here is my rationale for merge:
Currently with all the cross platform support, the merge patch has +1 from
Jenkins on Linux. So no regression has been introduced in Hadoop on Linux.

As regards to Windows support I want to make two points:
1. Until Jenkins is setup and we are passing all the tests, I am okay, as
Konstntin proposed, if we do not officially declare Windows as supported. I
do not want to tie the patch merge with setting up Windows Jenkins. I have
been maintaining this branch for a long time and keeping it in sync with
trunk is non-trivial.
2. After Jenkins is setup, there is a concern in the community about -1
from Windows hindering patch commits. As others have already suggested in
the thread, I am okay committing new patches even if -1 is posted by
Jenkins on Windows. The team that worked on Windows will fix the issues. I
do not see many such issues cropping up.

Re: [Vote] Merge branch-trunk-win to trunk

[Konstantin Boudnik <co...@apache.org>]

It seems that with the HW in place, the matter of setting at least nightly
build is trivial for anyone with up to date Windows knowledge. I wish I could
help. Going without a validation is a recipe for a disaster IMO.

-1 until some reasonable solution is implemented.
  Cos

On Fri, Mar 01, 2013 at 01:57PM, Konstantin Shvachko wrote:
> Commitment is a good thing.
> I think the two builds that I proposed are a prerequisite for Win support.
> If we commit windows patch people will start breaking it the next day.
> Which we wont know without the nightly build and wont be able to fix
> without the on-demand one.
> 
> Making two builds is less than 2 days work, imho, given that there is
> a Windows node available and that mvn targets are in place. Correct me
> if I missed any complications in the process.
> 
> Thanks,
> --Konst
> 
> On Fri, Mar 1, 2013 at 1:28 PM, Chris Douglas <cd...@apache.org> wrote:
> > Konstantin-
> >
> > There's no debate on the necessity of CI and related infrastructure to
> > support the platform well. Suresh outlined the support to effect this
> > here: http://s.apache.org/s1
> >
> > Is the commitment to establish this infrastructure after the merge
> > sufficient? -C
> >
> > On Fri, Mar 1, 2013 at 12:18 PM, Konstantin Shvachko
> > <sh...@gmail.com> wrote:
> >> -1
> >> We should have a CI infrastructure in place before we can commit to
> >> supporting Windows platform.
> >>
> >> Eric is right Win/Cygwin was supported since day one.
> >> I had a Windows box under my desk running nightly builds back in 2006-07.
> >> People were irritated but I was filing windows bugs until 0.22 release.
> >> Times changing and I am glad to see wider support for Win platform.
> >>
> >> But in order to make it work you guys need to put the CI process in place
> >>
> >> 1. windows jenkins build: could be nightly or PreCommit.
> >> - Nightly would mean that changes can be committed to trunk based on
> >> linux PreCommit build. And people will file bugs if the change broke
> >> Windows nightly build.
> >> - PreCommit-win build will mean automatic reporting failed tests to
> >> respective jira blocking commits the same way as it is now with linux
> >> PreCommit builds.
> >> We should discuss which way is more efficient for developers.
> >>
> >> 2. On-demand-windows Jenkins build.
> >> I see it as a build to which I can attach my patch and the build will
> >> run my changes on a dedicated windows box.
> >> That way people can test their changes without having personal windows nodes.
> >>
> >> I think this is the minimal set of requirement for us to be able to
> >> commit to the new platform.
> >> Right now I see only one windows related build
> >> https://builds.apache.org/view/Hadoop/job/Hadoop-1-win/
> >> Which was failing since Sept 8, 2012 and did not run in the last month.
> >>
> >> Thanks,
> >> --Konst
> >>
> >> On Thu, Feb 28, 2013 at 8:47 PM, Eric Baldeschwieler
> >> <er...@hortonworks.com> wrote:
> >>> +1 (non-binding)
> >>>
> >>> A few of observations:
> >>>
> >>> - Windows has actually been a supported platform for Hadoop since 0.1 .  Doug championed supporting windows then and we've continued to do it with varying vigor over time.  To my knowledge we've never made a decision to drop windows support.  The change here is improving our support and dropping the requirement of cigwin.  We had Nutch windows users on the list in 2006 and we've been supporting windows FS requirements since inception.
> >>>
> >>> - A little pragmatism will go a long way.  As a community we've got to stay committed to keeping hadoop simple (so it does work on many platforms) and extending it to take advantage of key emerging OS/hardware features, such as containers, new FSs, virtualization, flash ...  We should all plan to let new features & optimizations emerge that don't work everywhere, if they are compelling and central to hadoop's mission of being THE best fabric for storing and processing big data.
> >>>
> >>> - A UI project like KDE has to deal with the MANY differences between windows and linux UI APIs.  Hadoop faces no such complex challenge and hence can be maintained from a single codeline IMO.  It is mostly abstracted from the OS APIs via Java and our design choices.  Where it is not we can continue to add plugable abstractions.
> >>>

Re: [Vote] Merge branch-trunk-win to trunk

[Konstantin Boudnik <co...@apache.org>]

It seems that with the HW in place, the matter of setting at least nightly
build is trivial for anyone with up to date Windows knowledge. I wish I could
help. Going without a validation is a recipe for a disaster IMO.

-1 until some reasonable solution is implemented.
  Cos

On Fri, Mar 01, 2013 at 01:57PM, Konstantin Shvachko wrote:
> Commitment is a good thing.
> I think the two builds that I proposed are a prerequisite for Win support.
> If we commit windows patch people will start breaking it the next day.
> Which we wont know without the nightly build and wont be able to fix
> without the on-demand one.
> 
> Making two builds is less than 2 days work, imho, given that there is
> a Windows node available and that mvn targets are in place. Correct me
> if I missed any complications in the process.
> 
> Thanks,
> --Konst
> 
> On Fri, Mar 1, 2013 at 1:28 PM, Chris Douglas <cd...@apache.org> wrote:
> > Konstantin-
> >
> > There's no debate on the necessity of CI and related infrastructure to
> > support the platform well. Suresh outlined the support to effect this
> > here: http://s.apache.org/s1
> >
> > Is the commitment to establish this infrastructure after the merge
> > sufficient? -C
> >
> > On Fri, Mar 1, 2013 at 12:18 PM, Konstantin Shvachko
> > <sh...@gmail.com> wrote:
> >> -1
> >> We should have a CI infrastructure in place before we can commit to
> >> supporting Windows platform.
> >>
> >> Eric is right Win/Cygwin was supported since day one.
> >> I had a Windows box under my desk running nightly builds back in 2006-07.
> >> People were irritated but I was filing windows bugs until 0.22 release.
> >> Times changing and I am glad to see wider support for Win platform.
> >>
> >> But in order to make it work you guys need to put the CI process in place
> >>
> >> 1. windows jenkins build: could be nightly or PreCommit.
> >> - Nightly would mean that changes can be committed to trunk based on
> >> linux PreCommit build. And people will file bugs if the change broke
> >> Windows nightly build.
> >> - PreCommit-win build will mean automatic reporting failed tests to
> >> respective jira blocking commits the same way as it is now with linux
> >> PreCommit builds.
> >> We should discuss which way is more efficient for developers.
> >>
> >> 2. On-demand-windows Jenkins build.
> >> I see it as a build to which I can attach my patch and the build will
> >> run my changes on a dedicated windows box.
> >> That way people can test their changes without having personal windows nodes.
> >>
> >> I think this is the minimal set of requirement for us to be able to
> >> commit to the new platform.
> >> Right now I see only one windows related build
> >> https://builds.apache.org/view/Hadoop/job/Hadoop-1-win/
> >> Which was failing since Sept 8, 2012 and did not run in the last month.
> >>
> >> Thanks,
> >> --Konst
> >>
> >> On Thu, Feb 28, 2013 at 8:47 PM, Eric Baldeschwieler
> >> <er...@hortonworks.com> wrote:
> >>> +1 (non-binding)
> >>>
> >>> A few of observations:
> >>>
> >>> - Windows has actually been a supported platform for Hadoop since 0.1 .  Doug championed supporting windows then and we've continued to do it with varying vigor over time.  To my knowledge we've never made a decision to drop windows support.  The change here is improving our support and dropping the requirement of cigwin.  We had Nutch windows users on the list in 2006 and we've been supporting windows FS requirements since inception.
> >>>
> >>> - A little pragmatism will go a long way.  As a community we've got to stay committed to keeping hadoop simple (so it does work on many platforms) and extending it to take advantage of key emerging OS/hardware features, such as containers, new FSs, virtualization, flash ...  We should all plan to let new features & optimizations emerge that don't work everywhere, if they are compelling and central to hadoop's mission of being THE best fabric for storing and processing big data.
> >>>
> >>> - A UI project like KDE has to deal with the MANY differences between windows and linux UI APIs.  Hadoop faces no such complex challenge and hence can be maintained from a single codeline IMO.  It is mostly abstracted from the OS APIs via Java and our design choices.  Where it is not we can continue to add plugable abstractions.
> >>>

Re: [Vote] Merge branch-trunk-win to trunk

[Konstantin Boudnik <co...@apache.org>]

Ok, looks like we are converging on this across a few hundred emails ;)

So, as has been stated elsewhere: test-patch will be improved to fully support
Windows; furthermore -1 from Windows' test-patch won't block Linux commits.
This is ok with me.

Can we have a JIRA ticket for that test-patch work assigned to the real owner,
so it can be tracked? I am +1 in this case.

Cos

On Fri, Mar 01, 2013 at 04:57PM, Chris Douglas wrote:
> On Fri, Mar 1, 2013 at 1:57 PM, Konstantin Shvachko
> <sh...@gmail.com> wrote:
> > Commitment is a good thing.
> > I think the two builds that I proposed are a prerequisite for Win support.
> > If we commit windows patch people will start breaking it the next day.
> > Which we wont know without the nightly build and wont be able to fix
> > without the on-demand one.
> 
> As several people have pointed out already, the surface of possible
> conflicts is relatively limited, and- as you did in 2007- the devs on
> Windows will report and fix bugs in that platform as they find them.
> CI is important for detecting and preventing bugs, but this isn't
> software we're launching into orbit.
> 
> > Making two builds is less than 2 days work, imho, given that there is
> > a Windows node available and that mvn targets are in place. Correct me
> > if I missed any complications in the process.
> 
> On Fri, Mar 1, 2013 at 3:47 PM, Konstantin Boudnik <co...@apache.org> wrote:
> > It seems that with the HW in place, the matter of setting at least nightly
> > build is trivial for anyone with up to date Windows knowledge. I wish I could
> > help. Going without a validation is a recipe for a disaster IMO.
> 
> Fair enough, though that also implies that the window for regressions
> is small, and it leaves little room to doubt that this will receive
> priority. Until it's merged, spurious notifications that the current
> trunk breaks Windows are an awkward introduction to devs' workflow.
> The order of merge/CI is a choice between mild annoyances, really.
> 
> But it might be moot. Giri: you're doing the work on this. When do you
> think it can be complete? -C

Re: [Vote] Merge branch-trunk-win to trunk

[Konstantin Boudnik <co...@apache.org>]

Ok, looks like we are converging on this across a few hundred emails ;)

So, as has been stated elsewhere: test-patch will be improved to fully support
Windows; furthermore -1 from Windows' test-patch won't block Linux commits.
This is ok with me.

Can we have a JIRA ticket for that test-patch work assigned to the real owner,
so it can be tracked? I am +1 in this case.

Cos

On Fri, Mar 01, 2013 at 04:57PM, Chris Douglas wrote:
> On Fri, Mar 1, 2013 at 1:57 PM, Konstantin Shvachko
> <sh...@gmail.com> wrote:
> > Commitment is a good thing.
> > I think the two builds that I proposed are a prerequisite for Win support.
> > If we commit windows patch people will start breaking it the next day.
> > Which we wont know without the nightly build and wont be able to fix
> > without the on-demand one.
> 
> As several people have pointed out already, the surface of possible
> conflicts is relatively limited, and- as you did in 2007- the devs on
> Windows will report and fix bugs in that platform as they find them.
> CI is important for detecting and preventing bugs, but this isn't
> software we're launching into orbit.
> 
> > Making two builds is less than 2 days work, imho, given that there is
> > a Windows node available and that mvn targets are in place. Correct me
> > if I missed any complications in the process.
> 
> On Fri, Mar 1, 2013 at 3:47 PM, Konstantin Boudnik <co...@apache.org> wrote:
> > It seems that with the HW in place, the matter of setting at least nightly
> > build is trivial for anyone with up to date Windows knowledge. I wish I could
> > help. Going without a validation is a recipe for a disaster IMO.
> 
> Fair enough, though that also implies that the window for regressions
> is small, and it leaves little room to doubt that this will receive
> priority. Until it's merged, spurious notifications that the current
> trunk breaks Windows are an awkward introduction to devs' workflow.
> The order of merge/CI is a choice between mild annoyances, really.
> 
> But it might be moot. Giri: you're doing the work on this. When do you
> think it can be complete? -C

Re: [Vote] Merge branch-trunk-win to trunk

[Konstantin Boudnik <co...@apache.org>]

Ok, looks like we are converging on this across a few hundred emails ;)

So, as has been stated elsewhere: test-patch will be improved to fully support
Windows; furthermore -1 from Windows' test-patch won't block Linux commits.
This is ok with me.

Can we have a JIRA ticket for that test-patch work assigned to the real owner,
so it can be tracked? I am +1 in this case.

Cos

On Fri, Mar 01, 2013 at 04:57PM, Chris Douglas wrote:
> On Fri, Mar 1, 2013 at 1:57 PM, Konstantin Shvachko
> <sh...@gmail.com> wrote:
> > Commitment is a good thing.
> > I think the two builds that I proposed are a prerequisite for Win support.
> > If we commit windows patch people will start breaking it the next day.
> > Which we wont know without the nightly build and wont be able to fix
> > without the on-demand one.
> 
> As several people have pointed out already, the surface of possible
> conflicts is relatively limited, and- as you did in 2007- the devs on
> Windows will report and fix bugs in that platform as they find them.
> CI is important for detecting and preventing bugs, but this isn't
> software we're launching into orbit.
> 
> > Making two builds is less than 2 days work, imho, given that there is
> > a Windows node available and that mvn targets are in place. Correct me
> > if I missed any complications in the process.
> 
> On Fri, Mar 1, 2013 at 3:47 PM, Konstantin Boudnik <co...@apache.org> wrote:
> > It seems that with the HW in place, the matter of setting at least nightly
> > build is trivial for anyone with up to date Windows knowledge. I wish I could
> > help. Going without a validation is a recipe for a disaster IMO.
> 
> Fair enough, though that also implies that the window for regressions
> is small, and it leaves little room to doubt that this will receive
> priority. Until it's merged, spurious notifications that the current
> trunk breaks Windows are an awkward introduction to devs' workflow.
> The order of merge/CI is a choice between mild annoyances, really.
> 
> But it might be moot. Giri: you're doing the work on this. When do you
> think it can be complete? -C

Re: [Vote] Merge branch-trunk-win to trunk

[Chris Douglas <cd...@apache.org>]

On Fri, Mar 1, 2013 at 1:57 PM, Konstantin Shvachko
<sh...@gmail.com> wrote:
> Commitment is a good thing.
> I think the two builds that I proposed are a prerequisite for Win support.
> If we commit windows patch people will start breaking it the next day.
> Which we wont know without the nightly build and wont be able to fix
> without the on-demand one.

As several people have pointed out already, the surface of possible
conflicts is relatively limited, and- as you did in 2007- the devs on
Windows will report and fix bugs in that platform as they find them.
CI is important for detecting and preventing bugs, but this isn't
software we're launching into orbit.

> Making two builds is less than 2 days work, imho, given that there is
> a Windows node available and that mvn targets are in place. Correct me
> if I missed any complications in the process.

On Fri, Mar 1, 2013 at 3:47 PM, Konstantin Boudnik <co...@apache.org> wrote:
> It seems that with the HW in place, the matter of setting at least nightly
> build is trivial for anyone with up to date Windows knowledge. I wish I could
> help. Going without a validation is a recipe for a disaster IMO.

Fair enough, though that also implies that the window for regressions
is small, and it leaves little room to doubt that this will receive
priority. Until it's merged, spurious notifications that the current
trunk breaks Windows are an awkward introduction to devs' workflow.
The order of merge/CI is a choice between mild annoyances, really.

But it might be moot. Giri: you're doing the work on this. When do you
think it can be complete? -C

Re: [Vote] Merge branch-trunk-win to trunk

[Chris Douglas <cd...@apache.org>]

On Fri, Mar 1, 2013 at 1:57 PM, Konstantin Shvachko
<sh...@gmail.com> wrote:
> Commitment is a good thing.
> I think the two builds that I proposed are a prerequisite for Win support.
> If we commit windows patch people will start breaking it the next day.
> Which we wont know without the nightly build and wont be able to fix
> without the on-demand one.

As several people have pointed out already, the surface of possible
conflicts is relatively limited, and- as you did in 2007- the devs on
Windows will report and fix bugs in that platform as they find them.
CI is important for detecting and preventing bugs, but this isn't
software we're launching into orbit.

> Making two builds is less than 2 days work, imho, given that there is
> a Windows node available and that mvn targets are in place. Correct me
> if I missed any complications in the process.

On Fri, Mar 1, 2013 at 3:47 PM, Konstantin Boudnik <co...@apache.org> wrote:
> It seems that with the HW in place, the matter of setting at least nightly
> build is trivial for anyone with up to date Windows knowledge. I wish I could
> help. Going without a validation is a recipe for a disaster IMO.

Fair enough, though that also implies that the window for regressions
is small, and it leaves little room to doubt that this will receive
priority. Until it's merged, spurious notifications that the current
trunk breaks Windows are an awkward introduction to devs' workflow.
The order of merge/CI is a choice between mild annoyances, really.

But it might be moot. Giri: you're doing the work on this. When do you
think it can be complete? -C

Re: [Vote] Merge branch-trunk-win to trunk

[Chris Douglas <cd...@apache.org>]

On Fri, Mar 1, 2013 at 1:57 PM, Konstantin Shvachko
<sh...@gmail.com> wrote:
> Commitment is a good thing.
> I think the two builds that I proposed are a prerequisite for Win support.
> If we commit windows patch people will start breaking it the next day.
> Which we wont know without the nightly build and wont be able to fix
> without the on-demand one.

As several people have pointed out already, the surface of possible
conflicts is relatively limited, and- as you did in 2007- the devs on
Windows will report and fix bugs in that platform as they find them.
CI is important for detecting and preventing bugs, but this isn't
software we're launching into orbit.

> Making two builds is less than 2 days work, imho, given that there is
> a Windows node available and that mvn targets are in place. Correct me
> if I missed any complications in the process.

On Fri, Mar 1, 2013 at 3:47 PM, Konstantin Boudnik <co...@apache.org> wrote:
> It seems that with the HW in place, the matter of setting at least nightly
> build is trivial for anyone with up to date Windows knowledge. I wish I could
> help. Going without a validation is a recipe for a disaster IMO.

Fair enough, though that also implies that the window for regressions
is small, and it leaves little room to doubt that this will receive
priority. Until it's merged, spurious notifications that the current
trunk breaks Windows are an awkward introduction to devs' workflow.
The order of merge/CI is a choice between mild annoyances, really.

But it might be moot. Giri: you're doing the work on this. When do you
think it can be complete? -C

Re: [Vote] Merge branch-trunk-win to trunk

[Konstantin Shvachko <sh...@gmail.com>]

Sanjay,

This is really confusing now.
Does Hadoop intend to support Windows by committing this patch?
If not, when the declaration of the "official" support comes in and
what does it mean?
Committing a 500K patch just to make things "not worth" doesn't make
sense to me.
If the support for this is planned to be the same as today's for
cygwin then what's the point?

I thought it was a simple thing to ask: create a nightly build on
Jenkins and then
duplicate it with additional parameter a patch file.
If you have an internal build system how hard is it to push it to
Apache Jenkins.
Who is the volunteer for this work, please speak up when it can be done.

Thanks,
--Konstantin


On Fri, Mar 1, 2013 at 6:03 PM, sanjay Radia <sa...@hortonworks.com> wrote:
>
> On Mar 1, 2013, at 1:57 PM, Konstantin Shvachko wrote:
>
>> Commitment is a good thing.
>> I think the two builds that I proposed are a prerequisite for Win support.
>> If we commit windows patch people will start breaking it the next day.
>> Which we wont know without the nightly build and wont be able to fix
>> without the on-demand one.
>
> They clearly are a prerequisite for declaring "official" support for windows.
> But they should not be a prerequisite for the merge,.
> Currently we enable windows through cygwin. There is no jenkins. Folks have been  fixing  windows issues as they are discovered.
> Merging the branch makes the situation no worse than it is today - all tests pass on Linux, there is no regression.
> Merging now removes  the cygwin dependency.
>
> Jenkins is critical to make windows officially supported platform without cygwin.
> When Jenkins is enabled, the team that has worked on this branch will have to fix any bugs that have arisen in the mean time.
>
> sanjay
>
>
>
>

Re: [Vote] Merge branch-trunk-win to trunk

[Konstantin Shvachko <sh...@gmail.com>]

Sanjay,

This is really confusing now.
Does Hadoop intend to support Windows by committing this patch?
If not, when the declaration of the "official" support comes in and
what does it mean?
Committing a 500K patch just to make things "not worth" doesn't make
sense to me.
If the support for this is planned to be the same as today's for
cygwin then what's the point?

I thought it was a simple thing to ask: create a nightly build on
Jenkins and then
duplicate it with additional parameter a patch file.
If you have an internal build system how hard is it to push it to
Apache Jenkins.
Who is the volunteer for this work, please speak up when it can be done.

Thanks,
--Konstantin


On Fri, Mar 1, 2013 at 6:03 PM, sanjay Radia <sa...@hortonworks.com> wrote:
>
> On Mar 1, 2013, at 1:57 PM, Konstantin Shvachko wrote:
>
>> Commitment is a good thing.
>> I think the two builds that I proposed are a prerequisite for Win support.
>> If we commit windows patch people will start breaking it the next day.
>> Which we wont know without the nightly build and wont be able to fix
>> without the on-demand one.
>
> They clearly are a prerequisite for declaring "official" support for windows.
> But they should not be a prerequisite for the merge,.
> Currently we enable windows through cygwin. There is no jenkins. Folks have been  fixing  windows issues as they are discovered.
> Merging the branch makes the situation no worse than it is today - all tests pass on Linux, there is no regression.
> Merging now removes  the cygwin dependency.
>
> Jenkins is critical to make windows officially supported platform without cygwin.
> When Jenkins is enabled, the team that has worked on this branch will have to fix any bugs that have arisen in the mean time.
>
> sanjay
>
>
>
>

Re: [Vote] Merge branch-trunk-win to trunk

[Konstantin Shvachko <sh...@gmail.com>]

Sanjay,

This is really confusing now.
Does Hadoop intend to support Windows by committing this patch?
If not, when the declaration of the "official" support comes in and
what does it mean?
Committing a 500K patch just to make things "not worth" doesn't make
sense to me.
If the support for this is planned to be the same as today's for
cygwin then what's the point?

I thought it was a simple thing to ask: create a nightly build on
Jenkins and then
duplicate it with additional parameter a patch file.
If you have an internal build system how hard is it to push it to
Apache Jenkins.
Who is the volunteer for this work, please speak up when it can be done.

Thanks,
--Konstantin


On Fri, Mar 1, 2013 at 6:03 PM, sanjay Radia <sa...@hortonworks.com> wrote:
>
> On Mar 1, 2013, at 1:57 PM, Konstantin Shvachko wrote:
>
>> Commitment is a good thing.
>> I think the two builds that I proposed are a prerequisite for Win support.
>> If we commit windows patch people will start breaking it the next day.
>> Which we wont know without the nightly build and wont be able to fix
>> without the on-demand one.
>
> They clearly are a prerequisite for declaring "official" support for windows.
> But they should not be a prerequisite for the merge,.
> Currently we enable windows through cygwin. There is no jenkins. Folks have been  fixing  windows issues as they are discovered.
> Merging the branch makes the situation no worse than it is today - all tests pass on Linux, there is no regression.
> Merging now removes  the cygwin dependency.
>
> Jenkins is critical to make windows officially supported platform without cygwin.
> When Jenkins is enabled, the team that has worked on this branch will have to fix any bugs that have arisen in the mean time.
>
> sanjay
>
>
>
>

Re: [Vote] Merge branch-trunk-win to trunk

[sanjay Radia <sa...@hortonworks.com>]

On Mar 1, 2013, at 1:57 PM, Konstantin Shvachko wrote:

> Commitment is a good thing.
> I think the two builds that I proposed are a prerequisite for Win support.
> If we commit windows patch people will start breaking it the next day.
> Which we wont know without the nightly build and wont be able to fix
> without the on-demand one.

They clearly are a prerequisite for declaring "official" support for windows. 
But they should not be a prerequisite for the merge,. 
Currently we enable windows through cygwin. There is no jenkins. Folks have been  fixing  windows issues as they are discovered.
Merging the branch makes the situation no worse than it is today - all tests pass on Linux, there is no regression.
Merging now removes  the cygwin dependency.

Jenkins is critical to make windows officially supported platform without cygwin.
When Jenkins is enabled, the team that has worked on this branch will have to fix any bugs that have arisen in the mean time.

sanjay

Re: [Vote] Merge branch-trunk-win to trunk

[Konstantin Boudnik <co...@apache.org>]

It seems that with the HW in place, the matter of setting at least nightly
build is trivial for anyone with up to date Windows knowledge. I wish I could
help. Going without a validation is a recipe for a disaster IMO.

-1 until some reasonable solution is implemented.
  Cos

On Fri, Mar 01, 2013 at 01:57PM, Konstantin Shvachko wrote:
> Commitment is a good thing.
> I think the two builds that I proposed are a prerequisite for Win support.
> If we commit windows patch people will start breaking it the next day.
> Which we wont know without the nightly build and wont be able to fix
> without the on-demand one.
> 
> Making two builds is less than 2 days work, imho, given that there is
> a Windows node available and that mvn targets are in place. Correct me
> if I missed any complications in the process.
> 
> Thanks,
> --Konst
> 
> On Fri, Mar 1, 2013 at 1:28 PM, Chris Douglas <cd...@apache.org> wrote:
> > Konstantin-
> >
> > There's no debate on the necessity of CI and related infrastructure to
> > support the platform well. Suresh outlined the support to effect this
> > here: http://s.apache.org/s1
> >
> > Is the commitment to establish this infrastructure after the merge
> > sufficient? -C
> >
> > On Fri, Mar 1, 2013 at 12:18 PM, Konstantin Shvachko
> > <sh...@gmail.com> wrote:
> >> -1
> >> We should have a CI infrastructure in place before we can commit to
> >> supporting Windows platform.
> >>
> >> Eric is right Win/Cygwin was supported since day one.
> >> I had a Windows box under my desk running nightly builds back in 2006-07.
> >> People were irritated but I was filing windows bugs until 0.22 release.
> >> Times changing and I am glad to see wider support for Win platform.
> >>
> >> But in order to make it work you guys need to put the CI process in place
> >>
> >> 1. windows jenkins build: could be nightly or PreCommit.
> >> - Nightly would mean that changes can be committed to trunk based on
> >> linux PreCommit build. And people will file bugs if the change broke
> >> Windows nightly build.
> >> - PreCommit-win build will mean automatic reporting failed tests to
> >> respective jira blocking commits the same way as it is now with linux
> >> PreCommit builds.
> >> We should discuss which way is more efficient for developers.
> >>
> >> 2. On-demand-windows Jenkins build.
> >> I see it as a build to which I can attach my patch and the build will
> >> run my changes on a dedicated windows box.
> >> That way people can test their changes without having personal windows nodes.
> >>
> >> I think this is the minimal set of requirement for us to be able to
> >> commit to the new platform.
> >> Right now I see only one windows related build
> >> https://builds.apache.org/view/Hadoop/job/Hadoop-1-win/
> >> Which was failing since Sept 8, 2012 and did not run in the last month.
> >>
> >> Thanks,
> >> --Konst
> >>
> >> On Thu, Feb 28, 2013 at 8:47 PM, Eric Baldeschwieler
> >> <er...@hortonworks.com> wrote:
> >>> +1 (non-binding)
> >>>
> >>> A few of observations:
> >>>
> >>> - Windows has actually been a supported platform for Hadoop since 0.1 .  Doug championed supporting windows then and we've continued to do it with varying vigor over time.  To my knowledge we've never made a decision to drop windows support.  The change here is improving our support and dropping the requirement of cigwin.  We had Nutch windows users on the list in 2006 and we've been supporting windows FS requirements since inception.
> >>>
> >>> - A little pragmatism will go a long way.  As a community we've got to stay committed to keeping hadoop simple (so it does work on many platforms) and extending it to take advantage of key emerging OS/hardware features, such as containers, new FSs, virtualization, flash ...  We should all plan to let new features & optimizations emerge that don't work everywhere, if they are compelling and central to hadoop's mission of being THE best fabric for storing and processing big data.
> >>>
> >>> - A UI project like KDE has to deal with the MANY differences between windows and linux UI APIs.  Hadoop faces no such complex challenge and hence can be maintained from a single codeline IMO.  It is mostly abstracted from the OS APIs via Java and our design choices.  Where it is not we can continue to add plugable abstractions.
> >>>

Re: [Vote] Merge branch-trunk-win to trunk

[sanjay Radia <sa...@hortonworks.com>]

On Mar 1, 2013, at 1:57 PM, Konstantin Shvachko wrote:

> Commitment is a good thing.
> I think the two builds that I proposed are a prerequisite for Win support.
> If we commit windows patch people will start breaking it the next day.
> Which we wont know without the nightly build and wont be able to fix
> without the on-demand one.

They clearly are a prerequisite for declaring "official" support for windows. 
But they should not be a prerequisite for the merge,. 
Currently we enable windows through cygwin. There is no jenkins. Folks have been  fixing  windows issues as they are discovered.
Merging the branch makes the situation no worse than it is today - all tests pass on Linux, there is no regression.
Merging now removes  the cygwin dependency.

Jenkins is critical to make windows officially supported platform without cygwin.
When Jenkins is enabled, the team that has worked on this branch will have to fix any bugs that have arisen in the mean time.

sanjay

Re: [Vote] Merge branch-trunk-win to trunk

[sanjay Radia <sa...@hortonworks.com>]

On Mar 1, 2013, at 1:57 PM, Konstantin Shvachko wrote:

> Commitment is a good thing.
> I think the two builds that I proposed are a prerequisite for Win support.
> If we commit windows patch people will start breaking it the next day.
> Which we wont know without the nightly build and wont be able to fix
> without the on-demand one.

They clearly are a prerequisite for declaring "official" support for windows. 
But they should not be a prerequisite for the merge,. 
Currently we enable windows through cygwin. There is no jenkins. Folks have been  fixing  windows issues as they are discovered.
Merging the branch makes the situation no worse than it is today - all tests pass on Linux, there is no regression.
Merging now removes  the cygwin dependency.

Jenkins is critical to make windows officially supported platform without cygwin.
When Jenkins is enabled, the team that has worked on this branch will have to fix any bugs that have arisen in the mean time.

sanjay

Re: [Vote] Merge branch-trunk-win to trunk

[Chris Douglas <cd...@apache.org>]

On Fri, Mar 1, 2013 at 1:57 PM, Konstantin Shvachko
<sh...@gmail.com> wrote:
> Commitment is a good thing.
> I think the two builds that I proposed are a prerequisite for Win support.
> If we commit windows patch people will start breaking it the next day.
> Which we wont know without the nightly build and wont be able to fix
> without the on-demand one.

As several people have pointed out already, the surface of possible
conflicts is relatively limited, and- as you did in 2007- the devs on
Windows will report and fix bugs in that platform as they find them.
CI is important for detecting and preventing bugs, but this isn't
software we're launching into orbit.

> Making two builds is less than 2 days work, imho, given that there is
> a Windows node available and that mvn targets are in place. Correct me
> if I missed any complications in the process.

On Fri, Mar 1, 2013 at 3:47 PM, Konstantin Boudnik <co...@apache.org> wrote:
> It seems that with the HW in place, the matter of setting at least nightly
> build is trivial for anyone with up to date Windows knowledge. I wish I could
> help. Going without a validation is a recipe for a disaster IMO.

Fair enough, though that also implies that the window for regressions
is small, and it leaves little room to doubt that this will receive
priority. Until it's merged, spurious notifications that the current
trunk breaks Windows are an awkward introduction to devs' workflow.
The order of merge/CI is a choice between mild annoyances, really.

But it might be moot. Giri: you're doing the work on this. When do you
think it can be complete? -C

Re: [Vote] Merge branch-trunk-win to trunk

[Konstantin Shvachko <sh...@gmail.com>]

Commitment is a good thing.
I think the two builds that I proposed are a prerequisite for Win support.
If we commit windows patch people will start breaking it the next day.
Which we wont know without the nightly build and wont be able to fix
without the on-demand one.

Making two builds is less than 2 days work, imho, given that there is
a Windows node available and that mvn targets are in place. Correct me
if I missed any complications in the process.

Thanks,
--Konst

On Fri, Mar 1, 2013 at 1:28 PM, Chris Douglas <cd...@apache.org> wrote:
> Konstantin-
>
> There's no debate on the necessity of CI and related infrastructure to
> support the platform well. Suresh outlined the support to effect this
> here: http://s.apache.org/s1
>
> Is the commitment to establish this infrastructure after the merge
> sufficient? -C
>
> On Fri, Mar 1, 2013 at 12:18 PM, Konstantin Shvachko
> <sh...@gmail.com> wrote:
>> -1
>> We should have a CI infrastructure in place before we can commit to
>> supporting Windows platform.
>>
>> Eric is right Win/Cygwin was supported since day one.
>> I had a Windows box under my desk running nightly builds back in 2006-07.
>> People were irritated but I was filing windows bugs until 0.22 release.
>> Times changing and I am glad to see wider support for Win platform.
>>
>> But in order to make it work you guys need to put the CI process in place
>>
>> 1. windows jenkins build: could be nightly or PreCommit.
>> - Nightly would mean that changes can be committed to trunk based on
>> linux PreCommit build. And people will file bugs if the change broke
>> Windows nightly build.
>> - PreCommit-win build will mean automatic reporting failed tests to
>> respective jira blocking commits the same way as it is now with linux
>> PreCommit builds.
>> We should discuss which way is more efficient for developers.
>>
>> 2. On-demand-windows Jenkins build.
>> I see it as a build to which I can attach my patch and the build will
>> run my changes on a dedicated windows box.
>> That way people can test their changes without having personal windows nodes.
>>
>> I think this is the minimal set of requirement for us to be able to
>> commit to the new platform.
>> Right now I see only one windows related build
>> https://builds.apache.org/view/Hadoop/job/Hadoop-1-win/
>> Which was failing since Sept 8, 2012 and did not run in the last month.
>>
>> Thanks,
>> --Konst
>>
>> On Thu, Feb 28, 2013 at 8:47 PM, Eric Baldeschwieler
>> <er...@hortonworks.com> wrote:
>>> +1 (non-binding)
>>>
>>> A few of observations:
>>>
>>> - Windows has actually been a supported platform for Hadoop since 0.1 .  Doug championed supporting windows then and we've continued to do it with varying vigor over time.  To my knowledge we've never made a decision to drop windows support.  The change here is improving our support and dropping the requirement of cigwin.  We had Nutch windows users on the list in 2006 and we've been supporting windows FS requirements since inception.
>>>
>>> - A little pragmatism will go a long way.  As a community we've got to stay committed to keeping hadoop simple (so it does work on many platforms) and extending it to take advantage of key emerging OS/hardware features, such as containers, new FSs, virtualization, flash ...  We should all plan to let new features & optimizations emerge that don't work everywhere, if they are compelling and central to hadoop's mission of being THE best fabric for storing and processing big data.
>>>
>>> - A UI project like KDE has to deal with the MANY differences between windows and linux UI APIs.  Hadoop faces no such complex challenge and hence can be maintained from a single codeline IMO.  It is mostly abstracted from the OS APIs via Java and our design choices.  Where it is not we can continue to add plugable abstractions.
>>>

Re: [Vote] Merge branch-trunk-win to trunk

[Konstantin Shvachko <sh...@gmail.com>]

Commitment is a good thing.
I think the two builds that I proposed are a prerequisite for Win support.
If we commit windows patch people will start breaking it the next day.
Which we wont know without the nightly build and wont be able to fix
without the on-demand one.

Making two builds is less than 2 days work, imho, given that there is
a Windows node available and that mvn targets are in place. Correct me
if I missed any complications in the process.

Thanks,
--Konst

On Fri, Mar 1, 2013 at 1:28 PM, Chris Douglas <cd...@apache.org> wrote:
> Konstantin-
>
> There's no debate on the necessity of CI and related infrastructure to
> support the platform well. Suresh outlined the support to effect this
> here: http://s.apache.org/s1
>
> Is the commitment to establish this infrastructure after the merge
> sufficient? -C
>
> On Fri, Mar 1, 2013 at 12:18 PM, Konstantin Shvachko
> <sh...@gmail.com> wrote:
>> -1
>> We should have a CI infrastructure in place before we can commit to
>> supporting Windows platform.
>>
>> Eric is right Win/Cygwin was supported since day one.
>> I had a Windows box under my desk running nightly builds back in 2006-07.
>> People were irritated but I was filing windows bugs until 0.22 release.
>> Times changing and I am glad to see wider support for Win platform.
>>
>> But in order to make it work you guys need to put the CI process in place
>>
>> 1. windows jenkins build: could be nightly or PreCommit.
>> - Nightly would mean that changes can be committed to trunk based on
>> linux PreCommit build. And people will file bugs if the change broke
>> Windows nightly build.
>> - PreCommit-win build will mean automatic reporting failed tests to
>> respective jira blocking commits the same way as it is now with linux
>> PreCommit builds.
>> We should discuss which way is more efficient for developers.
>>
>> 2. On-demand-windows Jenkins build.
>> I see it as a build to which I can attach my patch and the build will
>> run my changes on a dedicated windows box.
>> That way people can test their changes without having personal windows nodes.
>>
>> I think this is the minimal set of requirement for us to be able to
>> commit to the new platform.
>> Right now I see only one windows related build
>> https://builds.apache.org/view/Hadoop/job/Hadoop-1-win/
>> Which was failing since Sept 8, 2012 and did not run in the last month.
>>
>> Thanks,
>> --Konst
>>
>> On Thu, Feb 28, 2013 at 8:47 PM, Eric Baldeschwieler
>> <er...@hortonworks.com> wrote:
>>> +1 (non-binding)
>>>
>>> A few of observations:
>>>
>>> - Windows has actually been a supported platform for Hadoop since 0.1 .  Doug championed supporting windows then and we've continued to do it with varying vigor over time.  To my knowledge we've never made a decision to drop windows support.  The change here is improving our support and dropping the requirement of cigwin.  We had Nutch windows users on the list in 2006 and we've been supporting windows FS requirements since inception.
>>>
>>> - A little pragmatism will go a long way.  As a community we've got to stay committed to keeping hadoop simple (so it does work on many platforms) and extending it to take advantage of key emerging OS/hardware features, such as containers, new FSs, virtualization, flash ...  We should all plan to let new features & optimizations emerge that don't work everywhere, if they are compelling and central to hadoop's mission of being THE best fabric for storing and processing big data.
>>>
>>> - A UI project like KDE has to deal with the MANY differences between windows and linux UI APIs.  Hadoop faces no such complex challenge and hence can be maintained from a single codeline IMO.  It is mostly abstracted from the OS APIs via Java and our design choices.  Where it is not we can continue to add plugable abstractions.
>>>

Re: [Vote] Merge branch-trunk-win to trunk

[Konstantin Shvachko <sh...@gmail.com>]

Commitment is a good thing.
I think the two builds that I proposed are a prerequisite for Win support.
If we commit windows patch people will start breaking it the next day.
Which we wont know without the nightly build and wont be able to fix
without the on-demand one.

Making two builds is less than 2 days work, imho, given that there is
a Windows node available and that mvn targets are in place. Correct me
if I missed any complications in the process.

Thanks,
--Konst

On Fri, Mar 1, 2013 at 1:28 PM, Chris Douglas <cd...@apache.org> wrote:
> Konstantin-
>
> There's no debate on the necessity of CI and related infrastructure to
> support the platform well. Suresh outlined the support to effect this
> here: http://s.apache.org/s1
>
> Is the commitment to establish this infrastructure after the merge
> sufficient? -C
>
> On Fri, Mar 1, 2013 at 12:18 PM, Konstantin Shvachko
> <sh...@gmail.com> wrote:
>> -1
>> We should have a CI infrastructure in place before we can commit to
>> supporting Windows platform.
>>
>> Eric is right Win/Cygwin was supported since day one.
>> I had a Windows box under my desk running nightly builds back in 2006-07.
>> People were irritated but I was filing windows bugs until 0.22 release.
>> Times changing and I am glad to see wider support for Win platform.
>>
>> But in order to make it work you guys need to put the CI process in place
>>
>> 1. windows jenkins build: could be nightly or PreCommit.
>> - Nightly would mean that changes can be committed to trunk based on
>> linux PreCommit build. And people will file bugs if the change broke
>> Windows nightly build.
>> - PreCommit-win build will mean automatic reporting failed tests to
>> respective jira blocking commits the same way as it is now with linux
>> PreCommit builds.
>> We should discuss which way is more efficient for developers.
>>
>> 2. On-demand-windows Jenkins build.
>> I see it as a build to which I can attach my patch and the build will
>> run my changes on a dedicated windows box.
>> That way people can test their changes without having personal windows nodes.
>>
>> I think this is the minimal set of requirement for us to be able to
>> commit to the new platform.
>> Right now I see only one windows related build
>> https://builds.apache.org/view/Hadoop/job/Hadoop-1-win/
>> Which was failing since Sept 8, 2012 and did not run in the last month.
>>
>> Thanks,
>> --Konst
>>
>> On Thu, Feb 28, 2013 at 8:47 PM, Eric Baldeschwieler
>> <er...@hortonworks.com> wrote:
>>> +1 (non-binding)
>>>
>>> A few of observations:
>>>
>>> - Windows has actually been a supported platform for Hadoop since 0.1 .  Doug championed supporting windows then and we've continued to do it with varying vigor over time.  To my knowledge we've never made a decision to drop windows support.  The change here is improving our support and dropping the requirement of cigwin.  We had Nutch windows users on the list in 2006 and we've been supporting windows FS requirements since inception.
>>>
>>> - A little pragmatism will go a long way.  As a community we've got to stay committed to keeping hadoop simple (so it does work on many platforms) and extending it to take advantage of key emerging OS/hardware features, such as containers, new FSs, virtualization, flash ...  We should all plan to let new features & optimizations emerge that don't work everywhere, if they are compelling and central to hadoop's mission of being THE best fabric for storing and processing big data.
>>>
>>> - A UI project like KDE has to deal with the MANY differences between windows and linux UI APIs.  Hadoop faces no such complex challenge and hence can be maintained from a single codeline IMO.  It is mostly abstracted from the OS APIs via Java and our design choices.  Where it is not we can continue to add plugable abstractions.
>>>

Re: [Vote] Merge branch-trunk-win to trunk

[Konstantin Shvachko <sh...@gmail.com>]

Commitment is a good thing.
I think the two builds that I proposed are a prerequisite for Win support.
If we commit windows patch people will start breaking it the next day.
Which we wont know without the nightly build and wont be able to fix
without the on-demand one.

Making two builds is less than 2 days work, imho, given that there is
a Windows node available and that mvn targets are in place. Correct me
if I missed any complications in the process.

Thanks,
--Konst

On Fri, Mar 1, 2013 at 1:28 PM, Chris Douglas <cd...@apache.org> wrote:
> Konstantin-
>
> There's no debate on the necessity of CI and related infrastructure to
> support the platform well. Suresh outlined the support to effect this
> here: http://s.apache.org/s1
>
> Is the commitment to establish this infrastructure after the merge
> sufficient? -C
>
> On Fri, Mar 1, 2013 at 12:18 PM, Konstantin Shvachko
> <sh...@gmail.com> wrote:
>> -1
>> We should have a CI infrastructure in place before we can commit to
>> supporting Windows platform.
>>
>> Eric is right Win/Cygwin was supported since day one.
>> I had a Windows box under my desk running nightly builds back in 2006-07.
>> People were irritated but I was filing windows bugs until 0.22 release.
>> Times changing and I am glad to see wider support for Win platform.
>>
>> But in order to make it work you guys need to put the CI process in place
>>
>> 1. windows jenkins build: could be nightly or PreCommit.
>> - Nightly would mean that changes can be committed to trunk based on
>> linux PreCommit build. And people will file bugs if the change broke
>> Windows nightly build.
>> - PreCommit-win build will mean automatic reporting failed tests to
>> respective jira blocking commits the same way as it is now with linux
>> PreCommit builds.
>> We should discuss which way is more efficient for developers.
>>
>> 2. On-demand-windows Jenkins build.
>> I see it as a build to which I can attach my patch and the build will
>> run my changes on a dedicated windows box.
>> That way people can test their changes without having personal windows nodes.
>>
>> I think this is the minimal set of requirement for us to be able to
>> commit to the new platform.
>> Right now I see only one windows related build
>> https://builds.apache.org/view/Hadoop/job/Hadoop-1-win/
>> Which was failing since Sept 8, 2012 and did not run in the last month.
>>
>> Thanks,
>> --Konst
>>
>> On Thu, Feb 28, 2013 at 8:47 PM, Eric Baldeschwieler
>> <er...@hortonworks.com> wrote:
>>> +1 (non-binding)
>>>
>>> A few of observations:
>>>
>>> - Windows has actually been a supported platform for Hadoop since 0.1 .  Doug championed supporting windows then and we've continued to do it with varying vigor over time.  To my knowledge we've never made a decision to drop windows support.  The change here is improving our support and dropping the requirement of cigwin.  We had Nutch windows users on the list in 2006 and we've been supporting windows FS requirements since inception.
>>>
>>> - A little pragmatism will go a long way.  As a community we've got to stay committed to keeping hadoop simple (so it does work on many platforms) and extending it to take advantage of key emerging OS/hardware features, such as containers, new FSs, virtualization, flash ...  We should all plan to let new features & optimizations emerge that don't work everywhere, if they are compelling and central to hadoop's mission of being THE best fabric for storing and processing big data.
>>>
>>> - A UI project like KDE has to deal with the MANY differences between windows and linux UI APIs.  Hadoop faces no such complex challenge and hence can be maintained from a single codeline IMO.  It is mostly abstracted from the OS APIs via Java and our design choices.  Where it is not we can continue to add plugable abstractions.
>>>

Re: [Vote] Merge branch-trunk-win to trunk

[Chris Douglas <cd...@apache.org>]

Konstantin-

There's no debate on the necessity of CI and related infrastructure to
support the platform well. Suresh outlined the support to effect this
here: http://s.apache.org/s1

Is the commitment to establish this infrastructure after the merge
sufficient? -C

On Fri, Mar 1, 2013 at 12:18 PM, Konstantin Shvachko
<sh...@gmail.com> wrote:
> -1
> We should have a CI infrastructure in place before we can commit to
> supporting Windows platform.
>
> Eric is right Win/Cygwin was supported since day one.
> I had a Windows box under my desk running nightly builds back in 2006-07.
> People were irritated but I was filing windows bugs until 0.22 release.
> Times changing and I am glad to see wider support for Win platform.
>
> But in order to make it work you guys need to put the CI process in place
>
> 1. windows jenkins build: could be nightly or PreCommit.
> - Nightly would mean that changes can be committed to trunk based on
> linux PreCommit build. And people will file bugs if the change broke
> Windows nightly build.
> - PreCommit-win build will mean automatic reporting failed tests to
> respective jira blocking commits the same way as it is now with linux
> PreCommit builds.
> We should discuss which way is more efficient for developers.
>
> 2. On-demand-windows Jenkins build.
> I see it as a build to which I can attach my patch and the build will
> run my changes on a dedicated windows box.
> That way people can test their changes without having personal windows nodes.
>
> I think this is the minimal set of requirement for us to be able to
> commit to the new platform.
> Right now I see only one windows related build
> https://builds.apache.org/view/Hadoop/job/Hadoop-1-win/
> Which was failing since Sept 8, 2012 and did not run in the last month.
>
> Thanks,
> --Konst
>
> On Thu, Feb 28, 2013 at 8:47 PM, Eric Baldeschwieler
> <er...@hortonworks.com> wrote:
>> +1 (non-binding)
>>
>> A few of observations:
>>
>> - Windows has actually been a supported platform for Hadoop since 0.1 .  Doug championed supporting windows then and we've continued to do it with varying vigor over time.  To my knowledge we've never made a decision to drop windows support.  The change here is improving our support and dropping the requirement of cigwin.  We had Nutch windows users on the list in 2006 and we've been supporting windows FS requirements since inception.
>>
>> - A little pragmatism will go a long way.  As a community we've got to stay committed to keeping hadoop simple (so it does work on many platforms) and extending it to take advantage of key emerging OS/hardware features, such as containers, new FSs, virtualization, flash ...  We should all plan to let new features & optimizations emerge that don't work everywhere, if they are compelling and central to hadoop's mission of being THE best fabric for storing and processing big data.
>>
>> - A UI project like KDE has to deal with the MANY differences between windows and linux UI APIs.  Hadoop faces no such complex challenge and hence can be maintained from a single codeline IMO.  It is mostly abstracted from the OS APIs via Java and our design choices.  Where it is not we can continue to add plugable abstractions.
>>

Re: [Vote] Merge branch-trunk-win to trunk

[Konstantin Boudnik <co...@apache.org>]

Here's a daily build for hadoop-2 branch
  https://builds.apache.org/job/Hadoop-branch2

It just builds the full Hadoop project without running any tests (for now).
Can be easily extended to do test runs/artifact deployment, if needed.

Cos

On Mon, Mar 25, 2013 at 07:14PM, Konstantin Boudnik wrote:
> Yes, you are right of course - the mis-merged commit is the cause. Thanks for
> pointing this out!
> 
> I think it would be beneficial if we had branch-2 on going build in the
> Jenkins. I will go ahead and create one tonight.
> 
> Cos
> 
> On Mon, Mar 25, 2013 at 05:09PM, Suresh Srinivas wrote:
> > Adding other mailing lists I missed earlier.
> > 
> > Cos,
> > 
> > There is progress being made on that ticket. Also it has nothing to do with
> > that.
> > 
> > Please follow the discussion here and why this happened due to an invalid
> > commit that was reverted -
> > https://issues.apache.org/jira/browse/HDFS-4615?focusedCommentId=13612650&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13612650
> > 
> > Regards,
> > Suresh
> > 
> > 
> > On Mon, Mar 25, 2013 at 1:17 PM, Konstantin Boudnik <co...@apache.org> wrote:
> > 
> > > It doesn't look like any progress has been done on the ticket below in the
> > > last 3 weeks. And now branch-2 can't be compiled because of
> > >
> > >
> > > hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSShell.java:[895,15]
> > > WINDOWS is not public in org.apache.hadoop.fs.Path; cannot be accessed from
> > > outside package
> > >
> > > That's exactly why I was -1'ing this...
> > >   Cos
> > >
> > > On Mon, Mar 04, 2013 at 05:41PM, Matt Foley wrote:
> > > > Thanks, gentlemen.  I've opened and taken responsibility for
> > > > https://issues.apache.org/jira/browse/HADOOP-9359.  Giri Kesavan has
> > > agreed
> > > > to help with the parts that require Jenkins admin access.
> > > >
> > > > Thanks,
> > > > --Matt
> > > >
> > > >
> > > >
> > > > On Mon, Mar 4, 2013 at 5:00 PM, Konstantin Shvachko <
> > > shv.hadoop@gmail.com>wrote:
> > > >
> > > > > +1 on the merge.
> > > > >
> > > > > I am glad we agreed.
> > > > > Having Jira to track the CI effort is a good idea.
> > > > >
> > > > > Thanks,
> > > > > --Konstantin
> > > > >
> > > > > On Mon, Mar 4, 2013 at 3:29 PM, Matt Foley <mf...@hortonworks.com>
> > > wrote:
> > > > > > Thanks.  I agree Windows -1's in test-patch should not block commits.
> > > > > >
> > > > > > --Matt
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Mon, Mar 4, 2013 at 2:30 PM, Konstantin Shvachko <
> > > > > shv.hadoop@gmail.com>
> > > > > > wrote:
> > > > > >>
> > > > > >> On Mon, Mar 4, 2013 at 12:22 PM, Matt Foley <mfoley@hortonworks.com
> > > >
> > > > > >> wrote:
> > > > > >> > Konstantine, you have voted -1, and stated some requirements
> > > before
> > > > > >> > you'll
> > > > > >> > withdraw that -1.  As I plan to do work to fulfill those
> > > > > requirements, I
> > > > > >> > want to make sure that what I'm proposing will, in fact, satisfy
> > > you.
> > > > > >> > That's why I'm asking, if we implement full "test-patch"
> > > integration
> > > > > for
> > > > > >> > Windows, does it seem to you that that would provide adequate
> > > support?
> > > > > >>
> > > > > >> Yes.
> > > > > >>
> > > > > >> > I have learned not to presume that my interpretation is correct.
> > >  My
> > > > > >> > interpretation of item #1 is that test-patch provides pre-commit
> > > > > build,
> > > > > >> > so
> > > > > >> > it would satisfy item #1.  But rather than assuming that I am
> > > > > >> > interpreting
> > > > > >> > it correctly, I simply want your agreement that it would, or if
> > > not,
> > > > > >> > clarification why it won't.
> > > > > >>
> > > > > >> I agree it will satisfy my item #1.
> > > > > >> I did not agree in my previous email, but I changed my mind based on
> > > > > >> the latest discussion. I have to explain why now.
> > > > > >> I was proposing nightly build because I did not want pre-commit
> > > build
> > > > > >> for Windows block commits to Linux. But if people are fine just
> > > ignoring
> > > > > >> -1s for the Windows part of the build it should be good.
> > > > > >>
> > > > > >> > Regarding item #2, it is also my interpretation that test-patch
> > > > > provides
> > > > > >> > an
> > > > > >> > on-demand (perhaps 20-minutes deferred) Jenkins build and unit
> > > test,
> > > > > >> > with
> > > > > >> > logs available to the developer, so it would satisfy item #2.  But
> > > > > >> > rather
> > > > > >> > than assuming that I am interpreting it correctly, I simply want
> > > your
> > > > > >> > agreement that it would, or if not, clarification why it won't.
> > > > > >>
> > > > > >> It will satisfy my item #2 in the following way:
> > > > > >> I can duplicate your pre-commit build for Windows and add an input
> > > > > >> parameter, which would let people run the build on their patches
> > > > > >> chosen from local machine rather than attaching them to Jiras.
> > > > > >>
> > > > > >> Thanks,
> > > > > >> --Konstantin
> > > > > >>
> > > > > >> > In agile terms, you are the Owner of these requirements.  Please
> > > give
> > > > > me
> > > > > >> > owner feedback as to whether my proposed work sounds like it will
> > > > > >> > satisfy
> > > > > >> > the requirements.
> > > > > >> >
> > > > > >> > Thank you,
> > > > > >> > --Matt
> > > > > >> >
> > > > > >> >
> > > > > >> > On Sun, Mar 3, 2013 at 12:16 PM, Konstantin Shvachko
> > > > > >> > <sh...@gmail.com>
> > > > > >> > wrote:
> > > > > >> >>
> > > > > >> >> Didn't I explain in details what I am asking for?
> > > > > >> >>
> > > > > >> >> Thanks,
> > > > > >> >> --Konst
> > > > > >> >>
> > > > > >> >> On Sun, Mar 3, 2013 at 11:08 AM, Matt Foley <
> > > mfoley@hortonworks.com>
> > > > > >> >> wrote:
> > > > > >> >> > Hi Konstantin,
> > > > > >> >> > I'd like to point out two things:
> > > > > >> >> > First, I already committed in this thread (email of Thu, Feb
> > > 28,
> > > > > 2013
> > > > > >> >> > at
> > > > > >> >> > 6:01 PM) to providing CI for Windows builds.  So please stop
> > > acting
> > > > > >> >> > like
> > > > > >> >> > I'm
> > > > > >> >> > resisting this idea or something.
> > > > > >> >> > Second, you didn't answer my question, you just kvetched about
> > > the
> > > > > >> >> > phrasing.
> > > > > >> >> > So I ask again:
> > > > > >> >> >
> > > > > >> >> > Will providing full "test-patch" integration (pre-commit build
> > > and
> > > > > >> >> > unit
> > > > > >> >> > test
> > > > > >> >> > triggered by Jira "Patch Available" state) satisfy your
> > > request for
> > > > > >> >> > functionality #1 and #2?  Yes or no, please.
> > > > > >> >> >
> > > > > >> >> > Thanks,
> > > > > >> >> > --Matt
> > > > > >> >> >
> > > > > >> >> >
> > > > > >> >> > On Sat, Mar 2, 2013 at 7:32 PM, Konstantin Shvachko
> > > > > >> >> > <sh...@gmail.com>
> > > > > >> >> > wrote:
> > > > > >> >> >>
> > > > > >> >> >> Hi Matt,
> > > > > >> >> >>
> > > > > >> >> >> On Sat, Mar 2, 2013 at 12:32 PM, Matt Foley <
> > > > > mfoley@hortonworks.com>
> > > > > >> >> >> wrote:
> > > > > >> >> >> > Konstantin,
> > > > > >> >> >> > I would like to explore what it would take to remove this
> > > > > >> >> >> > perceived
> > > > > >> >> >> > impediment --
> > > > > >> >> >>
> > > > > >> >> >> Glad you decided to explore. Thank you.
> > > > > >> >> >>
> > > > > >> >> >> > although I reserve the right to argue that this is not
> > > > > >> >> >> > pre-requisite to merging the cross-platform support patch.
> > > > > >> >> >>
> > > > > >> >> >> It's your right indeed. So as mine to question what the
> > > platform
> > > > > >> >> >> support means for you, which I believe remained unclear.
> > > > > >> >> >> I do not impede the change as you should have noticed. My
> > > > > >> >> >> requirement
> > > > > >> >> >> comes from my perception of the support, which means to me
> > > exactly
> > > > > >> >> >> two
> > > > > >> >> >> things:
> > > > > >> >> >> 1. The ability to recognise the code is broken for the
> > > platform
> > > > > >> >> >> 2. The ability to test new patches on the platform
> > > > > >> >> >> The latter is problematic, as many noticed in this thread, for
> > > > > those
> > > > > >> >> >> whose customary environment does not include Windows.
> > > > > >> >> >>
> > > > > >> >> >> > If we implemented full "test-patch" support for Windows on
> > > > > trunk,
> > > > > >> >> >> > would
> > > > > >> >> >> > that
> > > > > >> >> >> > fulfill both your items #1 and #2?  Please note that:
> > > > > >> >> >> > a) Pushing the "Patch Available" button in Jira shall cause
> > > a
> > > > > >> >> >> > pre-commit
> > > > > >> >> >> > build to start within, I believe, 20 minutes.
> > > > > >> >> >> > b) That build keeps logs for both java build and unit tests
> > > for
> > > > > >> >> >> > several
> > > > > >> >> >> > days, that are accessible to all viewers.
> > > > > >> >> >>
> > > > > >> >> >> In item #1 I mostly asking for the nightly build, which is
> > > simpler
> > > > > >> >> >> than "test-patch". The latter would be ideal from the platform
> > > > > >> >> >> support
> > > > > >> >> >> viewpoint, but it is for the community to decide if we want
> > > to add
> > > > > >> >> >> extra +3 hours to the build.
> > > > > >> >> >> Nightly build in my understanding is triggered by the timer
> > > rather
> > > > > >> >> >> than by Jira's "submit patch" button.  On Jenkins build
> > > > > >> >> >> configuration
> > > > > >> >> >> you can specify it under "Build periodically".
> > > > > >> >> >>
> > > > > >> >> >> > So, does this provide sufficient on-demand support that we
> > > don't
> > > > > >> >> >> > have
> > > > > >> >> >> > to
> > > > > >> >> >> > implement a whole new on-demand VM support structure of some
> > > > > sort
> > > > > >> >> >> > for
> > > > > >> >> >> > #2
> > > > > >> >> >> > (which would be an extraordinary and impractical demand)?
> > > > > >> >> >>
> > > > > >> >> >> I did not mention VMs. Item #2 means a build, which runs
> > > > > >> >> >> "test-patch"
> > > > > >> >> >> target with the file specified by a user (instead of a jira
> > > > > >> >> >> attachment).
> > > > > >> >> >> When user clicks "Build Now" link a box is displayed where the
> > > > > user
> > > > > >> >> >> can enter the file path containing the patch. This can be
> > > > > specified
> > > > > >> >> >> in
> > > > > >> >> >> the Build Configuration under "This build is parameterized" by
> > > > > >> >> >> choosing AddParameter / FileParameter. The build can run on
> > > the
> > > > > same
> > > > > >> >> >> Windows machine as the nightly build.
> > > > > >> >> >> Such build will let people test their patches for Windows on
> > > > > Jenkins
> > > > > >> >> >> if they don't posses a license for the right version of
> > > Windows.
> > > > > >> >> >> I hope this will not turn into extraordinary or impractical
> > > > > effort.
> > > > > >> >> >>
> > > > > >> >> >> Thanks,
> > > > > >> >> >> --Konst
> > > > > >> >> >>
> > > > > >> >> >> > Thanks,
> > > > > >> >> >> > --Matt
> > > > > >> >> >> >
> > > > > >> >> >> >
> > > > > >> >> >> > On Fri, Mar 1, 2013 at 12:18 PM, Konstantin Shvachko
> > > > > >> >> >> > <sh...@gmail.com>
> > > > > >> >> >> > wrote:
> > > > > >> >> >> >>
> > > > > >> >> >> >> -1
> > > > > >> >> >> >> We should have a CI infrastructure in place before we can
> > > > > commit
> > > > > >> >> >> >> to
> > > > > >> >> >> >> supporting Windows platform.
> > > > > >> >> >> >>
> > > > > >> >> >> >> Eric is right Win/Cygwin was supported since day one.
> > > > > >> >> >> >> I had a Windows box under my desk running nightly builds
> > > back
> > > > > in
> > > > > >> >> >> >> 2006-07.
> > > > > >> >> >> >> People were irritated but I was filing windows bugs until
> > > 0.22
> > > > > >> >> >> >> release.
> > > > > >> >> >> >> Times changing and I am glad to see wider support for Win
> > > > > >> >> >> >> platform.
> > > > > >> >> >> >>
> > > > > >> >> >> >> But in order to make it work you guys need to put the CI
> > > > > process
> > > > > >> >> >> >> in
> > > > > >> >> >> >> place
> > > > > >> >> >> >>
> > > > > >> >> >> >> 1. windows jenkins build: could be nightly or PreCommit.
> > > > > >> >> >> >> - Nightly would mean that changes can be committed to trunk
> > > > > based
> > > > > >> >> >> >> on
> > > > > >> >> >> >> linux PreCommit build. And people will file bugs if the
> > > change
> > > > > >> >> >> >> broke
> > > > > >> >> >> >> Windows nightly build.
> > > > > >> >> >> >> - PreCommit-win build will mean automatic reporting failed
> > > > > tests
> > > > > >> >> >> >> to
> > > > > >> >> >> >> respective jira blocking commits the same way as it is now
> > > with
> > > > > >> >> >> >> linux
> > > > > >> >> >> >> PreCommit builds.
> > > > > >> >> >> >> We should discuss which way is more efficient for
> > > developers.
> > > > > >> >> >> >>
> > > > > >> >> >> >> 2. On-demand-windows Jenkins build.
> > > > > >> >> >> >> I see it as a build to which I can attach my patch and the
> > > > > build
> > > > > >> >> >> >> will
> > > > > >> >> >> >> run my changes on a dedicated windows box.
> > > > > >> >> >> >> That way people can test their changes without having
> > > personal
> > > > > >> >> >> >> windows
> > > > > >> >> >> >> nodes.
> > > > > >> >> >> >>
> > > > > >> >> >> >> I think this is the minimal set of requirement for us to be
> > > > > able
> > > > > >> >> >> >> to
> > > > > >> >> >> >> commit to the new platform.
> > > > > >> >> >> >> Right now I see only one windows related build
> > > > > >> >> >> >> https://builds.apache.org/view/Hadoop/job/Hadoop-1-win/
> > > > > >> >> >> >> Which was failing since Sept 8, 2012 and did not run in the
> > > > > last
> > > > > >> >> >> >> month.
> > > > > >> >> >> >>
> > > > > >> >> >> >> Thanks,
> > > > > >> >> >> >> --Konst
> > > > > >> >> >> >>
> > > > > >> >> >> >> On Thu, Feb 28, 2013 at 8:47 PM, Eric Baldeschwieler
> > > > > >> >> >> >> <er...@hortonworks.com> wrote:
> > > > > >> >> >> >> > +1 (non-binding)
> > > > > >> >> >> >> >
> > > > > >> >> >> >> > A few of observations:
> > > > > >> >> >> >> >
> > > > > >> >> >> >> > - Windows has actually been a supported platform for
> > > Hadoop
> > > > > >> >> >> >> > since
> > > > > >> >> >> >> > 0.1
> > > > > >> >> >> >> > .
> > > > > >> >> >> >> > Doug championed supporting windows then and we've
> > > continued
> > > > > to
> > > > > >> >> >> >> > do
> > > > > >> >> >> >> > it
> > > > > >> >> >> >> > with
> > > > > >> >> >> >> > varying vigor over time.  To my knowledge we've never
> > > made a
> > > > > >> >> >> >> > decision
> > > > > >> >> >> >> > to
> > > > > >> >> >> >> > drop windows support.  The change here is improving our
> > > > > support
> > > > > >> >> >> >> > and
> > > > > >> >> >> >> > dropping
> > > > > >> >> >> >> > the requirement of cigwin.  We had Nutch windows users
> > > on the
> > > > > >> >> >> >> > list
> > > > > >> >> >> >> > in
> > > > > >> >> >> >> > 2006
> > > > > >> >> >> >> > and we've been supporting windows FS requirements since
> > > > > >> >> >> >> > inception.
> > > > > >> >> >> >> >
> > > > > >> >> >> >> > - A little pragmatism will go a long way.  As a community
> > > > > we've
> > > > > >> >> >> >> > got
> > > > > >> >> >> >> > to
> > > > > >> >> >> >> > stay committed to keeping hadoop simple (so it does work
> > > on
> > > > > >> >> >> >> > many
> > > > > >> >> >> >> > platforms)
> > > > > >> >> >> >> > and extending it to take advantage of key emerging
> > > > > OS/hardware
> > > > > >> >> >> >> > features,
> > > > > >> >> >> >> > such as containers, new FSs, virtualization, flash ...
> > >  We
> > > > > >> >> >> >> > should
> > > > > >> >> >> >> > all
> > > > > >> >> >> >> > plan
> > > > > >> >> >> >> > to let new features & optimizations emerge that don't
> > > work
> > > > > >> >> >> >> > everywhere, if
> > > > > >> >> >> >> > they are compelling and central to hadoop's mission of
> > > being
> > > > > >> >> >> >> > THE
> > > > > >> >> >> >> > best
> > > > > >> >> >> >> > fabric
> > > > > >> >> >> >> > for storing and processing big data.
> > > > > >> >> >> >> >
> > > > > >> >> >> >> > - A UI project like KDE has to deal with the MANY
> > > differences
> > > > > >> >> >> >> > between
> > > > > >> >> >> >> > windows and linux UI APIs.  Hadoop faces no such complex
> > > > > >> >> >> >> > challenge
> > > > > >> >> >> >> > and hence
> > > > > >> >> >> >> > can be maintained from a single codeline IMO.  It is
> > > mostly
> > > > > >> >> >> >> > abstracted from
> > > > > >> >> >> >> > the OS APIs via Java and our design choices.  Where it
> > > is not
> > > > > >> >> >> >> > we
> > > > > >> >> >> >> > can
> > > > > >> >> >> >> > continue to add plugable abstractions.
> > > > > >> >> >> >> >
> > > > > >> >> >> >
> > > > > >> >> >> >
> > > > > >> >> >
> > > > > >> >> >
> > > > > >> >
> > > > > >> >
> > > > > >
> > > > > >
> > > > >
> > >
> > 
> > 
> > 
> > -- 
> > http://hortonworks.com/download/

Re: [Vote] Merge branch-trunk-win to trunk

[Konstantin Boudnik <co...@apache.org>]

Here's a daily build for hadoop-2 branch
  https://builds.apache.org/job/Hadoop-branch2

It just builds the full Hadoop project without running any tests (for now).
Can be easily extended to do test runs/artifact deployment, if needed.

Cos

On Mon, Mar 25, 2013 at 07:14PM, Konstantin Boudnik wrote:
> Yes, you are right of course - the mis-merged commit is the cause. Thanks for
> pointing this out!
> 
> I think it would be beneficial if we had branch-2 on going build in the
> Jenkins. I will go ahead and create one tonight.
> 
> Cos
> 
> On Mon, Mar 25, 2013 at 05:09PM, Suresh Srinivas wrote:
> > Adding other mailing lists I missed earlier.
> > 
> > Cos,
> > 
> > There is progress being made on that ticket. Also it has nothing to do with
> > that.
> > 
> > Please follow the discussion here and why this happened due to an invalid
> > commit that was reverted -
> > https://issues.apache.org/jira/browse/HDFS-4615?focusedCommentId=13612650&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13612650
> > 
> > Regards,
> > Suresh
> > 
> > 
> > On Mon, Mar 25, 2013 at 1:17 PM, Konstantin Boudnik <co...@apache.org> wrote:
> > 
> > > It doesn't look like any progress has been done on the ticket below in the
> > > last 3 weeks. And now branch-2 can't be compiled because of
> > >
> > >
> > > hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSShell.java:[895,15]
> > > WINDOWS is not public in org.apache.hadoop.fs.Path; cannot be accessed from
> > > outside package
> > >
> > > That's exactly why I was -1'ing this...
> > >   Cos
> > >
> > > On Mon, Mar 04, 2013 at 05:41PM, Matt Foley wrote:
> > > > Thanks, gentlemen.  I've opened and taken responsibility for
> > > > https://issues.apache.org/jira/browse/HADOOP-9359.  Giri Kesavan has
> > > agreed
> > > > to help with the parts that require Jenkins admin access.
> > > >
> > > > Thanks,
> > > > --Matt
> > > >
> > > >
> > > >
> > > > On Mon, Mar 4, 2013 at 5:00 PM, Konstantin Shvachko <
> > > shv.hadoop@gmail.com>wrote:
> > > >
> > > > > +1 on the merge.
> > > > >
> > > > > I am glad we agreed.
> > > > > Having Jira to track the CI effort is a good idea.
> > > > >
> > > > > Thanks,
> > > > > --Konstantin
> > > > >
> > > > > On Mon, Mar 4, 2013 at 3:29 PM, Matt Foley <mf...@hortonworks.com>
> > > wrote:
> > > > > > Thanks.  I agree Windows -1's in test-patch should not block commits.
> > > > > >
> > > > > > --Matt
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Mon, Mar 4, 2013 at 2:30 PM, Konstantin Shvachko <
> > > > > shv.hadoop@gmail.com>
> > > > > > wrote:
> > > > > >>
> > > > > >> On Mon, Mar 4, 2013 at 12:22 PM, Matt Foley <mfoley@hortonworks.com
> > > >
> > > > > >> wrote:
> > > > > >> > Konstantine, you have voted -1, and stated some requirements
> > > before
> > > > > >> > you'll
> > > > > >> > withdraw that -1.  As I plan to do work to fulfill those
> > > > > requirements, I
> > > > > >> > want to make sure that what I'm proposing will, in fact, satisfy
> > > you.
> > > > > >> > That's why I'm asking, if we implement full "test-patch"
> > > integration
> > > > > for
> > > > > >> > Windows, does it seem to you that that would provide adequate
> > > support?
> > > > > >>
> > > > > >> Yes.
> > > > > >>
> > > > > >> > I have learned not to presume that my interpretation is correct.
> > >  My
> > > > > >> > interpretation of item #1 is that test-patch provides pre-commit
> > > > > build,
> > > > > >> > so
> > > > > >> > it would satisfy item #1.  But rather than assuming that I am
> > > > > >> > interpreting
> > > > > >> > it correctly, I simply want your agreement that it would, or if
> > > not,
> > > > > >> > clarification why it won't.
> > > > > >>
> > > > > >> I agree it will satisfy my item #1.
> > > > > >> I did not agree in my previous email, but I changed my mind based on
> > > > > >> the latest discussion. I have to explain why now.
> > > > > >> I was proposing nightly build because I did not want pre-commit
> > > build
> > > > > >> for Windows block commits to Linux. But if people are fine just
> > > ignoring
> > > > > >> -1s for the Windows part of the build it should be good.
> > > > > >>
> > > > > >> > Regarding item #2, it is also my interpretation that test-patch
> > > > > provides
> > > > > >> > an
> > > > > >> > on-demand (perhaps 20-minutes deferred) Jenkins build and unit
> > > test,
> > > > > >> > with
> > > > > >> > logs available to the developer, so it would satisfy item #2.  But
> > > > > >> > rather
> > > > > >> > than assuming that I am interpreting it correctly, I simply want
> > > your
> > > > > >> > agreement that it would, or if not, clarification why it won't.
> > > > > >>
> > > > > >> It will satisfy my item #2 in the following way:
> > > > > >> I can duplicate your pre-commit build for Windows and add an input
> > > > > >> parameter, which would let people run the build on their patches
> > > > > >> chosen from local machine rather than attaching them to Jiras.
> > > > > >>
> > > > > >> Thanks,
> > > > > >> --Konstantin
> > > > > >>
> > > > > >> > In agile terms, you are the Owner of these requirements.  Please
> > > give
> > > > > me
> > > > > >> > owner feedback as to whether my proposed work sounds like it will
> > > > > >> > satisfy
> > > > > >> > the requirements.
> > > > > >> >
> > > > > >> > Thank you,
> > > > > >> > --Matt
> > > > > >> >
> > > > > >> >
> > > > > >> > On Sun, Mar 3, 2013 at 12:16 PM, Konstantin Shvachko
> > > > > >> > <sh...@gmail.com>
> > > > > >> > wrote:
> > > > > >> >>
> > > > > >> >> Didn't I explain in details what I am asking for?
> > > > > >> >>
> > > > > >> >> Thanks,
> > > > > >> >> --Konst
> > > > > >> >>
> > > > > >> >> On Sun, Mar 3, 2013 at 11:08 AM, Matt Foley <
> > > mfoley@hortonworks.com>
> > > > > >> >> wrote:
> > > > > >> >> > Hi Konstantin,
> > > > > >> >> > I'd like to point out two things:
> > > > > >> >> > First, I already committed in this thread (email of Thu, Feb
> > > 28,
> > > > > 2013
> > > > > >> >> > at
> > > > > >> >> > 6:01 PM) to providing CI for Windows builds.  So please stop
> > > acting
> > > > > >> >> > like
> > > > > >> >> > I'm
> > > > > >> >> > resisting this idea or something.
> > > > > >> >> > Second, you didn't answer my question, you just kvetched about
> > > the
> > > > > >> >> > phrasing.
> > > > > >> >> > So I ask again:
> > > > > >> >> >
> > > > > >> >> > Will providing full "test-patch" integration (pre-commit build
> > > and
> > > > > >> >> > unit
> > > > > >> >> > test
> > > > > >> >> > triggered by Jira "Patch Available" state) satisfy your
> > > request for
> > > > > >> >> > functionality #1 and #2?  Yes or no, please.
> > > > > >> >> >
> > > > > >> >> > Thanks,
> > > > > >> >> > --Matt
> > > > > >> >> >
> > > > > >> >> >
> > > > > >> >> > On Sat, Mar 2, 2013 at 7:32 PM, Konstantin Shvachko
> > > > > >> >> > <sh...@gmail.com>
> > > > > >> >> > wrote:
> > > > > >> >> >>
> > > > > >> >> >> Hi Matt,
> > > > > >> >> >>
> > > > > >> >> >> On Sat, Mar 2, 2013 at 12:32 PM, Matt Foley <
> > > > > mfoley@hortonworks.com>
> > > > > >> >> >> wrote:
> > > > > >> >> >> > Konstantin,
> > > > > >> >> >> > I would like to explore what it would take to remove this
> > > > > >> >> >> > perceived
> > > > > >> >> >> > impediment --
> > > > > >> >> >>
> > > > > >> >> >> Glad you decided to explore. Thank you.
> > > > > >> >> >>
> > > > > >> >> >> > although I reserve the right to argue that this is not
> > > > > >> >> >> > pre-requisite to merging the cross-platform support patch.
> > > > > >> >> >>
> > > > > >> >> >> It's your right indeed. So as mine to question what the
> > > platform
> > > > > >> >> >> support means for you, which I believe remained unclear.
> > > > > >> >> >> I do not impede the change as you should have noticed. My
> > > > > >> >> >> requirement
> > > > > >> >> >> comes from my perception of the support, which means to me
> > > exactly
> > > > > >> >> >> two
> > > > > >> >> >> things:
> > > > > >> >> >> 1. The ability to recognise the code is broken for the
> > > platform
> > > > > >> >> >> 2. The ability to test new patches on the platform
> > > > > >> >> >> The latter is problematic, as many noticed in this thread, for
> > > > > those
> > > > > >> >> >> whose customary environment does not include Windows.
> > > > > >> >> >>
> > > > > >> >> >> > If we implemented full "test-patch" support for Windows on
> > > > > trunk,
> > > > > >> >> >> > would
> > > > > >> >> >> > that
> > > > > >> >> >> > fulfill both your items #1 and #2?  Please note that:
> > > > > >> >> >> > a) Pushing the "Patch Available" button in Jira shall cause
> > > a
> > > > > >> >> >> > pre-commit
> > > > > >> >> >> > build to start within, I believe, 20 minutes.
> > > > > >> >> >> > b) That build keeps logs for both java build and unit tests
> > > for
> > > > > >> >> >> > several
> > > > > >> >> >> > days, that are accessible to all viewers.
> > > > > >> >> >>
> > > > > >> >> >> In item #1 I mostly asking for the nightly build, which is
> > > simpler
> > > > > >> >> >> than "test-patch". The latter would be ideal from the platform
> > > > > >> >> >> support
> > > > > >> >> >> viewpoint, but it is for the community to decide if we want
> > > to add
> > > > > >> >> >> extra +3 hours to the build.
> > > > > >> >> >> Nightly build in my understanding is triggered by the timer
> > > rather
> > > > > >> >> >> than by Jira's "submit patch" button.  On Jenkins build
> > > > > >> >> >> configuration
> > > > > >> >> >> you can specify it under "Build periodically".
> > > > > >> >> >>
> > > > > >> >> >> > So, does this provide sufficient on-demand support that we
> > > don't
> > > > > >> >> >> > have
> > > > > >> >> >> > to
> > > > > >> >> >> > implement a whole new on-demand VM support structure of some
> > > > > sort
> > > > > >> >> >> > for
> > > > > >> >> >> > #2
> > > > > >> >> >> > (which would be an extraordinary and impractical demand)?
> > > > > >> >> >>
> > > > > >> >> >> I did not mention VMs. Item #2 means a build, which runs
> > > > > >> >> >> "test-patch"
> > > > > >> >> >> target with the file specified by a user (instead of a jira
> > > > > >> >> >> attachment).
> > > > > >> >> >> When user clicks "Build Now" link a box is displayed where the
> > > > > user
> > > > > >> >> >> can enter the file path containing the patch. This can be
> > > > > specified
> > > > > >> >> >> in
> > > > > >> >> >> the Build Configuration under "This build is parameterized" by
> > > > > >> >> >> choosing AddParameter / FileParameter. The build can run on
> > > the
> > > > > same
> > > > > >> >> >> Windows machine as the nightly build.
> > > > > >> >> >> Such build will let people test their patches for Windows on
> > > > > Jenkins
> > > > > >> >> >> if they don't posses a license for the right version of
> > > Windows.
> > > > > >> >> >> I hope this will not turn into extraordinary or impractical
> > > > > effort.
> > > > > >> >> >>
> > > > > >> >> >> Thanks,
> > > > > >> >> >> --Konst
> > > > > >> >> >>
> > > > > >> >> >> > Thanks,
> > > > > >> >> >> > --Matt
> > > > > >> >> >> >
> > > > > >> >> >> >
> > > > > >> >> >> > On Fri, Mar 1, 2013 at 12:18 PM, Konstantin Shvachko
> > > > > >> >> >> > <sh...@gmail.com>
> > > > > >> >> >> > wrote:
> > > > > >> >> >> >>
> > > > > >> >> >> >> -1
> > > > > >> >> >> >> We should have a CI infrastructure in place before we can
> > > > > commit
> > > > > >> >> >> >> to
> > > > > >> >> >> >> supporting Windows platform.
> > > > > >> >> >> >>
> > > > > >> >> >> >> Eric is right Win/Cygwin was supported since day one.
> > > > > >> >> >> >> I had a Windows box under my desk running nightly builds
> > > back
> > > > > in
> > > > > >> >> >> >> 2006-07.
> > > > > >> >> >> >> People were irritated but I was filing windows bugs until
> > > 0.22
> > > > > >> >> >> >> release.
> > > > > >> >> >> >> Times changing and I am glad to see wider support for Win
> > > > > >> >> >> >> platform.
> > > > > >> >> >> >>
> > > > > >> >> >> >> But in order to make it work you guys need to put the CI
> > > > > process
> > > > > >> >> >> >> in
> > > > > >> >> >> >> place
> > > > > >> >> >> >>
> > > > > >> >> >> >> 1. windows jenkins build: could be nightly or PreCommit.
> > > > > >> >> >> >> - Nightly would mean that changes can be committed to trunk
> > > > > based
> > > > > >> >> >> >> on
> > > > > >> >> >> >> linux PreCommit build. And people will file bugs if the
> > > change
> > > > > >> >> >> >> broke
> > > > > >> >> >> >> Windows nightly build.
> > > > > >> >> >> >> - PreCommit-win build will mean automatic reporting failed
> > > > > tests
> > > > > >> >> >> >> to
> > > > > >> >> >> >> respective jira blocking commits the same way as it is now
> > > with
> > > > > >> >> >> >> linux
> > > > > >> >> >> >> PreCommit builds.
> > > > > >> >> >> >> We should discuss which way is more efficient for
> > > developers.
> > > > > >> >> >> >>
> > > > > >> >> >> >> 2. On-demand-windows Jenkins build.
> > > > > >> >> >> >> I see it as a build to which I can attach my patch and the
> > > > > build
> > > > > >> >> >> >> will
> > > > > >> >> >> >> run my changes on a dedicated windows box.
> > > > > >> >> >> >> That way people can test their changes without having
> > > personal
> > > > > >> >> >> >> windows
> > > > > >> >> >> >> nodes.
> > > > > >> >> >> >>
> > > > > >> >> >> >> I think this is the minimal set of requirement for us to be
> > > > > able
> > > > > >> >> >> >> to
> > > > > >> >> >> >> commit to the new platform.
> > > > > >> >> >> >> Right now I see only one windows related build
> > > > > >> >> >> >> https://builds.apache.org/view/Hadoop/job/Hadoop-1-win/
> > > > > >> >> >> >> Which was failing since Sept 8, 2012 and did not run in the
> > > > > last
> > > > > >> >> >> >> month.
> > > > > >> >> >> >>
> > > > > >> >> >> >> Thanks,
> > > > > >> >> >> >> --Konst
> > > > > >> >> >> >>
> > > > > >> >> >> >> On Thu, Feb 28, 2013 at 8:47 PM, Eric Baldeschwieler
> > > > > >> >> >> >> <er...@hortonworks.com> wrote:
> > > > > >> >> >> >> > +1 (non-binding)
> > > > > >> >> >> >> >
> > > > > >> >> >> >> > A few of observations:
> > > > > >> >> >> >> >
> > > > > >> >> >> >> > - Windows has actually been a supported platform for
> > > Hadoop
> > > > > >> >> >> >> > since
> > > > > >> >> >> >> > 0.1
> > > > > >> >> >> >> > .
> > > > > >> >> >> >> > Doug championed supporting windows then and we've
> > > continued
> > > > > to
> > > > > >> >> >> >> > do
> > > > > >> >> >> >> > it
> > > > > >> >> >> >> > with
> > > > > >> >> >> >> > varying vigor over time.  To my knowledge we've never
> > > made a
> > > > > >> >> >> >> > decision
> > > > > >> >> >> >> > to
> > > > > >> >> >> >> > drop windows support.  The change here is improving our
> > > > > support
> > > > > >> >> >> >> > and
> > > > > >> >> >> >> > dropping
> > > > > >> >> >> >> > the requirement of cigwin.  We had Nutch windows users
> > > on the
> > > > > >> >> >> >> > list
> > > > > >> >> >> >> > in
> > > > > >> >> >> >> > 2006
> > > > > >> >> >> >> > and we've been supporting windows FS requirements since
> > > > > >> >> >> >> > inception.
> > > > > >> >> >> >> >
> > > > > >> >> >> >> > - A little pragmatism will go a long way.  As a community
> > > > > we've
> > > > > >> >> >> >> > got
> > > > > >> >> >> >> > to
> > > > > >> >> >> >> > stay committed to keeping hadoop simple (so it does work
> > > on
> > > > > >> >> >> >> > many
> > > > > >> >> >> >> > platforms)
> > > > > >> >> >> >> > and extending it to take advantage of key emerging
> > > > > OS/hardware
> > > > > >> >> >> >> > features,
> > > > > >> >> >> >> > such as containers, new FSs, virtualization, flash ...
> > >  We
> > > > > >> >> >> >> > should
> > > > > >> >> >> >> > all
> > > > > >> >> >> >> > plan
> > > > > >> >> >> >> > to let new features & optimizations emerge that don't
> > > work
> > > > > >> >> >> >> > everywhere, if
> > > > > >> >> >> >> > they are compelling and central to hadoop's mission of
> > > being
> > > > > >> >> >> >> > THE
> > > > > >> >> >> >> > best
> > > > > >> >> >> >> > fabric
> > > > > >> >> >> >> > for storing and processing big data.
> > > > > >> >> >> >> >
> > > > > >> >> >> >> > - A UI project like KDE has to deal with the MANY
> > > differences
> > > > > >> >> >> >> > between
> > > > > >> >> >> >> > windows and linux UI APIs.  Hadoop faces no such complex
> > > > > >> >> >> >> > challenge
> > > > > >> >> >> >> > and hence
> > > > > >> >> >> >> > can be maintained from a single codeline IMO.  It is
> > > mostly
> > > > > >> >> >> >> > abstracted from
> > > > > >> >> >> >> > the OS APIs via Java and our design choices.  Where it
> > > is not
> > > > > >> >> >> >> > we
> > > > > >> >> >> >> > can
> > > > > >> >> >> >> > continue to add plugable abstractions.
> > > > > >> >> >> >> >
> > > > > >> >> >> >
> > > > > >> >> >> >
> > > > > >> >> >
> > > > > >> >> >
> > > > > >> >
> > > > > >> >
> > > > > >
> > > > > >
> > > > >
> > >
> > 
> > 
> > 
> > -- 
> > http://hortonworks.com/download/

Re: [Vote] Merge branch-trunk-win to trunk

[Konstantin Boudnik <co...@apache.org>]

Here's a daily build for hadoop-2 branch
  https://builds.apache.org/job/Hadoop-branch2

It just builds the full Hadoop project without running any tests (for now).
Can be easily extended to do test runs/artifact deployment, if needed.

Cos

On Mon, Mar 25, 2013 at 07:14PM, Konstantin Boudnik wrote:
> Yes, you are right of course - the mis-merged commit is the cause. Thanks for
> pointing this out!
> 
> I think it would be beneficial if we had branch-2 on going build in the
> Jenkins. I will go ahead and create one tonight.
> 
> Cos
> 
> On Mon, Mar 25, 2013 at 05:09PM, Suresh Srinivas wrote:
> > Adding other mailing lists I missed earlier.
> > 
> > Cos,
> > 
> > There is progress being made on that ticket. Also it has nothing to do with
> > that.
> > 
> > Please follow the discussion here and why this happened due to an invalid
> > commit that was reverted -
> > https://issues.apache.org/jira/browse/HDFS-4615?focusedCommentId=13612650&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13612650
> > 
> > Regards,
> > Suresh
> > 
> > 
> > On Mon, Mar 25, 2013 at 1:17 PM, Konstantin Boudnik <co...@apache.org> wrote:
> > 
> > > It doesn't look like any progress has been done on the ticket below in the
> > > last 3 weeks. And now branch-2 can't be compiled because of
> > >
> > >
> > > hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSShell.java:[895,15]
> > > WINDOWS is not public in org.apache.hadoop.fs.Path; cannot be accessed from
> > > outside package
> > >
> > > That's exactly why I was -1'ing this...
> > >   Cos
> > >
> > > On Mon, Mar 04, 2013 at 05:41PM, Matt Foley wrote:
> > > > Thanks, gentlemen.  I've opened and taken responsibility for
> > > > https://issues.apache.org/jira/browse/HADOOP-9359.  Giri Kesavan has
> > > agreed
> > > > to help with the parts that require Jenkins admin access.
> > > >
> > > > Thanks,
> > > > --Matt
> > > >
> > > >
> > > >
> > > > On Mon, Mar 4, 2013 at 5:00 PM, Konstantin Shvachko <
> > > shv.hadoop@gmail.com>wrote:
> > > >
> > > > > +1 on the merge.
> > > > >
> > > > > I am glad we agreed.
> > > > > Having Jira to track the CI effort is a good idea.
> > > > >
> > > > > Thanks,
> > > > > --Konstantin
> > > > >
> > > > > On Mon, Mar 4, 2013 at 3:29 PM, Matt Foley <mf...@hortonworks.com>
> > > wrote:
> > > > > > Thanks.  I agree Windows -1's in test-patch should not block commits.
> > > > > >
> > > > > > --Matt
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Mon, Mar 4, 2013 at 2:30 PM, Konstantin Shvachko <
> > > > > shv.hadoop@gmail.com>
> > > > > > wrote:
> > > > > >>
> > > > > >> On Mon, Mar 4, 2013 at 12:22 PM, Matt Foley <mfoley@hortonworks.com
> > > >
> > > > > >> wrote:
> > > > > >> > Konstantine, you have voted -1, and stated some requirements
> > > before
> > > > > >> > you'll
> > > > > >> > withdraw that -1.  As I plan to do work to fulfill those
> > > > > requirements, I
> > > > > >> > want to make sure that what I'm proposing will, in fact, satisfy
> > > you.
> > > > > >> > That's why I'm asking, if we implement full "test-patch"
> > > integration
> > > > > for
> > > > > >> > Windows, does it seem to you that that would provide adequate
> > > support?
> > > > > >>
> > > > > >> Yes.
> > > > > >>
> > > > > >> > I have learned not to presume that my interpretation is correct.
> > >  My
> > > > > >> > interpretation of item #1 is that test-patch provides pre-commit
> > > > > build,
> > > > > >> > so
> > > > > >> > it would satisfy item #1.  But rather than assuming that I am
> > > > > >> > interpreting
> > > > > >> > it correctly, I simply want your agreement that it would, or if
> > > not,
> > > > > >> > clarification why it won't.
> > > > > >>
> > > > > >> I agree it will satisfy my item #1.
> > > > > >> I did not agree in my previous email, but I changed my mind based on
> > > > > >> the latest discussion. I have to explain why now.
> > > > > >> I was proposing nightly build because I did not want pre-commit
> > > build
> > > > > >> for Windows block commits to Linux. But if people are fine just
> > > ignoring
> > > > > >> -1s for the Windows part of the build it should be good.
> > > > > >>
> > > > > >> > Regarding item #2, it is also my interpretation that test-patch
> > > > > provides
> > > > > >> > an
> > > > > >> > on-demand (perhaps 20-minutes deferred) Jenkins build and unit
> > > test,
> > > > > >> > with
> > > > > >> > logs available to the developer, so it would satisfy item #2.  But
> > > > > >> > rather
> > > > > >> > than assuming that I am interpreting it correctly, I simply want
> > > your
> > > > > >> > agreement that it would, or if not, clarification why it won't.
> > > > > >>
> > > > > >> It will satisfy my item #2 in the following way:
> > > > > >> I can duplicate your pre-commit build for Windows and add an input
> > > > > >> parameter, which would let people run the build on their patches
> > > > > >> chosen from local machine rather than attaching them to Jiras.
> > > > > >>
> > > > > >> Thanks,
> > > > > >> --Konstantin
> > > > > >>
> > > > > >> > In agile terms, you are the Owner of these requirements.  Please
> > > give
> > > > > me
> > > > > >> > owner feedback as to whether my proposed work sounds like it will
> > > > > >> > satisfy
> > > > > >> > the requirements.
> > > > > >> >
> > > > > >> > Thank you,
> > > > > >> > --Matt
> > > > > >> >
> > > > > >> >
> > > > > >> > On Sun, Mar 3, 2013 at 12:16 PM, Konstantin Shvachko
> > > > > >> > <sh...@gmail.com>
> > > > > >> > wrote:
> > > > > >> >>
> > > > > >> >> Didn't I explain in details what I am asking for?
> > > > > >> >>
> > > > > >> >> Thanks,
> > > > > >> >> --Konst
> > > > > >> >>
> > > > > >> >> On Sun, Mar 3, 2013 at 11:08 AM, Matt Foley <
> > > mfoley@hortonworks.com>
> > > > > >> >> wrote:
> > > > > >> >> > Hi Konstantin,
> > > > > >> >> > I'd like to point out two things:
> > > > > >> >> > First, I already committed in this thread (email of Thu, Feb
> > > 28,
> > > > > 2013
> > > > > >> >> > at
> > > > > >> >> > 6:01 PM) to providing CI for Windows builds.  So please stop
> > > acting
> > > > > >> >> > like
> > > > > >> >> > I'm
> > > > > >> >> > resisting this idea or something.
> > > > > >> >> > Second, you didn't answer my question, you just kvetched about
> > > the
> > > > > >> >> > phrasing.
> > > > > >> >> > So I ask again:
> > > > > >> >> >
> > > > > >> >> > Will providing full "test-patch" integration (pre-commit build
> > > and
> > > > > >> >> > unit
> > > > > >> >> > test
> > > > > >> >> > triggered by Jira "Patch Available" state) satisfy your
> > > request for
> > > > > >> >> > functionality #1 and #2?  Yes or no, please.
> > > > > >> >> >
> > > > > >> >> > Thanks,
> > > > > >> >> > --Matt
> > > > > >> >> >
> > > > > >> >> >
> > > > > >> >> > On Sat, Mar 2, 2013 at 7:32 PM, Konstantin Shvachko
> > > > > >> >> > <sh...@gmail.com>
> > > > > >> >> > wrote:
> > > > > >> >> >>
> > > > > >> >> >> Hi Matt,
> > > > > >> >> >>
> > > > > >> >> >> On Sat, Mar 2, 2013 at 12:32 PM, Matt Foley <
> > > > > mfoley@hortonworks.com>
> > > > > >> >> >> wrote:
> > > > > >> >> >> > Konstantin,
> > > > > >> >> >> > I would like to explore what it would take to remove this
> > > > > >> >> >> > perceived
> > > > > >> >> >> > impediment --
> > > > > >> >> >>
> > > > > >> >> >> Glad you decided to explore. Thank you.
> > > > > >> >> >>
> > > > > >> >> >> > although I reserve the right to argue that this is not
> > > > > >> >> >> > pre-requisite to merging the cross-platform support patch.
> > > > > >> >> >>
> > > > > >> >> >> It's your right indeed. So as mine to question what the
> > > platform
> > > > > >> >> >> support means for you, which I believe remained unclear.
> > > > > >> >> >> I do not impede the change as you should have noticed. My
> > > > > >> >> >> requirement
> > > > > >> >> >> comes from my perception of the support, which means to me
> > > exactly
> > > > > >> >> >> two
> > > > > >> >> >> things:
> > > > > >> >> >> 1. The ability to recognise the code is broken for the
> > > platform
> > > > > >> >> >> 2. The ability to test new patches on the platform
> > > > > >> >> >> The latter is problematic, as many noticed in this thread, for
> > > > > those
> > > > > >> >> >> whose customary environment does not include Windows.
> > > > > >> >> >>
> > > > > >> >> >> > If we implemented full "test-patch" support for Windows on
> > > > > trunk,
> > > > > >> >> >> > would
> > > > > >> >> >> > that
> > > > > >> >> >> > fulfill both your items #1 and #2?  Please note that:
> > > > > >> >> >> > a) Pushing the "Patch Available" button in Jira shall cause
> > > a
> > > > > >> >> >> > pre-commit
> > > > > >> >> >> > build to start within, I believe, 20 minutes.
> > > > > >> >> >> > b) That build keeps logs for both java build and unit tests
> > > for
> > > > > >> >> >> > several
> > > > > >> >> >> > days, that are accessible to all viewers.
> > > > > >> >> >>
> > > > > >> >> >> In item #1 I mostly asking for the nightly build, which is
> > > simpler
> > > > > >> >> >> than "test-patch". The latter would be ideal from the platform
> > > > > >> >> >> support
> > > > > >> >> >> viewpoint, but it is for the community to decide if we want
> > > to add
> > > > > >> >> >> extra +3 hours to the build.
> > > > > >> >> >> Nightly build in my understanding is triggered by the timer
> > > rather
> > > > > >> >> >> than by Jira's "submit patch" button.  On Jenkins build
> > > > > >> >> >> configuration
> > > > > >> >> >> you can specify it under "Build periodically".
> > > > > >> >> >>
> > > > > >> >> >> > So, does this provide sufficient on-demand support that we
> > > don't
> > > > > >> >> >> > have
> > > > > >> >> >> > to
> > > > > >> >> >> > implement a whole new on-demand VM support structure of some
> > > > > sort
> > > > > >> >> >> > for
> > > > > >> >> >> > #2
> > > > > >> >> >> > (which would be an extraordinary and impractical demand)?
> > > > > >> >> >>
> > > > > >> >> >> I did not mention VMs. Item #2 means a build, which runs
> > > > > >> >> >> "test-patch"
> > > > > >> >> >> target with the file specified by a user (instead of a jira
> > > > > >> >> >> attachment).
> > > > > >> >> >> When user clicks "Build Now" link a box is displayed where the
> > > > > user
> > > > > >> >> >> can enter the file path containing the patch. This can be
> > > > > specified
> > > > > >> >> >> in
> > > > > >> >> >> the Build Configuration under "This build is parameterized" by
> > > > > >> >> >> choosing AddParameter / FileParameter. The build can run on
> > > the
> > > > > same
> > > > > >> >> >> Windows machine as the nightly build.
> > > > > >> >> >> Such build will let people test their patches for Windows on
> > > > > Jenkins
> > > > > >> >> >> if they don't posses a license for the right version of
> > > Windows.
> > > > > >> >> >> I hope this will not turn into extraordinary or impractical
> > > > > effort.
> > > > > >> >> >>
> > > > > >> >> >> Thanks,
> > > > > >> >> >> --Konst
> > > > > >> >> >>
> > > > > >> >> >> > Thanks,
> > > > > >> >> >> > --Matt
> > > > > >> >> >> >
> > > > > >> >> >> >
> > > > > >> >> >> > On Fri, Mar 1, 2013 at 12:18 PM, Konstantin Shvachko
> > > > > >> >> >> > <sh...@gmail.com>
> > > > > >> >> >> > wrote:
> > > > > >> >> >> >>
> > > > > >> >> >> >> -1
> > > > > >> >> >> >> We should have a CI infrastructure in place before we can
> > > > > commit
> > > > > >> >> >> >> to
> > > > > >> >> >> >> supporting Windows platform.
> > > > > >> >> >> >>
> > > > > >> >> >> >> Eric is right Win/Cygwin was supported since day one.
> > > > > >> >> >> >> I had a Windows box under my desk running nightly builds
> > > back
> > > > > in
> > > > > >> >> >> >> 2006-07.
> > > > > >> >> >> >> People were irritated but I was filing windows bugs until
> > > 0.22
> > > > > >> >> >> >> release.
> > > > > >> >> >> >> Times changing and I am glad to see wider support for Win
> > > > > >> >> >> >> platform.
> > > > > >> >> >> >>
> > > > > >> >> >> >> But in order to make it work you guys need to put the CI
> > > > > process
> > > > > >> >> >> >> in
> > > > > >> >> >> >> place
> > > > > >> >> >> >>
> > > > > >> >> >> >> 1. windows jenkins build: could be nightly or PreCommit.
> > > > > >> >> >> >> - Nightly would mean that changes can be committed to trunk
> > > > > based
> > > > > >> >> >> >> on
> > > > > >> >> >> >> linux PreCommit build. And people will file bugs if the
> > > change
> > > > > >> >> >> >> broke
> > > > > >> >> >> >> Windows nightly build.
> > > > > >> >> >> >> - PreCommit-win build will mean automatic reporting failed
> > > > > tests
> > > > > >> >> >> >> to
> > > > > >> >> >> >> respective jira blocking commits the same way as it is now
> > > with
> > > > > >> >> >> >> linux
> > > > > >> >> >> >> PreCommit builds.
> > > > > >> >> >> >> We should discuss which way is more efficient for
> > > developers.
> > > > > >> >> >> >>
> > > > > >> >> >> >> 2. On-demand-windows Jenkins build.
> > > > > >> >> >> >> I see it as a build to which I can attach my patch and the
> > > > > build
> > > > > >> >> >> >> will
> > > > > >> >> >> >> run my changes on a dedicated windows box.
> > > > > >> >> >> >> That way people can test their changes without having
> > > personal
> > > > > >> >> >> >> windows
> > > > > >> >> >> >> nodes.
> > > > > >> >> >> >>
> > > > > >> >> >> >> I think this is the minimal set of requirement for us to be
> > > > > able
> > > > > >> >> >> >> to
> > > > > >> >> >> >> commit to the new platform.
> > > > > >> >> >> >> Right now I see only one windows related build
> > > > > >> >> >> >> https://builds.apache.org/view/Hadoop/job/Hadoop-1-win/
> > > > > >> >> >> >> Which was failing since Sept 8, 2012 and did not run in the
> > > > > last
> > > > > >> >> >> >> month.
> > > > > >> >> >> >>
> > > > > >> >> >> >> Thanks,
> > > > > >> >> >> >> --Konst
> > > > > >> >> >> >>
> > > > > >> >> >> >> On Thu, Feb 28, 2013 at 8:47 PM, Eric Baldeschwieler
> > > > > >> >> >> >> <er...@hortonworks.com> wrote:
> > > > > >> >> >> >> > +1 (non-binding)
> > > > > >> >> >> >> >
> > > > > >> >> >> >> > A few of observations:
> > > > > >> >> >> >> >
> > > > > >> >> >> >> > - Windows has actually been a supported platform for
> > > Hadoop
> > > > > >> >> >> >> > since
> > > > > >> >> >> >> > 0.1
> > > > > >> >> >> >> > .
> > > > > >> >> >> >> > Doug championed supporting windows then and we've
> > > continued
> > > > > to
> > > > > >> >> >> >> > do
> > > > > >> >> >> >> > it
> > > > > >> >> >> >> > with
> > > > > >> >> >> >> > varying vigor over time.  To my knowledge we've never
> > > made a
> > > > > >> >> >> >> > decision
> > > > > >> >> >> >> > to
> > > > > >> >> >> >> > drop windows support.  The change here is improving our
> > > > > support
> > > > > >> >> >> >> > and
> > > > > >> >> >> >> > dropping
> > > > > >> >> >> >> > the requirement of cigwin.  We had Nutch windows users
> > > on the
> > > > > >> >> >> >> > list
> > > > > >> >> >> >> > in
> > > > > >> >> >> >> > 2006
> > > > > >> >> >> >> > and we've been supporting windows FS requirements since
> > > > > >> >> >> >> > inception.
> > > > > >> >> >> >> >
> > > > > >> >> >> >> > - A little pragmatism will go a long way.  As a community
> > > > > we've
> > > > > >> >> >> >> > got
> > > > > >> >> >> >> > to
> > > > > >> >> >> >> > stay committed to keeping hadoop simple (so it does work
> > > on
> > > > > >> >> >> >> > many
> > > > > >> >> >> >> > platforms)
> > > > > >> >> >> >> > and extending it to take advantage of key emerging
> > > > > OS/hardware
> > > > > >> >> >> >> > features,
> > > > > >> >> >> >> > such as containers, new FSs, virtualization, flash ...
> > >  We
> > > > > >> >> >> >> > should
> > > > > >> >> >> >> > all
> > > > > >> >> >> >> > plan
> > > > > >> >> >> >> > to let new features & optimizations emerge that don't
> > > work
> > > > > >> >> >> >> > everywhere, if
> > > > > >> >> >> >> > they are compelling and central to hadoop's mission of
> > > being
> > > > > >> >> >> >> > THE
> > > > > >> >> >> >> > best
> > > > > >> >> >> >> > fabric
> > > > > >> >> >> >> > for storing and processing big data.
> > > > > >> >> >> >> >
> > > > > >> >> >> >> > - A UI project like KDE has to deal with the MANY
> > > differences
> > > > > >> >> >> >> > between
> > > > > >> >> >> >> > windows and linux UI APIs.  Hadoop faces no such complex
> > > > > >> >> >> >> > challenge
> > > > > >> >> >> >> > and hence
> > > > > >> >> >> >> > can be maintained from a single codeline IMO.  It is
> > > mostly
> > > > > >> >> >> >> > abstracted from
> > > > > >> >> >> >> > the OS APIs via Java and our design choices.  Where it
> > > is not
> > > > > >> >> >> >> > we
> > > > > >> >> >> >> > can
> > > > > >> >> >> >> > continue to add plugable abstractions.
> > > > > >> >> >> >> >
> > > > > >> >> >> >
> > > > > >> >> >> >
> > > > > >> >> >
> > > > > >> >> >
> > > > > >> >
> > > > > >> >
> > > > > >
> > > > > >
> > > > >
> > >
> > 
> > 
> > 
> > -- 
> > http://hortonworks.com/download/

Re: [Vote] Merge branch-trunk-win to trunk

[Konstantin Boudnik <co...@apache.org>]

Yes, you are right of course - the mis-merged commit is the cause. Thanks for
pointing this out!

I think it would be beneficial if we had branch-2 on going build in the
Jenkins. I will go ahead and create one tonight.

Cos

On Mon, Mar 25, 2013 at 05:09PM, Suresh Srinivas wrote:
> Adding other mailing lists I missed earlier.
> 
> Cos,
> 
> There is progress being made on that ticket. Also it has nothing to do with
> that.
> 
> Please follow the discussion here and why this happened due to an invalid
> commit that was reverted -
> https://issues.apache.org/jira/browse/HDFS-4615?focusedCommentId=13612650&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13612650
> 
> Regards,
> Suresh
> 
> 
> On Mon, Mar 25, 2013 at 1:17 PM, Konstantin Boudnik <co...@apache.org> wrote:
> 
> > It doesn't look like any progress has been done on the ticket below in the
> > last 3 weeks. And now branch-2 can't be compiled because of
> >
> >
> > hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSShell.java:[895,15]
> > WINDOWS is not public in org.apache.hadoop.fs.Path; cannot be accessed from
> > outside package
> >
> > That's exactly why I was -1'ing this...
> >   Cos
> >
> > On Mon, Mar 04, 2013 at 05:41PM, Matt Foley wrote:
> > > Thanks, gentlemen.  I've opened and taken responsibility for
> > > https://issues.apache.org/jira/browse/HADOOP-9359.  Giri Kesavan has
> > agreed
> > > to help with the parts that require Jenkins admin access.
> > >
> > > Thanks,
> > > --Matt
> > >
> > >
> > >
> > > On Mon, Mar 4, 2013 at 5:00 PM, Konstantin Shvachko <
> > shv.hadoop@gmail.com>wrote:
> > >
> > > > +1 on the merge.
> > > >
> > > > I am glad we agreed.
> > > > Having Jira to track the CI effort is a good idea.
> > > >
> > > > Thanks,
> > > > --Konstantin
> > > >
> > > > On Mon, Mar 4, 2013 at 3:29 PM, Matt Foley <mf...@hortonworks.com>
> > wrote:
> > > > > Thanks.  I agree Windows -1's in test-patch should not block commits.
> > > > >
> > > > > --Matt
> > > > >
> > > > >
> > > > >
> > > > > On Mon, Mar 4, 2013 at 2:30 PM, Konstantin Shvachko <
> > > > shv.hadoop@gmail.com>
> > > > > wrote:
> > > > >>
> > > > >> On Mon, Mar 4, 2013 at 12:22 PM, Matt Foley <mfoley@hortonworks.com
> > >
> > > > >> wrote:
> > > > >> > Konstantine, you have voted -1, and stated some requirements
> > before
> > > > >> > you'll
> > > > >> > withdraw that -1.  As I plan to do work to fulfill those
> > > > requirements, I
> > > > >> > want to make sure that what I'm proposing will, in fact, satisfy
> > you.
> > > > >> > That's why I'm asking, if we implement full "test-patch"
> > integration
> > > > for
> > > > >> > Windows, does it seem to you that that would provide adequate
> > support?
> > > > >>
> > > > >> Yes.
> > > > >>
> > > > >> > I have learned not to presume that my interpretation is correct.
> >  My
> > > > >> > interpretation of item #1 is that test-patch provides pre-commit
> > > > build,
> > > > >> > so
> > > > >> > it would satisfy item #1.  But rather than assuming that I am
> > > > >> > interpreting
> > > > >> > it correctly, I simply want your agreement that it would, or if
> > not,
> > > > >> > clarification why it won't.
> > > > >>
> > > > >> I agree it will satisfy my item #1.
> > > > >> I did not agree in my previous email, but I changed my mind based on
> > > > >> the latest discussion. I have to explain why now.
> > > > >> I was proposing nightly build because I did not want pre-commit
> > build
> > > > >> for Windows block commits to Linux. But if people are fine just
> > ignoring
> > > > >> -1s for the Windows part of the build it should be good.
> > > > >>
> > > > >> > Regarding item #2, it is also my interpretation that test-patch
> > > > provides
> > > > >> > an
> > > > >> > on-demand (perhaps 20-minutes deferred) Jenkins build and unit
> > test,
> > > > >> > with
> > > > >> > logs available to the developer, so it would satisfy item #2.  But
> > > > >> > rather
> > > > >> > than assuming that I am interpreting it correctly, I simply want
> > your
> > > > >> > agreement that it would, or if not, clarification why it won't.
> > > > >>
> > > > >> It will satisfy my item #2 in the following way:
> > > > >> I can duplicate your pre-commit build for Windows and add an input
> > > > >> parameter, which would let people run the build on their patches
> > > > >> chosen from local machine rather than attaching them to Jiras.
> > > > >>
> > > > >> Thanks,
> > > > >> --Konstantin
> > > > >>
> > > > >> > In agile terms, you are the Owner of these requirements.  Please
> > give
> > > > me
> > > > >> > owner feedback as to whether my proposed work sounds like it will
> > > > >> > satisfy
> > > > >> > the requirements.
> > > > >> >
> > > > >> > Thank you,
> > > > >> > --Matt
> > > > >> >
> > > > >> >
> > > > >> > On Sun, Mar 3, 2013 at 12:16 PM, Konstantin Shvachko
> > > > >> > <sh...@gmail.com>
> > > > >> > wrote:
> > > > >> >>
> > > > >> >> Didn't I explain in details what I am asking for?
> > > > >> >>
> > > > >> >> Thanks,
> > > > >> >> --Konst
> > > > >> >>
> > > > >> >> On Sun, Mar 3, 2013 at 11:08 AM, Matt Foley <
> > mfoley@hortonworks.com>
> > > > >> >> wrote:
> > > > >> >> > Hi Konstantin,
> > > > >> >> > I'd like to point out two things:
> > > > >> >> > First, I already committed in this thread (email of Thu, Feb
> > 28,
> > > > 2013
> > > > >> >> > at
> > > > >> >> > 6:01 PM) to providing CI for Windows builds.  So please stop
> > acting
> > > > >> >> > like
> > > > >> >> > I'm
> > > > >> >> > resisting this idea or something.
> > > > >> >> > Second, you didn't answer my question, you just kvetched about
> > the
> > > > >> >> > phrasing.
> > > > >> >> > So I ask again:
> > > > >> >> >
> > > > >> >> > Will providing full "test-patch" integration (pre-commit build
> > and
> > > > >> >> > unit
> > > > >> >> > test
> > > > >> >> > triggered by Jira "Patch Available" state) satisfy your
> > request for
> > > > >> >> > functionality #1 and #2?  Yes or no, please.
> > > > >> >> >
> > > > >> >> > Thanks,
> > > > >> >> > --Matt
> > > > >> >> >
> > > > >> >> >
> > > > >> >> > On Sat, Mar 2, 2013 at 7:32 PM, Konstantin Shvachko
> > > > >> >> > <sh...@gmail.com>
> > > > >> >> > wrote:
> > > > >> >> >>
> > > > >> >> >> Hi Matt,
> > > > >> >> >>
> > > > >> >> >> On Sat, Mar 2, 2013 at 12:32 PM, Matt Foley <
> > > > mfoley@hortonworks.com>
> > > > >> >> >> wrote:
> > > > >> >> >> > Konstantin,
> > > > >> >> >> > I would like to explore what it would take to remove this
> > > > >> >> >> > perceived
> > > > >> >> >> > impediment --
> > > > >> >> >>
> > > > >> >> >> Glad you decided to explore. Thank you.
> > > > >> >> >>
> > > > >> >> >> > although I reserve the right to argue that this is not
> > > > >> >> >> > pre-requisite to merging the cross-platform support patch.
> > > > >> >> >>
> > > > >> >> >> It's your right indeed. So as mine to question what the
> > platform
> > > > >> >> >> support means for you, which I believe remained unclear.
> > > > >> >> >> I do not impede the change as you should have noticed. My
> > > > >> >> >> requirement
> > > > >> >> >> comes from my perception of the support, which means to me
> > exactly
> > > > >> >> >> two
> > > > >> >> >> things:
> > > > >> >> >> 1. The ability to recognise the code is broken for the
> > platform
> > > > >> >> >> 2. The ability to test new patches on the platform
> > > > >> >> >> The latter is problematic, as many noticed in this thread, for
> > > > those
> > > > >> >> >> whose customary environment does not include Windows.
> > > > >> >> >>
> > > > >> >> >> > If we implemented full "test-patch" support for Windows on
> > > > trunk,
> > > > >> >> >> > would
> > > > >> >> >> > that
> > > > >> >> >> > fulfill both your items #1 and #2?  Please note that:
> > > > >> >> >> > a) Pushing the "Patch Available" button in Jira shall cause
> > a
> > > > >> >> >> > pre-commit
> > > > >> >> >> > build to start within, I believe, 20 minutes.
> > > > >> >> >> > b) That build keeps logs for both java build and unit tests
> > for
> > > > >> >> >> > several
> > > > >> >> >> > days, that are accessible to all viewers.
> > > > >> >> >>
> > > > >> >> >> In item #1 I mostly asking for the nightly build, which is
> > simpler
> > > > >> >> >> than "test-patch". The latter would be ideal from the platform
> > > > >> >> >> support
> > > > >> >> >> viewpoint, but it is for the community to decide if we want
> > to add
> > > > >> >> >> extra +3 hours to the build.
> > > > >> >> >> Nightly build in my understanding is triggered by the timer
> > rather
> > > > >> >> >> than by Jira's "submit patch" button.  On Jenkins build
> > > > >> >> >> configuration
> > > > >> >> >> you can specify it under "Build periodically".
> > > > >> >> >>
> > > > >> >> >> > So, does this provide sufficient on-demand support that we
> > don't
> > > > >> >> >> > have
> > > > >> >> >> > to
> > > > >> >> >> > implement a whole new on-demand VM support structure of some
> > > > sort
> > > > >> >> >> > for
> > > > >> >> >> > #2
> > > > >> >> >> > (which would be an extraordinary and impractical demand)?
> > > > >> >> >>
> > > > >> >> >> I did not mention VMs. Item #2 means a build, which runs
> > > > >> >> >> "test-patch"
> > > > >> >> >> target with the file specified by a user (instead of a jira
> > > > >> >> >> attachment).
> > > > >> >> >> When user clicks "Build Now" link a box is displayed where the
> > > > user
> > > > >> >> >> can enter the file path containing the patch. This can be
> > > > specified
> > > > >> >> >> in
> > > > >> >> >> the Build Configuration under "This build is parameterized" by
> > > > >> >> >> choosing AddParameter / FileParameter. The build can run on
> > the
> > > > same
> > > > >> >> >> Windows machine as the nightly build.
> > > > >> >> >> Such build will let people test their patches for Windows on
> > > > Jenkins
> > > > >> >> >> if they don't posses a license for the right version of
> > Windows.
> > > > >> >> >> I hope this will not turn into extraordinary or impractical
> > > > effort.
> > > > >> >> >>
> > > > >> >> >> Thanks,
> > > > >> >> >> --Konst
> > > > >> >> >>
> > > > >> >> >> > Thanks,
> > > > >> >> >> > --Matt
> > > > >> >> >> >
> > > > >> >> >> >
> > > > >> >> >> > On Fri, Mar 1, 2013 at 12:18 PM, Konstantin Shvachko
> > > > >> >> >> > <sh...@gmail.com>
> > > > >> >> >> > wrote:
> > > > >> >> >> >>
> > > > >> >> >> >> -1
> > > > >> >> >> >> We should have a CI infrastructure in place before we can
> > > > commit
> > > > >> >> >> >> to
> > > > >> >> >> >> supporting Windows platform.
> > > > >> >> >> >>
> > > > >> >> >> >> Eric is right Win/Cygwin was supported since day one.
> > > > >> >> >> >> I had a Windows box under my desk running nightly builds
> > back
> > > > in
> > > > >> >> >> >> 2006-07.
> > > > >> >> >> >> People were irritated but I was filing windows bugs until
> > 0.22
> > > > >> >> >> >> release.
> > > > >> >> >> >> Times changing and I am glad to see wider support for Win
> > > > >> >> >> >> platform.
> > > > >> >> >> >>
> > > > >> >> >> >> But in order to make it work you guys need to put the CI
> > > > process
> > > > >> >> >> >> in
> > > > >> >> >> >> place
> > > > >> >> >> >>
> > > > >> >> >> >> 1. windows jenkins build: could be nightly or PreCommit.
> > > > >> >> >> >> - Nightly would mean that changes can be committed to trunk
> > > > based
> > > > >> >> >> >> on
> > > > >> >> >> >> linux PreCommit build. And people will file bugs if the
> > change
> > > > >> >> >> >> broke
> > > > >> >> >> >> Windows nightly build.
> > > > >> >> >> >> - PreCommit-win build will mean automatic reporting failed
> > > > tests
> > > > >> >> >> >> to
> > > > >> >> >> >> respective jira blocking commits the same way as it is now
> > with
> > > > >> >> >> >> linux
> > > > >> >> >> >> PreCommit builds.
> > > > >> >> >> >> We should discuss which way is more efficient for
> > developers.
> > > > >> >> >> >>
> > > > >> >> >> >> 2. On-demand-windows Jenkins build.
> > > > >> >> >> >> I see it as a build to which I can attach my patch and the
> > > > build
> > > > >> >> >> >> will
> > > > >> >> >> >> run my changes on a dedicated windows box.
> > > > >> >> >> >> That way people can test their changes without having
> > personal
> > > > >> >> >> >> windows
> > > > >> >> >> >> nodes.
> > > > >> >> >> >>
> > > > >> >> >> >> I think this is the minimal set of requirement for us to be
> > > > able
> > > > >> >> >> >> to
> > > > >> >> >> >> commit to the new platform.
> > > > >> >> >> >> Right now I see only one windows related build
> > > > >> >> >> >> https://builds.apache.org/view/Hadoop/job/Hadoop-1-win/
> > > > >> >> >> >> Which was failing since Sept 8, 2012 and did not run in the
> > > > last
> > > > >> >> >> >> month.
> > > > >> >> >> >>
> > > > >> >> >> >> Thanks,
> > > > >> >> >> >> --Konst
> > > > >> >> >> >>
> > > > >> >> >> >> On Thu, Feb 28, 2013 at 8:47 PM, Eric Baldeschwieler
> > > > >> >> >> >> <er...@hortonworks.com> wrote:
> > > > >> >> >> >> > +1 (non-binding)
> > > > >> >> >> >> >
> > > > >> >> >> >> > A few of observations:
> > > > >> >> >> >> >
> > > > >> >> >> >> > - Windows has actually been a supported platform for
> > Hadoop
> > > > >> >> >> >> > since
> > > > >> >> >> >> > 0.1
> > > > >> >> >> >> > .
> > > > >> >> >> >> > Doug championed supporting windows then and we've
> > continued
> > > > to
> > > > >> >> >> >> > do
> > > > >> >> >> >> > it
> > > > >> >> >> >> > with
> > > > >> >> >> >> > varying vigor over time.  To my knowledge we've never
> > made a
> > > > >> >> >> >> > decision
> > > > >> >> >> >> > to
> > > > >> >> >> >> > drop windows support.  The change here is improving our
> > > > support
> > > > >> >> >> >> > and
> > > > >> >> >> >> > dropping
> > > > >> >> >> >> > the requirement of cigwin.  We had Nutch windows users
> > on the
> > > > >> >> >> >> > list
> > > > >> >> >> >> > in
> > > > >> >> >> >> > 2006
> > > > >> >> >> >> > and we've been supporting windows FS requirements since
> > > > >> >> >> >> > inception.
> > > > >> >> >> >> >
> > > > >> >> >> >> > - A little pragmatism will go a long way.  As a community
> > > > we've
> > > > >> >> >> >> > got
> > > > >> >> >> >> > to
> > > > >> >> >> >> > stay committed to keeping hadoop simple (so it does work
> > on
> > > > >> >> >> >> > many
> > > > >> >> >> >> > platforms)
> > > > >> >> >> >> > and extending it to take advantage of key emerging
> > > > OS/hardware
> > > > >> >> >> >> > features,
> > > > >> >> >> >> > such as containers, new FSs, virtualization, flash ...
> >  We
> > > > >> >> >> >> > should
> > > > >> >> >> >> > all
> > > > >> >> >> >> > plan
> > > > >> >> >> >> > to let new features & optimizations emerge that don't
> > work
> > > > >> >> >> >> > everywhere, if
> > > > >> >> >> >> > they are compelling and central to hadoop's mission of
> > being
> > > > >> >> >> >> > THE
> > > > >> >> >> >> > best
> > > > >> >> >> >> > fabric
> > > > >> >> >> >> > for storing and processing big data.
> > > > >> >> >> >> >
> > > > >> >> >> >> > - A UI project like KDE has to deal with the MANY
> > differences
> > > > >> >> >> >> > between
> > > > >> >> >> >> > windows and linux UI APIs.  Hadoop faces no such complex
> > > > >> >> >> >> > challenge
> > > > >> >> >> >> > and hence
> > > > >> >> >> >> > can be maintained from a single codeline IMO.  It is
> > mostly
> > > > >> >> >> >> > abstracted from
> > > > >> >> >> >> > the OS APIs via Java and our design choices.  Where it
> > is not
> > > > >> >> >> >> > we
> > > > >> >> >> >> > can
> > > > >> >> >> >> > continue to add plugable abstractions.
> > > > >> >> >> >> >
> > > > >> >> >> >
> > > > >> >> >> >
> > > > >> >> >
> > > > >> >> >
> > > > >> >
> > > > >> >
> > > > >
> > > > >
> > > >
> >
> 
> 
> 
> -- 
> http://hortonworks.com/download/

Re: [Vote] Merge branch-trunk-win to trunk

[Tsz Wo Sze <sz...@yahoo.com>]

Hi Konstantin S,

I accidentally merged some windows bug fixes to branch-2.  However, branch-2 did not compile -- it showed that the windows changes were missing in branch-2.  The ones causing compilation problems were already reverted (Thanks Sid and Suresh.)  Sorry for the inconvenience.  


Tsz-Wo




________________________________
 From: Konstantin Shvachko <sh...@gmail.com>
To: mapreduce-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org; common-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org 
Sent: Tuesday, March 26, 2013 5:25 AM
Subject: Re: [Vote] Merge branch-trunk-win to trunk
 
Andrew, this used to be on all -dev lists. Let's keep it that way.

To the point.
Does this mean that people are silently porting windows changes to branch-2?
New features on a branch should be voted first, no?

Thanks,
--Konstantin


On Mon, Mar 25, 2013 at 1:36 PM, Andrew Purtell <ap...@apache.org> wrote:
> Noticed this too. Simply a 'public' modifier is missing, but it's unclear
> how this could not have been caught prior to check-in.
>
>
> On Mon, Mar 25, 2013 at 9:17 PM, Konstantin Boudnik <co...@apache.org> wrote:
>
>> It doesn't look like any progress has been done on the ticket below in the
>> last 3 weeks. And now branch-2 can't be compiled because of
>>
>>
>> hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSShell.java:[895,15]
>> WINDOWS is not public in org.apache.hadoop.fs.Path; cannot be accessed from
>> outside package
>>
>> That's exactly why I was -1'ing this...
>>   Cos
>>
>> On Mon, Mar 04, 2013 at 05:41PM, Matt Foley wrote:
>> > Thanks, gentlemen.  I've opened and taken responsibility for
>> > https://issues.apache.org/jira/browse/HADOOP-9359.  Giri Kesavan has
>> agreed
>> > to help with the parts that require Jenkins admin access.
>> >
>> > Thanks,
>> > --Matt
>> >
>> >
>> >
>> > On Mon, Mar 4, 2013 at 5:00 PM, Konstantin Shvachko <
>> shv.hadoop@gmail.com>wrote:
>> >
>> > > +1 on the merge.
>> > >
>> > > I am glad we agreed.
>> > > Having Jira to track the CI effort is a good idea.
>> > >
>> > > Thanks,
>> > > --Konstantin
>> > >
>> > > On Mon, Mar 4, 2013 at 3:29 PM, Matt Foley <mf...@hortonworks.com>
>> wrote:
>> > > > Thanks.  I agree Windows -1's in test-patch should not block commits.
>> > > >
>> > > > --Matt
>> > > >
>> > > >
>> > > >
>> > > > On Mon, Mar 4, 2013 at 2:30 PM, Konstantin Shvachko <
>> > > shv.hadoop@gmail.com>
>> > > > wrote:
>> > > >>
>> > > >> On Mon, Mar 4, 2013 at 12:22 PM, Matt Foley <mfoley@hortonworks.com
>> >
>> > > >> wrote:
>> > > >> > Konstantine, you have voted -1, and stated some requirements
>> before
>> > > >> > you'll
>> > > >> > withdraw that -1.  As I plan to do work to fulfill those
>> > > requirements, I
>> > > >> > want to make sure that what I'm proposing will, in fact, satisfy
>> you.
>> > > >> > That's why I'm asking, if we implement full "test-patch"
>> integration
>> > > for
>> > > >> > Windows, does it seem to you that that would provide adequate
>> support?
>> > > >>
>> > > >> Yes.
>> > > >>
>> > > >> > I have learned not to presume that my interpretation is correct.
>>  My
>> > > >> > interpretation of item #1 is that test-patch provides pre-commit
>> > > build,
>> > > >> > so
>> > > >> > it would satisfy item #1.  But rather than assuming that I am
>> > > >> > interpreting
>> > > >> > it correctly, I simply want your agreement that it would, or if
>> not,
>> > > >> > clarification why it won't.
>> > > >>
>> > > >> I agree it will satisfy my item #1.
>> > > >> I did not agree in my previous email, but I changed my mind based on
>> > > >> the latest discussion. I have to explain why now.
>> > > >> I was proposing nightly build because I did not want pre-commit
>> build
>> > > >> for Windows block commits to Linux. But if people are fine just
>> ignoring
>> > > >> -1s for the Windows part of the build it should be good.
>> > > >>
>> > > >> > Regarding item #2, it is also my interpretation that test-patch
>> > > provides
>> > > >> > an
>> > > >> > on-demand (perhaps 20-minutes deferred) Jenkins build and unit
>> test,
>> > > >> > with
>> > > >> > logs available to the developer, so it would satisfy item #2.  But
>> > > >> > rather
>> > > >> > than assuming that I am interpreting it correctly, I simply want
>> your
>> > > >> > agreement that it would, or if not, clarification why it won't.
>> > > >>
>> > > >> It will satisfy my item #2 in the following way:
>> > > >> I can duplicate your pre-commit build for Windows and add an input
>> > > >> parameter, which would let people run the build on their patches
>> > > >> chosen from local machine rather than attaching them to Jiras.
>> > > >>
>> > > >> Thanks,
>> > > >> --Konstantin
>> > > >>
>> > > >> > In agile terms, you are the Owner of these requirements.  Please
>> give
>> > > me
>> > > >> > owner feedback as to whether my proposed work sounds like it will
>> > > >> > satisfy
>> > > >> > the requirements.
>> > > >> >
>> > > >> > Thank you,
>> > > >> > --Matt
>> > > >> >
>> > > >> >
>> > > >> > On Sun, Mar 3, 2013 at 12:16 PM, Konstantin Shvachko
>> > > >> > <sh...@gmail.com>
>> > > >> > wrote:
>> > > >> >>
>> > > >> >> Didn't I explain in details what I am asking for?
>> > > >> >>
>> > > >> >> Thanks,
>> > > >> >> --Konst
>> > > >> >>
>> > > >> >> On Sun, Mar 3, 2013 at 11:08 AM, Matt Foley <
>> mfoley@hortonworks.com>
>> > > >> >> wrote:
>> > > >> >> > Hi Konstantin,
>> > > >> >> > I'd like to point out two things:
>> > > >> >> > First, I already committed in this thread (email of Thu, Feb
>> 28,
>> > > 2013
>> > > >> >> > at
>> > > >> >> > 6:01 PM) to providing CI for Windows builds.  So please stop
>> acting
>> > > >> >> > like
>> > > >> >> > I'm
>> > > >> >> > resisting this idea or something.
>> > > >> >> > Second, you didn't answer my question, you just kvetched about
>> the
>> > > >> >> > phrasing.
>> > > >> >> > So I ask again:
>> > > >> >> >
>> > > >> >> > Will providing full "test-patch" integration (pre-commit build
>> and
>> > > >> >> > unit
>> > > >> >> > test
>> > > >> >> > triggered by Jira "Patch Available" state) satisfy your
>> request for
>> > > >> >> > functionality #1 and #2?  Yes or no, please.
>> > > >> >> >
>> > > >> >> > Thanks,
>> > > >> >> > --Matt
>> > > >> >> >
>> > > >> >> >
>> > > >> >> > On Sat, Mar 2, 2013 at 7:32 PM, Konstantin Shvachko
>> > > >> >> > <sh...@gmail.com>
>> > > >> >> > wrote:
>> > > >> >> >>
>> > > >> >> >> Hi Matt,
>> > > >> >> >>
>> > > >> >> >> On Sat, Mar 2, 2013 at 12:32 PM, Matt Foley <
>> > > mfoley@hortonworks.com>
>> > > >> >> >> wrote:
>> > > >> >> >> > Konstantin,
>> > > >> >> >> > I would like to explore what it would take to remove this
>> > > >> >> >> > perceived
>> > > >> >> >> > impediment --
>> > > >> >> >>
>> > > >> >> >> Glad you decided to explore. Thank you.
>> > > >> >> >>
>> > > >> >> >> > although I reserve the right to argue that this is not
>> > > >> >> >> > pre-requisite to merging the cross-platform support patch.
>> > > >> >> >>
>> > > >> >> >> It's your right indeed. So as mine to question what the
>> platform
>> > > >> >> >> support means for you, which I believe remained unclear.
>> > > >> >> >> I do not impede the change as you should have noticed. My
>> > > >> >> >> requirement
>> > > >> >> >> comes from my perception of the support, which means to me
>> exactly
>> > > >> >> >> two
>> > > >> >> >> things:
>> > > >> >> >> 1. The ability to recognise the code is broken for the
>> platform
>> > > >> >> >> 2. The ability to test new patches on the platform
>> > > >> >> >> The latter is problematic, as many noticed in this thread, for
>> > > those
>> > > >> >> >> whose customary environment does not include Windows.
>> > > >> >> >>
>> > > >> >> >> > If we implemented full "test-patch" support for Windows on
>> > > trunk,
>> > > >> >> >> > would
>> > > >> >> >> > that
>> > > >> >> >> > fulfill both your items #1 and #2?  Please note that:
>> > > >> >> >> > a) Pushing the "Patch Available" button in Jira shall cause
>> a
>> > > >> >> >> > pre-commit
>> > > >> >> >> > build to start within, I believe, 20 minutes.
>> > > >> >> >> > b) That build keeps logs for both java build and unit tests
>> for
>> > > >> >> >> > several
>> > > >> >> >> > days, that are accessible to all viewers.
>> > > >> >> >>
>> > > >> >> >> In item #1 I mostly asking for the nightly build, which is
>> simpler
>> > > >> >> >> than "test-patch". The latter would be ideal from the platform
>> > > >> >> >> support
>> > > >> >> >> viewpoint, but it is for the community to decide if we want
>> to add
>> > > >> >> >> extra +3 hours to the build.
>> > > >> >> >> Nightly build in my understanding is triggered by the timer
>> rather
>> > > >> >> >> than by Jira's "submit patch" button.  On Jenkins build
>> > > >> >> >> configuration
>> > > >> >> >> you can specify it under "Build periodically".
>> > > >> >> >>
>> > > >> >> >> > So, does this provide sufficient on-demand support that we
>> don't
>> > > >> >> >> > have
>> > > >> >> >> > to
>> > > >> >> >> > implement a whole new on-demand VM support structure of some
>> > > sort
>> > > >> >> >> > for
>> > > >> >> >> > #2
>> > > >> >> >> > (which would be an extraordinary and impractical demand)?
>> > > >> >> >>
>> > > >> >> >> I did not mention VMs. Item #2 means a build, which runs
>> > > >> >> >> "test-patch"
>> > > >> >> >> target with the file specified by a user (instead of a jira
>> > > >> >> >> attachment).
>> > > >> >> >> When user clicks "Build Now" link a box is displayed where the
>> > > user
>> > > >> >> >> can enter the file path containing the patch. This can be
>> > > specified
>> > > >> >> >> in
>> > > >> >> >> the Build Configuration under "This build is parameterized" by
>> > > >> >> >> choosing AddParameter / FileParameter. The build can run on
>> the
>> > > same
>> > > >> >> >> Windows machine as the nightly build.
>> > > >> >> >> Such build will let people test their patches for Windows on
>> > > Jenkins
>> > > >> >> >> if they don't posses a license for the right version of
>> Windows.
>> > > >> >> >> I hope this will not turn into extraordinary or impractical
>> > > effort.
>> > > >> >> >>
>> > > >> >> >> Thanks,
>> > > >> >> >> --Konst
>> > > >> >> >>
>> > > >> >> >> > Thanks,
>> > > >> >> >> > --Matt
>> > > >> >> >> >
>> > > >> >> >> >
>> > > >> >> >> > On Fri, Mar 1, 2013 at 12:18 PM, Konstantin Shvachko
>> > > >> >> >> > <sh...@gmail.com>
>> > > >> >> >> > wrote:
>> > > >> >> >> >>
>> > > >> >> >> >> -1
>> > > >> >> >> >> We should have a CI infrastructure in place before we can
>> > > commit
>> > > >> >> >> >> to
>> > > >> >> >> >> supporting Windows platform.
>> > > >> >> >> >>
>> > > >> >> >> >> Eric is right Win/Cygwin was supported since day one.
>> > > >> >> >> >> I had a Windows box under my desk running nightly builds
>> back
>> > > in
>> > > >> >> >> >> 2006-07.
>> > > >> >> >> >> People were irritated but I was filing windows bugs until
>> 0.22
>> > > >> >> >> >> release.
>> > > >> >> >> >> Times changing and I am glad to see wider support for Win
>> > > >> >> >> >> platform.
>> > > >> >> >> >>
>> > > >> >> >> >> But in order to make it work you guys need to put the CI
>> > > process
>> > > >> >> >> >> in
>> > > >> >> >> >> place
>> > > >> >> >> >>
>> > > >> >> >> >> 1. windows jenkins build: could be nightly or PreCommit.
>> > > >> >> >> >> - Nightly would mean that changes can be committed to trunk
>> > > based
>> > > >> >> >> >> on
>> > > >> >> >> >> linux PreCommit build. And people will file bugs if the
>> change
>> > > >> >> >> >> broke
>> > > >> >> >> >> Windows nightly build.
>> > > >> >> >> >> - PreCommit-win build will mean automatic reporting failed
>> > > tests
>> > > >> >> >> >> to
>> > > >> >> >> >> respective jira blocking commits the same way as it is now
>> with
>> > > >> >> >> >> linux
>> > > >> >> >> >> PreCommit builds.
>> > > >> >> >> >> We should discuss which way is more efficient for
>> developers.
>> > > >> >> >> >>
>> > > >> >> >> >> 2. On-demand-windows Jenkins build.
>> > > >> >> >> >> I see it as a build to which I can attach my patch and the
>> > > build
>> > > >> >> >> >> will
>> > > >> >> >> >> run my changes on a dedicated windows box.
>> > > >> >> >> >> That way people can test their changes without having
>> personal
>> > > >> >> >> >> windows
>> > > >> >> >> >> nodes.
>> > > >> >> >> >>
>> > > >> >> >> >> I think this is the minimal set of requirement for us to be
>> > > able
>> > > >> >> >> >> to
>> > > >> >> >> >> commit to the new platform.
>> > > >> >> >> >> Right now I see only one windows related build
>> > > >> >> >> >> https://builds.apache.org/view/Hadoop/job/Hadoop-1-win/
>> > > >> >> >> >> Which was failing since Sept 8, 2012 and did not run in the
>> > > last
>> > > >> >> >> >> month.
>> > > >> >> >> >>
>> > > >> >> >> >> Thanks,
>> > > >> >> >> >> --Konst
>> > > >> >> >> >>
>> > > >> >> >> >> On Thu, Feb 28, 2013 at 8:47 PM, Eric Baldeschwieler
>> > > >> >> >> >> <er...@hortonworks.com> wrote:
>> > > >> >> >> >> > +1 (non-binding)
>> > > >> >> >> >> >
>> > > >> >> >> >> > A few of observations:
>> > > >> >> >> >> >
>> > > >> >> >> >> > - Windows has actually been a supported platform for
>> Hadoop
>> > > >> >> >> >> > since
>> > > >> >> >> >> > 0.1
>> > > >> >> >> >> > .
>> > > >> >> >> >> > Doug championed supporting windows then and we've
>> continued
>> > > to
>> > > >> >> >> >> > do
>> > > >> >> >> >> > it
>> > > >> >> >> >> > with
>> > > >> >> >> >> > varying vigor over time.  To my knowledge we've never
>> made a
>> > > >> >> >> >> > decision
>> > > >> >> >> >> > to
>> > > >> >> >> >> > drop windows support.  The change here is improving our
>> > > support
>> > > >> >> >> >> > and
>> > > >> >> >> >> > dropping
>> > > >> >> >> >> > the requirement of cigwin.  We had Nutch windows users
>> on the
>> > > >> >> >> >> > list
>> > > >> >> >> >> > in
>> > > >> >> >> >> > 2006
>> > > >> >> >> >> > and we've been supporting windows FS requirements since
>> > > >> >> >> >> > inception.
>> > > >> >> >> >> >
>> > > >> >> >> >> > - A little pragmatism will go a long way.  As a community
>> > > we've
>> > > >> >> >> >> > got
>> > > >> >> >> >> > to
>> > > >> >> >> >> > stay committed to keeping hadoop simple (so it does work
>> on
>> > > >> >> >> >> > many
>> > > >> >> >> >> > platforms)
>> > > >> >> >> >> > and extending it to take advantage of key emerging
>> > > OS/hardware
>> > > >> >> >> >> > features,
>> > > >> >> >> >> > such as containers, new FSs, virtualization, flash ...
>>  We
>> > > >> >> >> >> > should
>> > > >> >> >> >> > all
>> > > >> >> >> >> > plan
>> > > >> >> >> >> > to let new features & optimizations emerge that don't
>> work
>> > > >> >> >> >> > everywhere, if
>> > > >> >> >> >> > they are compelling and central to hadoop's mission of
>> being
>> > > >> >> >> >> > THE
>> > > >> >> >> >> > best
>> > > >> >> >> >> > fabric
>> > > >> >> >> >> > for storing and processing big data.
>> > > >> >> >> >> >
>> > > >> >> >> >> > - A UI project like KDE has to deal with the MANY
>> differences
>> > > >> >> >> >> > between
>> > > >> >> >> >> > windows and linux UI APIs.  Hadoop faces no such complex
>> > > >> >> >> >> > challenge
>> > > >> >> >> >> > and hence
>> > > >> >> >> >> > can be maintained from a single codeline IMO.  It is
>> mostly
>> > > >> >> >> >> > abstracted from
>> > > >> >> >> >> > the OS APIs via Java and our design choices.  Where it
>> is not
>> > > >> >> >> >> > we
>> > > >> >> >> >> > can
>> > > >> >> >> >> > continue to add plugable abstractions.
>> > > >> >> >> >> >
>> > > >> >> >> >
>> > > >> >> >> >
>> > > >> >> >
>> > > >> >> >
>> > > >> >
>> > > >> >
>> > > >
>> > > >
>> > >
>>
>
>
>
> --
> Best regards,
>
>    - Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> (via Tom White)

Re: [Vote] Merge branch-trunk-win to trunk

[Andrew Purtell <ap...@apache.org>]

Sorry, that was my error selecting the wrong reply option.


On Mon, Mar 25, 2013 at 10:25 PM, Konstantin Shvachko
<sh...@gmail.com>wrote:

> Andrew, this used to be on all -dev lists. Let's keep it that way.
>
> To the point.
> Does this mean that people are silently porting windows changes to
> branch-2?
> New features on a branch should be voted first, no?
>
> Thanks,
> --Konstantin
>
>
> On Mon, Mar 25, 2013 at 1:36 PM, Andrew Purtell <ap...@apache.org>
> wrote:
> > Noticed this too. Simply a 'public' modifier is missing, but it's unclear
> > how this could not have been caught prior to check-in.
> >
> >
> > On Mon, Mar 25, 2013 at 9:17 PM, Konstantin Boudnik <co...@apache.org>
> wrote:
> >
> >> It doesn't look like any progress has been done on the ticket below in
> the
> >> last 3 weeks. And now branch-2 can't be compiled because of
> >>
> >>
> >>
> hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSShell.java:[895,15]
> >> WINDOWS is not public in org.apache.hadoop.fs.Path; cannot be accessed
> from
> >> outside package
> >>
> >> That's exactly why I was -1'ing this...
> >>   Cos
> >>
> >> On Mon, Mar 04, 2013 at 05:41PM, Matt Foley wrote:
> >> > Thanks, gentlemen.  I've opened and taken responsibility for
> >> > https://issues.apache.org/jira/browse/HADOOP-9359.  Giri Kesavan has
> >> agreed
> >> > to help with the parts that require Jenkins admin access.
> >> >
> >> > Thanks,
> >> > --Matt
> >> >
> >> >
> >> >
> >> > On Mon, Mar 4, 2013 at 5:00 PM, Konstantin Shvachko <
> >> shv.hadoop@gmail.com>wrote:
> >> >
> >> > > +1 on the merge.
> >> > >
> >> > > I am glad we agreed.
> >> > > Having Jira to track the CI effort is a good idea.
> >> > >
> >> > > Thanks,
> >> > > --Konstantin
> >> > >
> >> > > On Mon, Mar 4, 2013 at 3:29 PM, Matt Foley <mf...@hortonworks.com>
> >> wrote:
> >> > > > Thanks.  I agree Windows -1's in test-patch should not block
> commits.
> >> > > >
> >> > > > --Matt
> >> > > >
> >> > > >
> >> > > >
> >> > > > On Mon, Mar 4, 2013 at 2:30 PM, Konstantin Shvachko <
> >> > > shv.hadoop@gmail.com>
> >> > > > wrote:
> >> > > >>
> >> > > >> On Mon, Mar 4, 2013 at 12:22 PM, Matt Foley <
> mfoley@hortonworks.com
> >> >
> >> > > >> wrote:
> >> > > >> > Konstantine, you have voted -1, and stated some requirements
> >> before
> >> > > >> > you'll
> >> > > >> > withdraw that -1.  As I plan to do work to fulfill those
> >> > > requirements, I
> >> > > >> > want to make sure that what I'm proposing will, in fact,
> satisfy
> >> you.
> >> > > >> > That's why I'm asking, if we implement full "test-patch"
> >> integration
> >> > > for
> >> > > >> > Windows, does it seem to you that that would provide adequate
> >> support?
> >> > > >>
> >> > > >> Yes.
> >> > > >>
> >> > > >> > I have learned not to presume that my interpretation is
> correct.
> >>  My
> >> > > >> > interpretation of item #1 is that test-patch provides
> pre-commit
> >> > > build,
> >> > > >> > so
> >> > > >> > it would satisfy item #1.  But rather than assuming that I am
> >> > > >> > interpreting
> >> > > >> > it correctly, I simply want your agreement that it would, or if
> >> not,
> >> > > >> > clarification why it won't.
> >> > > >>
> >> > > >> I agree it will satisfy my item #1.
> >> > > >> I did not agree in my previous email, but I changed my mind
> based on
> >> > > >> the latest discussion. I have to explain why now.
> >> > > >> I was proposing nightly build because I did not want pre-commit
> >> build
> >> > > >> for Windows block commits to Linux. But if people are fine just
> >> ignoring
> >> > > >> -1s for the Windows part of the build it should be good.
> >> > > >>
> >> > > >> > Regarding item #2, it is also my interpretation that test-patch
> >> > > provides
> >> > > >> > an
> >> > > >> > on-demand (perhaps 20-minutes deferred) Jenkins build and unit
> >> test,
> >> > > >> > with
> >> > > >> > logs available to the developer, so it would satisfy item #2.
>  But
> >> > > >> > rather
> >> > > >> > than assuming that I am interpreting it correctly, I simply
> want
> >> your
> >> > > >> > agreement that it would, or if not, clarification why it won't.
> >> > > >>
> >> > > >> It will satisfy my item #2 in the following way:
> >> > > >> I can duplicate your pre-commit build for Windows and add an
> input
> >> > > >> parameter, which would let people run the build on their patches
> >> > > >> chosen from local machine rather than attaching them to Jiras.
> >> > > >>
> >> > > >> Thanks,
> >> > > >> --Konstantin
> >> > > >>
> >> > > >> > In agile terms, you are the Owner of these requirements.
>  Please
> >> give
> >> > > me
> >> > > >> > owner feedback as to whether my proposed work sounds like it
> will
> >> > > >> > satisfy
> >> > > >> > the requirements.
> >> > > >> >
> >> > > >> > Thank you,
> >> > > >> > --Matt
> >> > > >> >
> >> > > >> >
> >> > > >> > On Sun, Mar 3, 2013 at 12:16 PM, Konstantin Shvachko
> >> > > >> > <sh...@gmail.com>
> >> > > >> > wrote:
> >> > > >> >>
> >> > > >> >> Didn't I explain in details what I am asking for?
> >> > > >> >>
> >> > > >> >> Thanks,
> >> > > >> >> --Konst
> >> > > >> >>
> >> > > >> >> On Sun, Mar 3, 2013 at 11:08 AM, Matt Foley <
> >> mfoley@hortonworks.com>
> >> > > >> >> wrote:
> >> > > >> >> > Hi Konstantin,
> >> > > >> >> > I'd like to point out two things:
> >> > > >> >> > First, I already committed in this thread (email of Thu, Feb
> >> 28,
> >> > > 2013
> >> > > >> >> > at
> >> > > >> >> > 6:01 PM) to providing CI for Windows builds.  So please stop
> >> acting
> >> > > >> >> > like
> >> > > >> >> > I'm
> >> > > >> >> > resisting this idea or something.
> >> > > >> >> > Second, you didn't answer my question, you just kvetched
> about
> >> the
> >> > > >> >> > phrasing.
> >> > > >> >> > So I ask again:
> >> > > >> >> >
> >> > > >> >> > Will providing full "test-patch" integration (pre-commit
> build
> >> and
> >> > > >> >> > unit
> >> > > >> >> > test
> >> > > >> >> > triggered by Jira "Patch Available" state) satisfy your
> >> request for
> >> > > >> >> > functionality #1 and #2?  Yes or no, please.
> >> > > >> >> >
> >> > > >> >> > Thanks,
> >> > > >> >> > --Matt
> >> > > >> >> >
> >> > > >> >> >
> >> > > >> >> > On Sat, Mar 2, 2013 at 7:32 PM, Konstantin Shvachko
> >> > > >> >> > <sh...@gmail.com>
> >> > > >> >> > wrote:
> >> > > >> >> >>
> >> > > >> >> >> Hi Matt,
> >> > > >> >> >>
> >> > > >> >> >> On Sat, Mar 2, 2013 at 12:32 PM, Matt Foley <
> >> > > mfoley@hortonworks.com>
> >> > > >> >> >> wrote:
> >> > > >> >> >> > Konstantin,
> >> > > >> >> >> > I would like to explore what it would take to remove this
> >> > > >> >> >> > perceived
> >> > > >> >> >> > impediment --
> >> > > >> >> >>
> >> > > >> >> >> Glad you decided to explore. Thank you.
> >> > > >> >> >>
> >> > > >> >> >> > although I reserve the right to argue that this is not
> >> > > >> >> >> > pre-requisite to merging the cross-platform support
> patch.
> >> > > >> >> >>
> >> > > >> >> >> It's your right indeed. So as mine to question what the
> >> platform
> >> > > >> >> >> support means for you, which I believe remained unclear.
> >> > > >> >> >> I do not impede the change as you should have noticed. My
> >> > > >> >> >> requirement
> >> > > >> >> >> comes from my perception of the support, which means to me
> >> exactly
> >> > > >> >> >> two
> >> > > >> >> >> things:
> >> > > >> >> >> 1. The ability to recognise the code is broken for the
> >> platform
> >> > > >> >> >> 2. The ability to test new patches on the platform
> >> > > >> >> >> The latter is problematic, as many noticed in this thread,
> for
> >> > > those
> >> > > >> >> >> whose customary environment does not include Windows.
> >> > > >> >> >>
> >> > > >> >> >> > If we implemented full "test-patch" support for Windows
> on
> >> > > trunk,
> >> > > >> >> >> > would
> >> > > >> >> >> > that
> >> > > >> >> >> > fulfill both your items #1 and #2?  Please note that:
> >> > > >> >> >> > a) Pushing the "Patch Available" button in Jira shall
> cause
> >> a
> >> > > >> >> >> > pre-commit
> >> > > >> >> >> > build to start within, I believe, 20 minutes.
> >> > > >> >> >> > b) That build keeps logs for both java build and unit
> tests
> >> for
> >> > > >> >> >> > several
> >> > > >> >> >> > days, that are accessible to all viewers.
> >> > > >> >> >>
> >> > > >> >> >> In item #1 I mostly asking for the nightly build, which is
> >> simpler
> >> > > >> >> >> than "test-patch". The latter would be ideal from the
> platform
> >> > > >> >> >> support
> >> > > >> >> >> viewpoint, but it is for the community to decide if we want
> >> to add
> >> > > >> >> >> extra +3 hours to the build.
> >> > > >> >> >> Nightly build in my understanding is triggered by the timer
> >> rather
> >> > > >> >> >> than by Jira's "submit patch" button.  On Jenkins build
> >> > > >> >> >> configuration
> >> > > >> >> >> you can specify it under "Build periodically".
> >> > > >> >> >>
> >> > > >> >> >> > So, does this provide sufficient on-demand support that
> we
> >> don't
> >> > > >> >> >> > have
> >> > > >> >> >> > to
> >> > > >> >> >> > implement a whole new on-demand VM support structure of
> some
> >> > > sort
> >> > > >> >> >> > for
> >> > > >> >> >> > #2
> >> > > >> >> >> > (which would be an extraordinary and impractical demand)?
> >> > > >> >> >>
> >> > > >> >> >> I did not mention VMs. Item #2 means a build, which runs
> >> > > >> >> >> "test-patch"
> >> > > >> >> >> target with the file specified by a user (instead of a jira
> >> > > >> >> >> attachment).
> >> > > >> >> >> When user clicks "Build Now" link a box is displayed where
> the
> >> > > user
> >> > > >> >> >> can enter the file path containing the patch. This can be
> >> > > specified
> >> > > >> >> >> in
> >> > > >> >> >> the Build Configuration under "This build is
> parameterized" by
> >> > > >> >> >> choosing AddParameter / FileParameter. The build can run on
> >> the
> >> > > same
> >> > > >> >> >> Windows machine as the nightly build.
> >> > > >> >> >> Such build will let people test their patches for Windows
> on
> >> > > Jenkins
> >> > > >> >> >> if they don't posses a license for the right version of
> >> Windows.
> >> > > >> >> >> I hope this will not turn into extraordinary or impractical
> >> > > effort.
> >> > > >> >> >>
> >> > > >> >> >> Thanks,
> >> > > >> >> >> --Konst
> >> > > >> >> >>
> >> > > >> >> >> > Thanks,
> >> > > >> >> >> > --Matt
> >> > > >> >> >> >
> >> > > >> >> >> >
> >> > > >> >> >> > On Fri, Mar 1, 2013 at 12:18 PM, Konstantin Shvachko
> >> > > >> >> >> > <sh...@gmail.com>
> >> > > >> >> >> > wrote:
> >> > > >> >> >> >>
> >> > > >> >> >> >> -1
> >> > > >> >> >> >> We should have a CI infrastructure in place before we
> can
> >> > > commit
> >> > > >> >> >> >> to
> >> > > >> >> >> >> supporting Windows platform.
> >> > > >> >> >> >>
> >> > > >> >> >> >> Eric is right Win/Cygwin was supported since day one.
> >> > > >> >> >> >> I had a Windows box under my desk running nightly builds
> >> back
> >> > > in
> >> > > >> >> >> >> 2006-07.
> >> > > >> >> >> >> People were irritated but I was filing windows bugs
> until
> >> 0.22
> >> > > >> >> >> >> release.
> >> > > >> >> >> >> Times changing and I am glad to see wider support for
> Win
> >> > > >> >> >> >> platform.
> >> > > >> >> >> >>
> >> > > >> >> >> >> But in order to make it work you guys need to put the CI
> >> > > process
> >> > > >> >> >> >> in
> >> > > >> >> >> >> place
> >> > > >> >> >> >>
> >> > > >> >> >> >> 1. windows jenkins build: could be nightly or PreCommit.
> >> > > >> >> >> >> - Nightly would mean that changes can be committed to
> trunk
> >> > > based
> >> > > >> >> >> >> on
> >> > > >> >> >> >> linux PreCommit build. And people will file bugs if the
> >> change
> >> > > >> >> >> >> broke
> >> > > >> >> >> >> Windows nightly build.
> >> > > >> >> >> >> - PreCommit-win build will mean automatic reporting
> failed
> >> > > tests
> >> > > >> >> >> >> to
> >> > > >> >> >> >> respective jira blocking commits the same way as it is
> now
> >> with
> >> > > >> >> >> >> linux
> >> > > >> >> >> >> PreCommit builds.
> >> > > >> >> >> >> We should discuss which way is more efficient for
> >> developers.
> >> > > >> >> >> >>
> >> > > >> >> >> >> 2. On-demand-windows Jenkins build.
> >> > > >> >> >> >> I see it as a build to which I can attach my patch and
> the
> >> > > build
> >> > > >> >> >> >> will
> >> > > >> >> >> >> run my changes on a dedicated windows box.
> >> > > >> >> >> >> That way people can test their changes without having
> >> personal
> >> > > >> >> >> >> windows
> >> > > >> >> >> >> nodes.
> >> > > >> >> >> >>
> >> > > >> >> >> >> I think this is the minimal set of requirement for us
> to be
> >> > > able
> >> > > >> >> >> >> to
> >> > > >> >> >> >> commit to the new platform.
> >> > > >> >> >> >> Right now I see only one windows related build
> >> > > >> >> >> >> https://builds.apache.org/view/Hadoop/job/Hadoop-1-win/
> >> > > >> >> >> >> Which was failing since Sept 8, 2012 and did not run in
> the
> >> > > last
> >> > > >> >> >> >> month.
> >> > > >> >> >> >>
> >> > > >> >> >> >> Thanks,
> >> > > >> >> >> >> --Konst
> >> > > >> >> >> >>
> >> > > >> >> >> >> On Thu, Feb 28, 2013 at 8:47 PM, Eric Baldeschwieler
> >> > > >> >> >> >> <er...@hortonworks.com> wrote:
> >> > > >> >> >> >> > +1 (non-binding)
> >> > > >> >> >> >> >
> >> > > >> >> >> >> > A few of observations:
> >> > > >> >> >> >> >
> >> > > >> >> >> >> > - Windows has actually been a supported platform for
> >> Hadoop
> >> > > >> >> >> >> > since
> >> > > >> >> >> >> > 0.1
> >> > > >> >> >> >> > .
> >> > > >> >> >> >> > Doug championed supporting windows then and we've
> >> continued
> >> > > to
> >> > > >> >> >> >> > do
> >> > > >> >> >> >> > it
> >> > > >> >> >> >> > with
> >> > > >> >> >> >> > varying vigor over time.  To my knowledge we've never
> >> made a
> >> > > >> >> >> >> > decision
> >> > > >> >> >> >> > to
> >> > > >> >> >> >> > drop windows support.  The change here is improving
> our
> >> > > support
> >> > > >> >> >> >> > and
> >> > > >> >> >> >> > dropping
> >> > > >> >> >> >> > the requirement of cigwin.  We had Nutch windows users
> >> on the
> >> > > >> >> >> >> > list
> >> > > >> >> >> >> > in
> >> > > >> >> >> >> > 2006
> >> > > >> >> >> >> > and we've been supporting windows FS requirements
> since
> >> > > >> >> >> >> > inception.
> >> > > >> >> >> >> >
> >> > > >> >> >> >> > - A little pragmatism will go a long way.  As a
> community
> >> > > we've
> >> > > >> >> >> >> > got
> >> > > >> >> >> >> > to
> >> > > >> >> >> >> > stay committed to keeping hadoop simple (so it does
> work
> >> on
> >> > > >> >> >> >> > many
> >> > > >> >> >> >> > platforms)
> >> > > >> >> >> >> > and extending it to take advantage of key emerging
> >> > > OS/hardware
> >> > > >> >> >> >> > features,
> >> > > >> >> >> >> > such as containers, new FSs, virtualization, flash ...
> >>  We
> >> > > >> >> >> >> > should
> >> > > >> >> >> >> > all
> >> > > >> >> >> >> > plan
> >> > > >> >> >> >> > to let new features & optimizations emerge that don't
> >> work
> >> > > >> >> >> >> > everywhere, if
> >> > > >> >> >> >> > they are compelling and central to hadoop's mission of
> >> being
> >> > > >> >> >> >> > THE
> >> > > >> >> >> >> > best
> >> > > >> >> >> >> > fabric
> >> > > >> >> >> >> > for storing and processing big data.
> >> > > >> >> >> >> >
> >> > > >> >> >> >> > - A UI project like KDE has to deal with the MANY
> >> differences
> >> > > >> >> >> >> > between
> >> > > >> >> >> >> > windows and linux UI APIs.  Hadoop faces no such
> complex
> >> > > >> >> >> >> > challenge
> >> > > >> >> >> >> > and hence
> >> > > >> >> >> >> > can be maintained from a single codeline IMO.  It is
> >> mostly
> >> > > >> >> >> >> > abstracted from
> >> > > >> >> >> >> > the OS APIs via Java and our design choices.  Where it
> >> is not
> >> > > >> >> >> >> > we
> >> > > >> >> >> >> > can
> >> > > >> >> >> >> > continue to add plugable abstractions.
> >> > > >> >> >> >> >
> >> > > >> >> >> >
> >> > > >> >> >> >
> >> > > >> >> >
> >> > > >> >> >
> >> > > >> >
> >> > > >> >
> >> > > >
> >> > > >
> >> > >
> >>
> >
> >
> >
> > --
> > Best regards,
> >
> >    - Andy
> >
> > Problems worthy of attack prove their worth by hitting back. - Piet Hein
> > (via Tom White)
>



-- 
Best regards,

   - Andy

Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)

Re: [Vote] Merge branch-trunk-win to trunk

[Tsz Wo Sze <sz...@yahoo.com>]

Hi Konstantin S,

I accidentally merged some windows bug fixes to branch-2.  However, branch-2 did not compile -- it showed that the windows changes were missing in branch-2.  The ones causing compilation problems were already reverted (Thanks Sid and Suresh.)  Sorry for the inconvenience.  


Tsz-Wo




________________________________
 From: Konstantin Shvachko <sh...@gmail.com>
To: mapreduce-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org; common-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org 
Sent: Tuesday, March 26, 2013 5:25 AM
Subject: Re: [Vote] Merge branch-trunk-win to trunk
 
Andrew, this used to be on all -dev lists. Let's keep it that way.

To the point.
Does this mean that people are silently porting windows changes to branch-2?
New features on a branch should be voted first, no?

Thanks,
--Konstantin


On Mon, Mar 25, 2013 at 1:36 PM, Andrew Purtell <ap...@apache.org> wrote:
> Noticed this too. Simply a 'public' modifier is missing, but it's unclear
> how this could not have been caught prior to check-in.
>
>
> On Mon, Mar 25, 2013 at 9:17 PM, Konstantin Boudnik <co...@apache.org> wrote:
>
>> It doesn't look like any progress has been done on the ticket below in the
>> last 3 weeks. And now branch-2 can't be compiled because of
>>
>>
>> hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSShell.java:[895,15]
>> WINDOWS is not public in org.apache.hadoop.fs.Path; cannot be accessed from
>> outside package
>>
>> That's exactly why I was -1'ing this...
>>   Cos
>>
>> On Mon, Mar 04, 2013 at 05:41PM, Matt Foley wrote:
>> > Thanks, gentlemen.  I've opened and taken responsibility for
>> > https://issues.apache.org/jira/browse/HADOOP-9359.  Giri Kesavan has
>> agreed
>> > to help with the parts that require Jenkins admin access.
>> >
>> > Thanks,
>> > --Matt
>> >
>> >
>> >
>> > On Mon, Mar 4, 2013 at 5:00 PM, Konstantin Shvachko <
>> shv.hadoop@gmail.com>wrote:
>> >
>> > > +1 on the merge.
>> > >
>> > > I am glad we agreed.
>> > > Having Jira to track the CI effort is a good idea.
>> > >
>> > > Thanks,
>> > > --Konstantin
>> > >
>> > > On Mon, Mar 4, 2013 at 3:29 PM, Matt Foley <mf...@hortonworks.com>
>> wrote:
>> > > > Thanks.  I agree Windows -1's in test-patch should not block commits.
>> > > >
>> > > > --Matt
>> > > >
>> > > >
>> > > >
>> > > > On Mon, Mar 4, 2013 at 2:30 PM, Konstantin Shvachko <
>> > > shv.hadoop@gmail.com>
>> > > > wrote:
>> > > >>
>> > > >> On Mon, Mar 4, 2013 at 12:22 PM, Matt Foley <mfoley@hortonworks.com
>> >
>> > > >> wrote:
>> > > >> > Konstantine, you have voted -1, and stated some requirements
>> before
>> > > >> > you'll
>> > > >> > withdraw that -1.  As I plan to do work to fulfill those
>> > > requirements, I
>> > > >> > want to make sure that what I'm proposing will, in fact, satisfy
>> you.
>> > > >> > That's why I'm asking, if we implement full "test-patch"
>> integration
>> > > for
>> > > >> > Windows, does it seem to you that that would provide adequate
>> support?
>> > > >>
>> > > >> Yes.
>> > > >>
>> > > >> > I have learned not to presume that my interpretation is correct.
>>  My
>> > > >> > interpretation of item #1 is that test-patch provides pre-commit
>> > > build,
>> > > >> > so
>> > > >> > it would satisfy item #1.  But rather than assuming that I am
>> > > >> > interpreting
>> > > >> > it correctly, I simply want your agreement that it would, or if
>> not,
>> > > >> > clarification why it won't.
>> > > >>
>> > > >> I agree it will satisfy my item #1.
>> > > >> I did not agree in my previous email, but I changed my mind based on
>> > > >> the latest discussion. I have to explain why now.
>> > > >> I was proposing nightly build because I did not want pre-commit
>> build
>> > > >> for Windows block commits to Linux. But if people are fine just
>> ignoring
>> > > >> -1s for the Windows part of the build it should be good.
>> > > >>
>> > > >> > Regarding item #2, it is also my interpretation that test-patch
>> > > provides
>> > > >> > an
>> > > >> > on-demand (perhaps 20-minutes deferred) Jenkins build and unit
>> test,
>> > > >> > with
>> > > >> > logs available to the developer, so it would satisfy item #2.  But
>> > > >> > rather
>> > > >> > than assuming that I am interpreting it correctly, I simply want
>> your
>> > > >> > agreement that it would, or if not, clarification why it won't.
>> > > >>
>> > > >> It will satisfy my item #2 in the following way:
>> > > >> I can duplicate your pre-commit build for Windows and add an input
>> > > >> parameter, which would let people run the build on their patches
>> > > >> chosen from local machine rather than attaching them to Jiras.
>> > > >>
>> > > >> Thanks,
>> > > >> --Konstantin
>> > > >>
>> > > >> > In agile terms, you are the Owner of these requirements.  Please
>> give
>> > > me
>> > > >> > owner feedback as to whether my proposed work sounds like it will
>> > > >> > satisfy
>> > > >> > the requirements.
>> > > >> >
>> > > >> > Thank you,
>> > > >> > --Matt
>> > > >> >
>> > > >> >
>> > > >> > On Sun, Mar 3, 2013 at 12:16 PM, Konstantin Shvachko
>> > > >> > <sh...@gmail.com>
>> > > >> > wrote:
>> > > >> >>
>> > > >> >> Didn't I explain in details what I am asking for?
>> > > >> >>
>> > > >> >> Thanks,
>> > > >> >> --Konst
>> > > >> >>
>> > > >> >> On Sun, Mar 3, 2013 at 11:08 AM, Matt Foley <
>> mfoley@hortonworks.com>
>> > > >> >> wrote:
>> > > >> >> > Hi Konstantin,
>> > > >> >> > I'd like to point out two things:
>> > > >> >> > First, I already committed in this thread (email of Thu, Feb
>> 28,
>> > > 2013
>> > > >> >> > at
>> > > >> >> > 6:01 PM) to providing CI for Windows builds.  So please stop
>> acting
>> > > >> >> > like
>> > > >> >> > I'm
>> > > >> >> > resisting this idea or something.
>> > > >> >> > Second, you didn't answer my question, you just kvetched about
>> the
>> > > >> >> > phrasing.
>> > > >> >> > So I ask again:
>> > > >> >> >
>> > > >> >> > Will providing full "test-patch" integration (pre-commit build
>> and
>> > > >> >> > unit
>> > > >> >> > test
>> > > >> >> > triggered by Jira "Patch Available" state) satisfy your
>> request for
>> > > >> >> > functionality #1 and #2?  Yes or no, please.
>> > > >> >> >
>> > > >> >> > Thanks,
>> > > >> >> > --Matt
>> > > >> >> >
>> > > >> >> >
>> > > >> >> > On Sat, Mar 2, 2013 at 7:32 PM, Konstantin Shvachko
>> > > >> >> > <sh...@gmail.com>
>> > > >> >> > wrote:
>> > > >> >> >>
>> > > >> >> >> Hi Matt,
>> > > >> >> >>
>> > > >> >> >> On Sat, Mar 2, 2013 at 12:32 PM, Matt Foley <
>> > > mfoley@hortonworks.com>
>> > > >> >> >> wrote:
>> > > >> >> >> > Konstantin,
>> > > >> >> >> > I would like to explore what it would take to remove this
>> > > >> >> >> > perceived
>> > > >> >> >> > impediment --
>> > > >> >> >>
>> > > >> >> >> Glad you decided to explore. Thank you.
>> > > >> >> >>
>> > > >> >> >> > although I reserve the right to argue that this is not
>> > > >> >> >> > pre-requisite to merging the cross-platform support patch.
>> > > >> >> >>
>> > > >> >> >> It's your right indeed. So as mine to question what the
>> platform
>> > > >> >> >> support means for you, which I believe remained unclear.
>> > > >> >> >> I do not impede the change as you should have noticed. My
>> > > >> >> >> requirement
>> > > >> >> >> comes from my perception of the support, which means to me
>> exactly
>> > > >> >> >> two
>> > > >> >> >> things:
>> > > >> >> >> 1. The ability to recognise the code is broken for the
>> platform
>> > > >> >> >> 2. The ability to test new patches on the platform
>> > > >> >> >> The latter is problematic, as many noticed in this thread, for
>> > > those
>> > > >> >> >> whose customary environment does not include Windows.
>> > > >> >> >>
>> > > >> >> >> > If we implemented full "test-patch" support for Windows on
>> > > trunk,
>> > > >> >> >> > would
>> > > >> >> >> > that
>> > > >> >> >> > fulfill both your items #1 and #2?  Please note that:
>> > > >> >> >> > a) Pushing the "Patch Available" button in Jira shall cause
>> a
>> > > >> >> >> > pre-commit
>> > > >> >> >> > build to start within, I believe, 20 minutes.
>> > > >> >> >> > b) That build keeps logs for both java build and unit tests
>> for
>> > > >> >> >> > several
>> > > >> >> >> > days, that are accessible to all viewers.
>> > > >> >> >>
>> > > >> >> >> In item #1 I mostly asking for the nightly build, which is
>> simpler
>> > > >> >> >> than "test-patch". The latter would be ideal from the platform
>> > > >> >> >> support
>> > > >> >> >> viewpoint, but it is for the community to decide if we want
>> to add
>> > > >> >> >> extra +3 hours to the build.
>> > > >> >> >> Nightly build in my understanding is triggered by the timer
>> rather
>> > > >> >> >> than by Jira's "submit patch" button.  On Jenkins build
>> > > >> >> >> configuration
>> > > >> >> >> you can specify it under "Build periodically".
>> > > >> >> >>
>> > > >> >> >> > So, does this provide sufficient on-demand support that we
>> don't
>> > > >> >> >> > have
>> > > >> >> >> > to
>> > > >> >> >> > implement a whole new on-demand VM support structure of some
>> > > sort
>> > > >> >> >> > for
>> > > >> >> >> > #2
>> > > >> >> >> > (which would be an extraordinary and impractical demand)?
>> > > >> >> >>
>> > > >> >> >> I did not mention VMs. Item #2 means a build, which runs
>> > > >> >> >> "test-patch"
>> > > >> >> >> target with the file specified by a user (instead of a jira
>> > > >> >> >> attachment).
>> > > >> >> >> When user clicks "Build Now" link a box is displayed where the
>> > > user
>> > > >> >> >> can enter the file path containing the patch. This can be
>> > > specified
>> > > >> >> >> in
>> > > >> >> >> the Build Configuration under "This build is parameterized" by
>> > > >> >> >> choosing AddParameter / FileParameter. The build can run on
>> the
>> > > same
>> > > >> >> >> Windows machine as the nightly build.
>> > > >> >> >> Such build will let people test their patches for Windows on
>> > > Jenkins
>> > > >> >> >> if they don't posses a license for the right version of
>> Windows.
>> > > >> >> >> I hope this will not turn into extraordinary or impractical
>> > > effort.
>> > > >> >> >>
>> > > >> >> >> Thanks,
>> > > >> >> >> --Konst
>> > > >> >> >>
>> > > >> >> >> > Thanks,
>> > > >> >> >> > --Matt
>> > > >> >> >> >
>> > > >> >> >> >
>> > > >> >> >> > On Fri, Mar 1, 2013 at 12:18 PM, Konstantin Shvachko
>> > > >> >> >> > <sh...@gmail.com>
>> > > >> >> >> > wrote:
>> > > >> >> >> >>
>> > > >> >> >> >> -1
>> > > >> >> >> >> We should have a CI infrastructure in place before we can
>> > > commit
>> > > >> >> >> >> to
>> > > >> >> >> >> supporting Windows platform.
>> > > >> >> >> >>
>> > > >> >> >> >> Eric is right Win/Cygwin was supported since day one.
>> > > >> >> >> >> I had a Windows box under my desk running nightly builds
>> back
>> > > in
>> > > >> >> >> >> 2006-07.
>> > > >> >> >> >> People were irritated but I was filing windows bugs until
>> 0.22
>> > > >> >> >> >> release.
>> > > >> >> >> >> Times changing and I am glad to see wider support for Win
>> > > >> >> >> >> platform.
>> > > >> >> >> >>
>> > > >> >> >> >> But in order to make it work you guys need to put the CI
>> > > process
>> > > >> >> >> >> in
>> > > >> >> >> >> place
>> > > >> >> >> >>
>> > > >> >> >> >> 1. windows jenkins build: could be nightly or PreCommit.
>> > > >> >> >> >> - Nightly would mean that changes can be committed to trunk
>> > > based
>> > > >> >> >> >> on
>> > > >> >> >> >> linux PreCommit build. And people will file bugs if the
>> change
>> > > >> >> >> >> broke
>> > > >> >> >> >> Windows nightly build.
>> > > >> >> >> >> - PreCommit-win build will mean automatic reporting failed
>> > > tests
>> > > >> >> >> >> to
>> > > >> >> >> >> respective jira blocking commits the same way as it is now
>> with
>> > > >> >> >> >> linux
>> > > >> >> >> >> PreCommit builds.
>> > > >> >> >> >> We should discuss which way is more efficient for
>> developers.
>> > > >> >> >> >>
>> > > >> >> >> >> 2. On-demand-windows Jenkins build.
>> > > >> >> >> >> I see it as a build to which I can attach my patch and the
>> > > build
>> > > >> >> >> >> will
>> > > >> >> >> >> run my changes on a dedicated windows box.
>> > > >> >> >> >> That way people can test their changes without having
>> personal
>> > > >> >> >> >> windows
>> > > >> >> >> >> nodes.
>> > > >> >> >> >>
>> > > >> >> >> >> I think this is the minimal set of requirement for us to be
>> > > able
>> > > >> >> >> >> to
>> > > >> >> >> >> commit to the new platform.
>> > > >> >> >> >> Right now I see only one windows related build
>> > > >> >> >> >> https://builds.apache.org/view/Hadoop/job/Hadoop-1-win/
>> > > >> >> >> >> Which was failing since Sept 8, 2012 and did not run in the
>> > > last
>> > > >> >> >> >> month.
>> > > >> >> >> >>
>> > > >> >> >> >> Thanks,
>> > > >> >> >> >> --Konst
>> > > >> >> >> >>
>> > > >> >> >> >> On Thu, Feb 28, 2013 at 8:47 PM, Eric Baldeschwieler
>> > > >> >> >> >> <er...@hortonworks.com> wrote:
>> > > >> >> >> >> > +1 (non-binding)
>> > > >> >> >> >> >
>> > > >> >> >> >> > A few of observations:
>> > > >> >> >> >> >
>> > > >> >> >> >> > - Windows has actually been a supported platform for
>> Hadoop
>> > > >> >> >> >> > since
>> > > >> >> >> >> > 0.1
>> > > >> >> >> >> > .
>> > > >> >> >> >> > Doug championed supporting windows then and we've
>> continued
>> > > to
>> > > >> >> >> >> > do
>> > > >> >> >> >> > it
>> > > >> >> >> >> > with
>> > > >> >> >> >> > varying vigor over time.  To my knowledge we've never
>> made a
>> > > >> >> >> >> > decision
>> > > >> >> >> >> > to
>> > > >> >> >> >> > drop windows support.  The change here is improving our
>> > > support
>> > > >> >> >> >> > and
>> > > >> >> >> >> > dropping
>> > > >> >> >> >> > the requirement of cigwin.  We had Nutch windows users
>> on the
>> > > >> >> >> >> > list
>> > > >> >> >> >> > in
>> > > >> >> >> >> > 2006
>> > > >> >> >> >> > and we've been supporting windows FS requirements since
>> > > >> >> >> >> > inception.
>> > > >> >> >> >> >
>> > > >> >> >> >> > - A little pragmatism will go a long way.  As a community
>> > > we've
>> > > >> >> >> >> > got
>> > > >> >> >> >> > to
>> > > >> >> >> >> > stay committed to keeping hadoop simple (so it does work
>> on
>> > > >> >> >> >> > many
>> > > >> >> >> >> > platforms)
>> > > >> >> >> >> > and extending it to take advantage of key emerging
>> > > OS/hardware
>> > > >> >> >> >> > features,
>> > > >> >> >> >> > such as containers, new FSs, virtualization, flash ...
>>  We
>> > > >> >> >> >> > should
>> > > >> >> >> >> > all
>> > > >> >> >> >> > plan
>> > > >> >> >> >> > to let new features & optimizations emerge that don't
>> work
>> > > >> >> >> >> > everywhere, if
>> > > >> >> >> >> > they are compelling and central to hadoop's mission of
>> being
>> > > >> >> >> >> > THE
>> > > >> >> >> >> > best
>> > > >> >> >> >> > fabric
>> > > >> >> >> >> > for storing and processing big data.
>> > > >> >> >> >> >
>> > > >> >> >> >> > - A UI project like KDE has to deal with the MANY
>> differences
>> > > >> >> >> >> > between
>> > > >> >> >> >> > windows and linux UI APIs.  Hadoop faces no such complex
>> > > >> >> >> >> > challenge
>> > > >> >> >> >> > and hence
>> > > >> >> >> >> > can be maintained from a single codeline IMO.  It is
>> mostly
>> > > >> >> >> >> > abstracted from
>> > > >> >> >> >> > the OS APIs via Java and our design choices.  Where it
>> is not
>> > > >> >> >> >> > we
>> > > >> >> >> >> > can
>> > > >> >> >> >> > continue to add plugable abstractions.
>> > > >> >> >> >> >
>> > > >> >> >> >
>> > > >> >> >> >
>> > > >> >> >
>> > > >> >> >
>> > > >> >
>> > > >> >
>> > > >
>> > > >
>> > >
>>
>
>
>
> --
> Best regards,
>
>    - Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> (via Tom White)

Re: [Vote] Merge branch-trunk-win to trunk

[Andrew Purtell <ap...@apache.org>]

Sorry, that was my error selecting the wrong reply option.


On Mon, Mar 25, 2013 at 10:25 PM, Konstantin Shvachko
<sh...@gmail.com>wrote:

> Andrew, this used to be on all -dev lists. Let's keep it that way.
>
> To the point.
> Does this mean that people are silently porting windows changes to
> branch-2?
> New features on a branch should be voted first, no?
>
> Thanks,
> --Konstantin
>
>
> On Mon, Mar 25, 2013 at 1:36 PM, Andrew Purtell <ap...@apache.org>
> wrote:
> > Noticed this too. Simply a 'public' modifier is missing, but it's unclear
> > how this could not have been caught prior to check-in.
> >
> >
> > On Mon, Mar 25, 2013 at 9:17 PM, Konstantin Boudnik <co...@apache.org>
> wrote:
> >
> >> It doesn't look like any progress has been done on the ticket below in
> the
> >> last 3 weeks. And now branch-2 can't be compiled because of
> >>
> >>
> >>
> hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSShell.java:[895,15]
> >> WINDOWS is not public in org.apache.hadoop.fs.Path; cannot be accessed
> from
> >> outside package
> >>
> >> That's exactly why I was -1'ing this...
> >>   Cos
> >>
> >> On Mon, Mar 04, 2013 at 05:41PM, Matt Foley wrote:
> >> > Thanks, gentlemen.  I've opened and taken responsibility for
> >> > https://issues.apache.org/jira/browse/HADOOP-9359.  Giri Kesavan has
> >> agreed
> >> > to help with the parts that require Jenkins admin access.
> >> >
> >> > Thanks,
> >> > --Matt
> >> >
> >> >
> >> >
> >> > On Mon, Mar 4, 2013 at 5:00 PM, Konstantin Shvachko <
> >> shv.hadoop@gmail.com>wrote:
> >> >
> >> > > +1 on the merge.
> >> > >
> >> > > I am glad we agreed.
> >> > > Having Jira to track the CI effort is a good idea.
> >> > >
> >> > > Thanks,
> >> > > --Konstantin
> >> > >
> >> > > On Mon, Mar 4, 2013 at 3:29 PM, Matt Foley <mf...@hortonworks.com>
> >> wrote:
> >> > > > Thanks.  I agree Windows -1's in test-patch should not block
> commits.
> >> > > >
> >> > > > --Matt
> >> > > >
> >> > > >
> >> > > >
> >> > > > On Mon, Mar 4, 2013 at 2:30 PM, Konstantin Shvachko <
> >> > > shv.hadoop@gmail.com>
> >> > > > wrote:
> >> > > >>
> >> > > >> On Mon, Mar 4, 2013 at 12:22 PM, Matt Foley <
> mfoley@hortonworks.com
> >> >
> >> > > >> wrote:
> >> > > >> > Konstantine, you have voted -1, and stated some requirements
> >> before
> >> > > >> > you'll
> >> > > >> > withdraw that -1.  As I plan to do work to fulfill those
> >> > > requirements, I
> >> > > >> > want to make sure that what I'm proposing will, in fact,
> satisfy
> >> you.
> >> > > >> > That's why I'm asking, if we implement full "test-patch"
> >> integration
> >> > > for
> >> > > >> > Windows, does it seem to you that that would provide adequate
> >> support?
> >> > > >>
> >> > > >> Yes.
> >> > > >>
> >> > > >> > I have learned not to presume that my interpretation is
> correct.
> >>  My
> >> > > >> > interpretation of item #1 is that test-patch provides
> pre-commit
> >> > > build,
> >> > > >> > so
> >> > > >> > it would satisfy item #1.  But rather than assuming that I am
> >> > > >> > interpreting
> >> > > >> > it correctly, I simply want your agreement that it would, or if
> >> not,
> >> > > >> > clarification why it won't.
> >> > > >>
> >> > > >> I agree it will satisfy my item #1.
> >> > > >> I did not agree in my previous email, but I changed my mind
> based on
> >> > > >> the latest discussion. I have to explain why now.
> >> > > >> I was proposing nightly build because I did not want pre-commit
> >> build
> >> > > >> for Windows block commits to Linux. But if people are fine just
> >> ignoring
> >> > > >> -1s for the Windows part of the build it should be good.
> >> > > >>
> >> > > >> > Regarding item #2, it is also my interpretation that test-patch
> >> > > provides
> >> > > >> > an
> >> > > >> > on-demand (perhaps 20-minutes deferred) Jenkins build and unit
> >> test,
> >> > > >> > with
> >> > > >> > logs available to the developer, so it would satisfy item #2.
>  But
> >> > > >> > rather
> >> > > >> > than assuming that I am interpreting it correctly, I simply
> want
> >> your
> >> > > >> > agreement that it would, or if not, clarification why it won't.
> >> > > >>
> >> > > >> It will satisfy my item #2 in the following way:
> >> > > >> I can duplicate your pre-commit build for Windows and add an
> input
> >> > > >> parameter, which would let people run the build on their patches
> >> > > >> chosen from local machine rather than attaching them to Jiras.
> >> > > >>
> >> > > >> Thanks,
> >> > > >> --Konstantin
> >> > > >>
> >> > > >> > In agile terms, you are the Owner of these requirements.
>  Please
> >> give
> >> > > me
> >> > > >> > owner feedback as to whether my proposed work sounds like it
> will
> >> > > >> > satisfy
> >> > > >> > the requirements.
> >> > > >> >
> >> > > >> > Thank you,
> >> > > >> > --Matt
> >> > > >> >
> >> > > >> >
> >> > > >> > On Sun, Mar 3, 2013 at 12:16 PM, Konstantin Shvachko
> >> > > >> > <sh...@gmail.com>
> >> > > >> > wrote:
> >> > > >> >>
> >> > > >> >> Didn't I explain in details what I am asking for?
> >> > > >> >>
> >> > > >> >> Thanks,
> >> > > >> >> --Konst
> >> > > >> >>
> >> > > >> >> On Sun, Mar 3, 2013 at 11:08 AM, Matt Foley <
> >> mfoley@hortonworks.com>
> >> > > >> >> wrote:
> >> > > >> >> > Hi Konstantin,
> >> > > >> >> > I'd like to point out two things:
> >> > > >> >> > First, I already committed in this thread (email of Thu, Feb
> >> 28,
> >> > > 2013
> >> > > >> >> > at
> >> > > >> >> > 6:01 PM) to providing CI for Windows builds.  So please stop
> >> acting
> >> > > >> >> > like
> >> > > >> >> > I'm
> >> > > >> >> > resisting this idea or something.
> >> > > >> >> > Second, you didn't answer my question, you just kvetched
> about
> >> the
> >> > > >> >> > phrasing.
> >> > > >> >> > So I ask again:
> >> > > >> >> >
> >> > > >> >> > Will providing full "test-patch" integration (pre-commit
> build
> >> and
> >> > > >> >> > unit
> >> > > >> >> > test
> >> > > >> >> > triggered by Jira "Patch Available" state) satisfy your
> >> request for
> >> > > >> >> > functionality #1 and #2?  Yes or no, please.
> >> > > >> >> >
> >> > > >> >> > Thanks,
> >> > > >> >> > --Matt
> >> > > >> >> >
> >> > > >> >> >
> >> > > >> >> > On Sat, Mar 2, 2013 at 7:32 PM, Konstantin Shvachko
> >> > > >> >> > <sh...@gmail.com>
> >> > > >> >> > wrote:
> >> > > >> >> >>
> >> > > >> >> >> Hi Matt,
> >> > > >> >> >>
> >> > > >> >> >> On Sat, Mar 2, 2013 at 12:32 PM, Matt Foley <
> >> > > mfoley@hortonworks.com>
> >> > > >> >> >> wrote:
> >> > > >> >> >> > Konstantin,
> >> > > >> >> >> > I would like to explore what it would take to remove this
> >> > > >> >> >> > perceived
> >> > > >> >> >> > impediment --
> >> > > >> >> >>
> >> > > >> >> >> Glad you decided to explore. Thank you.
> >> > > >> >> >>
> >> > > >> >> >> > although I reserve the right to argue that this is not
> >> > > >> >> >> > pre-requisite to merging the cross-platform support
> patch.
> >> > > >> >> >>
> >> > > >> >> >> It's your right indeed. So as mine to question what the
> >> platform
> >> > > >> >> >> support means for you, which I believe remained unclear.
> >> > > >> >> >> I do not impede the change as you should have noticed. My
> >> > > >> >> >> requirement
> >> > > >> >> >> comes from my perception of the support, which means to me
> >> exactly
> >> > > >> >> >> two
> >> > > >> >> >> things:
> >> > > >> >> >> 1. The ability to recognise the code is broken for the
> >> platform
> >> > > >> >> >> 2. The ability to test new patches on the platform
> >> > > >> >> >> The latter is problematic, as many noticed in this thread,
> for
> >> > > those
> >> > > >> >> >> whose customary environment does not include Windows.
> >> > > >> >> >>
> >> > > >> >> >> > If we implemented full "test-patch" support for Windows
> on
> >> > > trunk,
> >> > > >> >> >> > would
> >> > > >> >> >> > that
> >> > > >> >> >> > fulfill both your items #1 and #2?  Please note that:
> >> > > >> >> >> > a) Pushing the "Patch Available" button in Jira shall
> cause
> >> a
> >> > > >> >> >> > pre-commit
> >> > > >> >> >> > build to start within, I believe, 20 minutes.
> >> > > >> >> >> > b) That build keeps logs for both java build and unit
> tests
> >> for
> >> > > >> >> >> > several
> >> > > >> >> >> > days, that are accessible to all viewers.
> >> > > >> >> >>
> >> > > >> >> >> In item #1 I mostly asking for the nightly build, which is
> >> simpler
> >> > > >> >> >> than "test-patch". The latter would be ideal from the
> platform
> >> > > >> >> >> support
> >> > > >> >> >> viewpoint, but it is for the community to decide if we want
> >> to add
> >> > > >> >> >> extra +3 hours to the build.
> >> > > >> >> >> Nightly build in my understanding is triggered by the timer
> >> rather
> >> > > >> >> >> than by Jira's "submit patch" button.  On Jenkins build
> >> > > >> >> >> configuration
> >> > > >> >> >> you can specify it under "Build periodically".
> >> > > >> >> >>
> >> > > >> >> >> > So, does this provide sufficient on-demand support that
> we
> >> don't
> >> > > >> >> >> > have
> >> > > >> >> >> > to
> >> > > >> >> >> > implement a whole new on-demand VM support structure of
> some
> >> > > sort
> >> > > >> >> >> > for
> >> > > >> >> >> > #2
> >> > > >> >> >> > (which would be an extraordinary and impractical demand)?
> >> > > >> >> >>
> >> > > >> >> >> I did not mention VMs. Item #2 means a build, which runs
> >> > > >> >> >> "test-patch"
> >> > > >> >> >> target with the file specified by a user (instead of a jira
> >> > > >> >> >> attachment).
> >> > > >> >> >> When user clicks "Build Now" link a box is displayed where
> the
> >> > > user
> >> > > >> >> >> can enter the file path containing the patch. This can be
> >> > > specified
> >> > > >> >> >> in
> >> > > >> >> >> the Build Configuration under "This build is
> parameterized" by
> >> > > >> >> >> choosing AddParameter / FileParameter. The build can run on
> >> the
> >> > > same
> >> > > >> >> >> Windows machine as the nightly build.
> >> > > >> >> >> Such build will let people test their patches for Windows
> on
> >> > > Jenkins
> >> > > >> >> >> if they don't posses a license for the right version of
> >> Windows.
> >> > > >> >> >> I hope this will not turn into extraordinary or impractical
> >> > > effort.
> >> > > >> >> >>
> >> > > >> >> >> Thanks,
> >> > > >> >> >> --Konst
> >> > > >> >> >>
> >> > > >> >> >> > Thanks,
> >> > > >> >> >> > --Matt
> >> > > >> >> >> >
> >> > > >> >> >> >
> >> > > >> >> >> > On Fri, Mar 1, 2013 at 12:18 PM, Konstantin Shvachko
> >> > > >> >> >> > <sh...@gmail.com>
> >> > > >> >> >> > wrote:
> >> > > >> >> >> >>
> >> > > >> >> >> >> -1
> >> > > >> >> >> >> We should have a CI infrastructure in place before we
> can
> >> > > commit
> >> > > >> >> >> >> to
> >> > > >> >> >> >> supporting Windows platform.
> >> > > >> >> >> >>
> >> > > >> >> >> >> Eric is right Win/Cygwin was supported since day one.
> >> > > >> >> >> >> I had a Windows box under my desk running nightly builds
> >> back
> >> > > in
> >> > > >> >> >> >> 2006-07.
> >> > > >> >> >> >> People were irritated but I was filing windows bugs
> until
> >> 0.22
> >> > > >> >> >> >> release.
> >> > > >> >> >> >> Times changing and I am glad to see wider support for
> Win
> >> > > >> >> >> >> platform.
> >> > > >> >> >> >>
> >> > > >> >> >> >> But in order to make it work you guys need to put the CI
> >> > > process
> >> > > >> >> >> >> in
> >> > > >> >> >> >> place
> >> > > >> >> >> >>
> >> > > >> >> >> >> 1. windows jenkins build: could be nightly or PreCommit.
> >> > > >> >> >> >> - Nightly would mean that changes can be committed to
> trunk
> >> > > based
> >> > > >> >> >> >> on
> >> > > >> >> >> >> linux PreCommit build. And people will file bugs if the
> >> change
> >> > > >> >> >> >> broke
> >> > > >> >> >> >> Windows nightly build.
> >> > > >> >> >> >> - PreCommit-win build will mean automatic reporting
> failed
> >> > > tests
> >> > > >> >> >> >> to
> >> > > >> >> >> >> respective jira blocking commits the same way as it is
> now
> >> with
> >> > > >> >> >> >> linux
> >> > > >> >> >> >> PreCommit builds.
> >> > > >> >> >> >> We should discuss which way is more efficient for
> >> developers.
> >> > > >> >> >> >>
> >> > > >> >> >> >> 2. On-demand-windows Jenkins build.
> >> > > >> >> >> >> I see it as a build to which I can attach my patch and
> the
> >> > > build
> >> > > >> >> >> >> will
> >> > > >> >> >> >> run my changes on a dedicated windows box.
> >> > > >> >> >> >> That way people can test their changes without having
> >> personal
> >> > > >> >> >> >> windows
> >> > > >> >> >> >> nodes.
> >> > > >> >> >> >>
> >> > > >> >> >> >> I think this is the minimal set of requirement for us
> to be
> >> > > able
> >> > > >> >> >> >> to
> >> > > >> >> >> >> commit to the new platform.
> >> > > >> >> >> >> Right now I see only one windows related build
> >> > > >> >> >> >> https://builds.apache.org/view/Hadoop/job/Hadoop-1-win/
> >> > > >> >> >> >> Which was failing since Sept 8, 2012 and did not run in
> the
> >> > > last
> >> > > >> >> >> >> month.
> >> > > >> >> >> >>
> >> > > >> >> >> >> Thanks,
> >> > > >> >> >> >> --Konst
> >> > > >> >> >> >>
> >> > > >> >> >> >> On Thu, Feb 28, 2013 at 8:47 PM, Eric Baldeschwieler
> >> > > >> >> >> >> <er...@hortonworks.com> wrote:
> >> > > >> >> >> >> > +1 (non-binding)
> >> > > >> >> >> >> >
> >> > > >> >> >> >> > A few of observations:
> >> > > >> >> >> >> >
> >> > > >> >> >> >> > - Windows has actually been a supported platform for
> >> Hadoop
> >> > > >> >> >> >> > since
> >> > > >> >> >> >> > 0.1
> >> > > >> >> >> >> > .
> >> > > >> >> >> >> > Doug championed supporting windows then and we've
> >> continued
> >> > > to
> >> > > >> >> >> >> > do
> >> > > >> >> >> >> > it
> >> > > >> >> >> >> > with
> >> > > >> >> >> >> > varying vigor over time.  To my knowledge we've never
> >> made a
> >> > > >> >> >> >> > decision
> >> > > >> >> >> >> > to
> >> > > >> >> >> >> > drop windows support.  The change here is improving
> our
> >> > > support
> >> > > >> >> >> >> > and
> >> > > >> >> >> >> > dropping
> >> > > >> >> >> >> > the requirement of cigwin.  We had Nutch windows users
> >> on the
> >> > > >> >> >> >> > list
> >> > > >> >> >> >> > in
> >> > > >> >> >> >> > 2006
> >> > > >> >> >> >> > and we've been supporting windows FS requirements
> since
> >> > > >> >> >> >> > inception.
> >> > > >> >> >> >> >
> >> > > >> >> >> >> > - A little pragmatism will go a long way.  As a
> community
> >> > > we've
> >> > > >> >> >> >> > got
> >> > > >> >> >> >> > to
> >> > > >> >> >> >> > stay committed to keeping hadoop simple (so it does
> work
> >> on
> >> > > >> >> >> >> > many
> >> > > >> >> >> >> > platforms)
> >> > > >> >> >> >> > and extending it to take advantage of key emerging
> >> > > OS/hardware
> >> > > >> >> >> >> > features,
> >> > > >> >> >> >> > such as containers, new FSs, virtualization, flash ...
> >>  We
> >> > > >> >> >> >> > should
> >> > > >> >> >> >> > all
> >> > > >> >> >> >> > plan
> >> > > >> >> >> >> > to let new features & optimizations emerge that don't
> >> work
> >> > > >> >> >> >> > everywhere, if
> >> > > >> >> >> >> > they are compelling and central to hadoop's mission of
> >> being
> >> > > >> >> >> >> > THE
> >> > > >> >> >> >> > best
> >> > > >> >> >> >> > fabric
> >> > > >> >> >> >> > for storing and processing big data.
> >> > > >> >> >> >> >
> >> > > >> >> >> >> > - A UI project like KDE has to deal with the MANY
> >> differences
> >> > > >> >> >> >> > between
> >> > > >> >> >> >> > windows and linux UI APIs.  Hadoop faces no such
> complex
> >> > > >> >> >> >> > challenge
> >> > > >> >> >> >> > and hence
> >> > > >> >> >> >> > can be maintained from a single codeline IMO.  It is
> >> mostly
> >> > > >> >> >> >> > abstracted from
> >> > > >> >> >> >> > the OS APIs via Java and our design choices.  Where it
> >> is not
> >> > > >> >> >> >> > we
> >> > > >> >> >> >> > can
> >> > > >> >> >> >> > continue to add plugable abstractions.
> >> > > >> >> >> >> >
> >> > > >> >> >> >
> >> > > >> >> >> >
> >> > > >> >> >
> >> > > >> >> >
> >> > > >> >
> >> > > >> >
> >> > > >
> >> > > >
> >> > >
> >>
> >
> >
> >
> > --
> > Best regards,
> >
> >    - Andy
> >
> > Problems worthy of attack prove their worth by hitting back. - Piet Hein
> > (via Tom White)
>



-- 
Best regards,

   - Andy

Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)

Re: [Vote] Merge branch-trunk-win to trunk

[Andrew Purtell <ap...@apache.org>]

Sorry, that was my error selecting the wrong reply option.


On Mon, Mar 25, 2013 at 10:25 PM, Konstantin Shvachko
<sh...@gmail.com>wrote:

> Andrew, this used to be on all -dev lists. Let's keep it that way.
>
> To the point.
> Does this mean that people are silently porting windows changes to
> branch-2?
> New features on a branch should be voted first, no?
>
> Thanks,
> --Konstantin
>
>
> On Mon, Mar 25, 2013 at 1:36 PM, Andrew Purtell <ap...@apache.org>
> wrote:
> > Noticed this too. Simply a 'public' modifier is missing, but it's unclear
> > how this could not have been caught prior to check-in.
> >
> >
> > On Mon, Mar 25, 2013 at 9:17 PM, Konstantin Boudnik <co...@apache.org>
> wrote:
> >
> >> It doesn't look like any progress has been done on the ticket below in
> the
> >> last 3 weeks. And now branch-2 can't be compiled because of
> >>
> >>
> >>
> hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSShell.java:[895,15]
> >> WINDOWS is not public in org.apache.hadoop.fs.Path; cannot be accessed
> from
> >> outside package
> >>
> >> That's exactly why I was -1'ing this...
> >>   Cos
> >>
> >> On Mon, Mar 04, 2013 at 05:41PM, Matt Foley wrote:
> >> > Thanks, gentlemen.  I've opened and taken responsibility for
> >> > https://issues.apache.org/jira/browse/HADOOP-9359.  Giri Kesavan has
> >> agreed
> >> > to help with the parts that require Jenkins admin access.
> >> >
> >> > Thanks,
> >> > --Matt
> >> >
> >> >
> >> >
> >> > On Mon, Mar 4, 2013 at 5:00 PM, Konstantin Shvachko <
> >> shv.hadoop@gmail.com>wrote:
> >> >
> >> > > +1 on the merge.
> >> > >
> >> > > I am glad we agreed.
> >> > > Having Jira to track the CI effort is a good idea.
> >> > >
> >> > > Thanks,
> >> > > --Konstantin
> >> > >
> >> > > On Mon, Mar 4, 2013 at 3:29 PM, Matt Foley <mf...@hortonworks.com>
> >> wrote:
> >> > > > Thanks.  I agree Windows -1's in test-patch should not block
> commits.
> >> > > >
> >> > > > --Matt
> >> > > >
> >> > > >
> >> > > >
> >> > > > On Mon, Mar 4, 2013 at 2:30 PM, Konstantin Shvachko <
> >> > > shv.hadoop@gmail.com>
> >> > > > wrote:
> >> > > >>
> >> > > >> On Mon, Mar 4, 2013 at 12:22 PM, Matt Foley <
> mfoley@hortonworks.com
> >> >
> >> > > >> wrote:
> >> > > >> > Konstantine, you have voted -1, and stated some requirements
> >> before
> >> > > >> > you'll
> >> > > >> > withdraw that -1.  As I plan to do work to fulfill those
> >> > > requirements, I
> >> > > >> > want to make sure that what I'm proposing will, in fact,
> satisfy
> >> you.
> >> > > >> > That's why I'm asking, if we implement full "test-patch"
> >> integration
> >> > > for
> >> > > >> > Windows, does it seem to you that that would provide adequate
> >> support?
> >> > > >>
> >> > > >> Yes.
> >> > > >>
> >> > > >> > I have learned not to presume that my interpretation is
> correct.
> >>  My
> >> > > >> > interpretation of item #1 is that test-patch provides
> pre-commit
> >> > > build,
> >> > > >> > so
> >> > > >> > it would satisfy item #1.  But rather than assuming that I am
> >> > > >> > interpreting
> >> > > >> > it correctly, I simply want your agreement that it would, or if
> >> not,
> >> > > >> > clarification why it won't.
> >> > > >>
> >> > > >> I agree it will satisfy my item #1.
> >> > > >> I did not agree in my previous email, but I changed my mind
> based on
> >> > > >> the latest discussion. I have to explain why now.
> >> > > >> I was proposing nightly build because I did not want pre-commit
> >> build
> >> > > >> for Windows block commits to Linux. But if people are fine just
> >> ignoring
> >> > > >> -1s for the Windows part of the build it should be good.
> >> > > >>
> >> > > >> > Regarding item #2, it is also my interpretation that test-patch
> >> > > provides
> >> > > >> > an
> >> > > >> > on-demand (perhaps 20-minutes deferred) Jenkins build and unit
> >> test,
> >> > > >> > with
> >> > > >> > logs available to the developer, so it would satisfy item #2.
>  But
> >> > > >> > rather
> >> > > >> > than assuming that I am interpreting it correctly, I simply
> want
> >> your
> >> > > >> > agreement that it would, or if not, clarification why it won't.
> >> > > >>
> >> > > >> It will satisfy my item #2 in the following way:
> >> > > >> I can duplicate your pre-commit build for Windows and add an
> input
> >> > > >> parameter, which would let people run the build on their patches
> >> > > >> chosen from local machine rather than attaching them to Jiras.
> >> > > >>
> >> > > >> Thanks,
> >> > > >> --Konstantin
> >> > > >>
> >> > > >> > In agile terms, you are the Owner of these requirements.
>  Please
> >> give
> >> > > me
> >> > > >> > owner feedback as to whether my proposed work sounds like it
> will
> >> > > >> > satisfy
> >> > > >> > the requirements.
> >> > > >> >
> >> > > >> > Thank you,
> >> > > >> > --Matt
> >> > > >> >
> >> > > >> >
> >> > > >> > On Sun, Mar 3, 2013 at 12:16 PM, Konstantin Shvachko
> >> > > >> > <sh...@gmail.com>
> >> > > >> > wrote:
> >> > > >> >>
> >> > > >> >> Didn't I explain in details what I am asking for?
> >> > > >> >>
> >> > > >> >> Thanks,
> >> > > >> >> --Konst
> >> > > >> >>
> >> > > >> >> On Sun, Mar 3, 2013 at 11:08 AM, Matt Foley <
> >> mfoley@hortonworks.com>
> >> > > >> >> wrote:
> >> > > >> >> > Hi Konstantin,
> >> > > >> >> > I'd like to point out two things:
> >> > > >> >> > First, I already committed in this thread (email of Thu, Feb
> >> 28,
> >> > > 2013
> >> > > >> >> > at
> >> > > >> >> > 6:01 PM) to providing CI for Windows builds.  So please stop
> >> acting
> >> > > >> >> > like
> >> > > >> >> > I'm
> >> > > >> >> > resisting this idea or something.
> >> > > >> >> > Second, you didn't answer my question, you just kvetched
> about
> >> the
> >> > > >> >> > phrasing.
> >> > > >> >> > So I ask again:
> >> > > >> >> >
> >> > > >> >> > Will providing full "test-patch" integration (pre-commit
> build
> >> and
> >> > > >> >> > unit
> >> > > >> >> > test
> >> > > >> >> > triggered by Jira "Patch Available" state) satisfy your
> >> request for
> >> > > >> >> > functionality #1 and #2?  Yes or no, please.
> >> > > >> >> >
> >> > > >> >> > Thanks,
> >> > > >> >> > --Matt
> >> > > >> >> >
> >> > > >> >> >
> >> > > >> >> > On Sat, Mar 2, 2013 at 7:32 PM, Konstantin Shvachko
> >> > > >> >> > <sh...@gmail.com>
> >> > > >> >> > wrote:
> >> > > >> >> >>
> >> > > >> >> >> Hi Matt,
> >> > > >> >> >>
> >> > > >> >> >> On Sat, Mar 2, 2013 at 12:32 PM, Matt Foley <
> >> > > mfoley@hortonworks.com>
> >> > > >> >> >> wrote:
> >> > > >> >> >> > Konstantin,
> >> > > >> >> >> > I would like to explore what it would take to remove this
> >> > > >> >> >> > perceived
> >> > > >> >> >> > impediment --
> >> > > >> >> >>
> >> > > >> >> >> Glad you decided to explore. Thank you.
> >> > > >> >> >>
> >> > > >> >> >> > although I reserve the right to argue that this is not
> >> > > >> >> >> > pre-requisite to merging the cross-platform support
> patch.
> >> > > >> >> >>
> >> > > >> >> >> It's your right indeed. So as mine to question what the
> >> platform
> >> > > >> >> >> support means for you, which I believe remained unclear.
> >> > > >> >> >> I do not impede the change as you should have noticed. My
> >> > > >> >> >> requirement
> >> > > >> >> >> comes from my perception of the support, which means to me
> >> exactly
> >> > > >> >> >> two
> >> > > >> >> >> things:
> >> > > >> >> >> 1. The ability to recognise the code is broken for the
> >> platform
> >> > > >> >> >> 2. The ability to test new patches on the platform
> >> > > >> >> >> The latter is problematic, as many noticed in this thread,
> for
> >> > > those
> >> > > >> >> >> whose customary environment does not include Windows.
> >> > > >> >> >>
> >> > > >> >> >> > If we implemented full "test-patch" support for Windows
> on
> >> > > trunk,
> >> > > >> >> >> > would
> >> > > >> >> >> > that
> >> > > >> >> >> > fulfill both your items #1 and #2?  Please note that:
> >> > > >> >> >> > a) Pushing the "Patch Available" button in Jira shall
> cause
> >> a
> >> > > >> >> >> > pre-commit
> >> > > >> >> >> > build to start within, I believe, 20 minutes.
> >> > > >> >> >> > b) That build keeps logs for both java build and unit
> tests
> >> for
> >> > > >> >> >> > several
> >> > > >> >> >> > days, that are accessible to all viewers.
> >> > > >> >> >>
> >> > > >> >> >> In item #1 I mostly asking for the nightly build, which is
> >> simpler
> >> > > >> >> >> than "test-patch". The latter would be ideal from the
> platform
> >> > > >> >> >> support
> >> > > >> >> >> viewpoint, but it is for the community to decide if we want
> >> to add
> >> > > >> >> >> extra +3 hours to the build.
> >> > > >> >> >> Nightly build in my understanding is triggered by the timer
> >> rather
> >> > > >> >> >> than by Jira's "submit patch" button.  On Jenkins build
> >> > > >> >> >> configuration
> >> > > >> >> >> you can specify it under "Build periodically".
> >> > > >> >> >>
> >> > > >> >> >> > So, does this provide sufficient on-demand support that
> we
> >> don't
> >> > > >> >> >> > have
> >> > > >> >> >> > to
> >> > > >> >> >> > implement a whole new on-demand VM support structure of
> some
> >> > > sort
> >> > > >> >> >> > for
> >> > > >> >> >> > #2
> >> > > >> >> >> > (which would be an extraordinary and impractical demand)?
> >> > > >> >> >>
> >> > > >> >> >> I did not mention VMs. Item #2 means a build, which runs
> >> > > >> >> >> "test-patch"
> >> > > >> >> >> target with the file specified by a user (instead of a jira
> >> > > >> >> >> attachment).
> >> > > >> >> >> When user clicks "Build Now" link a box is displayed where
> the
> >> > > user
> >> > > >> >> >> can enter the file path containing the patch. This can be
> >> > > specified
> >> > > >> >> >> in
> >> > > >> >> >> the Build Configuration under "This build is
> parameterized" by
> >> > > >> >> >> choosing AddParameter / FileParameter. The build can run on
> >> the
> >> > > same
> >> > > >> >> >> Windows machine as the nightly build.
> >> > > >> >> >> Such build will let people test their patches for Windows
> on
> >> > > Jenkins
> >> > > >> >> >> if they don't posses a license for the right version of
> >> Windows.
> >> > > >> >> >> I hope this will not turn into extraordinary or impractical
> >> > > effort.
> >> > > >> >> >>
> >> > > >> >> >> Thanks,
> >> > > >> >> >> --Konst
> >> > > >> >> >>
> >> > > >> >> >> > Thanks,
> >> > > >> >> >> > --Matt
> >> > > >> >> >> >
> >> > > >> >> >> >
> >> > > >> >> >> > On Fri, Mar 1, 2013 at 12:18 PM, Konstantin Shvachko
> >> > > >> >> >> > <sh...@gmail.com>
> >> > > >> >> >> > wrote:
> >> > > >> >> >> >>
> >> > > >> >> >> >> -1
> >> > > >> >> >> >> We should have a CI infrastructure in place before we
> can
> >> > > commit
> >> > > >> >> >> >> to
> >> > > >> >> >> >> supporting Windows platform.
> >> > > >> >> >> >>
> >> > > >> >> >> >> Eric is right Win/Cygwin was supported since day one.
> >> > > >> >> >> >> I had a Windows box under my desk running nightly builds
> >> back
> >> > > in
> >> > > >> >> >> >> 2006-07.
> >> > > >> >> >> >> People were irritated but I was filing windows bugs
> until
> >> 0.22
> >> > > >> >> >> >> release.
> >> > > >> >> >> >> Times changing and I am glad to see wider support for
> Win
> >> > > >> >> >> >> platform.
> >> > > >> >> >> >>
> >> > > >> >> >> >> But in order to make it work you guys need to put the CI
> >> > > process
> >> > > >> >> >> >> in
> >> > > >> >> >> >> place
> >> > > >> >> >> >>
> >> > > >> >> >> >> 1. windows jenkins build: could be nightly or PreCommit.
> >> > > >> >> >> >> - Nightly would mean that changes can be committed to
> trunk
> >> > > based
> >> > > >> >> >> >> on
> >> > > >> >> >> >> linux PreCommit build. And people will file bugs if the
> >> change
> >> > > >> >> >> >> broke
> >> > > >> >> >> >> Windows nightly build.
> >> > > >> >> >> >> - PreCommit-win build will mean automatic reporting
> failed
> >> > > tests
> >> > > >> >> >> >> to
> >> > > >> >> >> >> respective jira blocking commits the same way as it is
> now
> >> with
> >> > > >> >> >> >> linux
> >> > > >> >> >> >> PreCommit builds.
> >> > > >> >> >> >> We should discuss which way is more efficient for
> >> developers.
> >> > > >> >> >> >>
> >> > > >> >> >> >> 2. On-demand-windows Jenkins build.
> >> > > >> >> >> >> I see it as a build to which I can attach my patch and
> the
> >> > > build
> >> > > >> >> >> >> will
> >> > > >> >> >> >> run my changes on a dedicated windows box.
> >> > > >> >> >> >> That way people can test their changes without having
> >> personal
> >> > > >> >> >> >> windows
> >> > > >> >> >> >> nodes.
> >> > > >> >> >> >>
> >> > > >> >> >> >> I think this is the minimal set of requirement for us
> to be
> >> > > able
> >> > > >> >> >> >> to
> >> > > >> >> >> >> commit to the new platform.
> >> > > >> >> >> >> Right now I see only one windows related build
> >> > > >> >> >> >> https://builds.apache.org/view/Hadoop/job/Hadoop-1-win/
> >> > > >> >> >> >> Which was failing since Sept 8, 2012 and did not run in
> the
> >> > > last
> >> > > >> >> >> >> month.
> >> > > >> >> >> >>
> >> > > >> >> >> >> Thanks,
> >> > > >> >> >> >> --Konst
> >> > > >> >> >> >>
> >> > > >> >> >> >> On Thu, Feb 28, 2013 at 8:47 PM, Eric Baldeschwieler
> >> > > >> >> >> >> <er...@hortonworks.com> wrote:
> >> > > >> >> >> >> > +1 (non-binding)
> >> > > >> >> >> >> >
> >> > > >> >> >> >> > A few of observations:
> >> > > >> >> >> >> >
> >> > > >> >> >> >> > - Windows has actually been a supported platform for
> >> Hadoop
> >> > > >> >> >> >> > since
> >> > > >> >> >> >> > 0.1
> >> > > >> >> >> >> > .
> >> > > >> >> >> >> > Doug championed supporting windows then and we've
> >> continued
> >> > > to
> >> > > >> >> >> >> > do
> >> > > >> >> >> >> > it
> >> > > >> >> >> >> > with
> >> > > >> >> >> >> > varying vigor over time.  To my knowledge we've never
> >> made a
> >> > > >> >> >> >> > decision
> >> > > >> >> >> >> > to
> >> > > >> >> >> >> > drop windows support.  The change here is improving
> our
> >> > > support
> >> > > >> >> >> >> > and
> >> > > >> >> >> >> > dropping
> >> > > >> >> >> >> > the requirement of cigwin.  We had Nutch windows users
> >> on the
> >> > > >> >> >> >> > list
> >> > > >> >> >> >> > in
> >> > > >> >> >> >> > 2006
> >> > > >> >> >> >> > and we've been supporting windows FS requirements
> since
> >> > > >> >> >> >> > inception.
> >> > > >> >> >> >> >
> >> > > >> >> >> >> > - A little pragmatism will go a long way.  As a
> community
> >> > > we've
> >> > > >> >> >> >> > got
> >> > > >> >> >> >> > to
> >> > > >> >> >> >> > stay committed to keeping hadoop simple (so it does
> work
> >> on
> >> > > >> >> >> >> > many
> >> > > >> >> >> >> > platforms)
> >> > > >> >> >> >> > and extending it to take advantage of key emerging
> >> > > OS/hardware
> >> > > >> >> >> >> > features,
> >> > > >> >> >> >> > such as containers, new FSs, virtualization, flash ...
> >>  We
> >> > > >> >> >> >> > should
> >> > > >> >> >> >> > all
> >> > > >> >> >> >> > plan
> >> > > >> >> >> >> > to let new features & optimizations emerge that don't
> >> work
> >> > > >> >> >> >> > everywhere, if
> >> > > >> >> >> >> > they are compelling and central to hadoop's mission of
> >> being
> >> > > >> >> >> >> > THE
> >> > > >> >> >> >> > best
> >> > > >> >> >> >> > fabric
> >> > > >> >> >> >> > for storing and processing big data.
> >> > > >> >> >> >> >
> >> > > >> >> >> >> > - A UI project like KDE has to deal with the MANY
> >> differences
> >> > > >> >> >> >> > between
> >> > > >> >> >> >> > windows and linux UI APIs.  Hadoop faces no such
> complex
> >> > > >> >> >> >> > challenge
> >> > > >> >> >> >> > and hence
> >> > > >> >> >> >> > can be maintained from a single codeline IMO.  It is
> >> mostly
> >> > > >> >> >> >> > abstracted from
> >> > > >> >> >> >> > the OS APIs via Java and our design choices.  Where it
> >> is not
> >> > > >> >> >> >> > we
> >> > > >> >> >> >> > can
> >> > > >> >> >> >> > continue to add plugable abstractions.
> >> > > >> >> >> >> >
> >> > > >> >> >> >
> >> > > >> >> >> >
> >> > > >> >> >
> >> > > >> >> >
> >> > > >> >
> >> > > >> >
> >> > > >
> >> > > >
> >> > >
> >>
> >
> >
> >
> > --
> > Best regards,
> >
> >    - Andy
> >
> > Problems worthy of attack prove their worth by hitting back. - Piet Hein
> > (via Tom White)
>



-- 
Best regards,

   - Andy

Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)

Re: [Vote] Merge branch-trunk-win to trunk

[Konstantin Shvachko <sh...@gmail.com>]

Andrew, this used to be on all -dev lists. Let's keep it that way.

To the point.
Does this mean that people are silently porting windows changes to branch-2?
New features on a branch should be voted first, no?

Thanks,
--Konstantin


On Mon, Mar 25, 2013 at 1:36 PM, Andrew Purtell <ap...@apache.org> wrote:
> Noticed this too. Simply a 'public' modifier is missing, but it's unclear
> how this could not have been caught prior to check-in.
>
>
> On Mon, Mar 25, 2013 at 9:17 PM, Konstantin Boudnik <co...@apache.org> wrote:
>
>> It doesn't look like any progress has been done on the ticket below in the
>> last 3 weeks. And now branch-2 can't be compiled because of
>>
>>
>> hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSShell.java:[895,15]
>> WINDOWS is not public in org.apache.hadoop.fs.Path; cannot be accessed from
>> outside package
>>
>> That's exactly why I was -1'ing this...
>>   Cos
>>
>> On Mon, Mar 04, 2013 at 05:41PM, Matt Foley wrote:
>> > Thanks, gentlemen.  I've opened and taken responsibility for
>> > https://issues.apache.org/jira/browse/HADOOP-9359.  Giri Kesavan has
>> agreed
>> > to help with the parts that require Jenkins admin access.
>> >
>> > Thanks,
>> > --Matt
>> >
>> >
>> >
>> > On Mon, Mar 4, 2013 at 5:00 PM, Konstantin Shvachko <
>> shv.hadoop@gmail.com>wrote:
>> >
>> > > +1 on the merge.
>> > >
>> > > I am glad we agreed.
>> > > Having Jira to track the CI effort is a good idea.
>> > >
>> > > Thanks,
>> > > --Konstantin
>> > >
>> > > On Mon, Mar 4, 2013 at 3:29 PM, Matt Foley <mf...@hortonworks.com>
>> wrote:
>> > > > Thanks.  I agree Windows -1's in test-patch should not block commits.
>> > > >
>> > > > --Matt
>> > > >
>> > > >
>> > > >
>> > > > On Mon, Mar 4, 2013 at 2:30 PM, Konstantin Shvachko <
>> > > shv.hadoop@gmail.com>
>> > > > wrote:
>> > > >>
>> > > >> On Mon, Mar 4, 2013 at 12:22 PM, Matt Foley <mfoley@hortonworks.com
>> >
>> > > >> wrote:
>> > > >> > Konstantine, you have voted -1, and stated some requirements
>> before
>> > > >> > you'll
>> > > >> > withdraw that -1.  As I plan to do work to fulfill those
>> > > requirements, I
>> > > >> > want to make sure that what I'm proposing will, in fact, satisfy
>> you.
>> > > >> > That's why I'm asking, if we implement full "test-patch"
>> integration
>> > > for
>> > > >> > Windows, does it seem to you that that would provide adequate
>> support?
>> > > >>
>> > > >> Yes.
>> > > >>
>> > > >> > I have learned not to presume that my interpretation is correct.
>>  My
>> > > >> > interpretation of item #1 is that test-patch provides pre-commit
>> > > build,
>> > > >> > so
>> > > >> > it would satisfy item #1.  But rather than assuming that I am
>> > > >> > interpreting
>> > > >> > it correctly, I simply want your agreement that it would, or if
>> not,
>> > > >> > clarification why it won't.
>> > > >>
>> > > >> I agree it will satisfy my item #1.
>> > > >> I did not agree in my previous email, but I changed my mind based on
>> > > >> the latest discussion. I have to explain why now.
>> > > >> I was proposing nightly build because I did not want pre-commit
>> build
>> > > >> for Windows block commits to Linux. But if people are fine just
>> ignoring
>> > > >> -1s for the Windows part of the build it should be good.
>> > > >>
>> > > >> > Regarding item #2, it is also my interpretation that test-patch
>> > > provides
>> > > >> > an
>> > > >> > on-demand (perhaps 20-minutes deferred) Jenkins build and unit
>> test,
>> > > >> > with
>> > > >> > logs available to the developer, so it would satisfy item #2.  But
>> > > >> > rather
>> > > >> > than assuming that I am interpreting it correctly, I simply want
>> your
>> > > >> > agreement that it would, or if not, clarification why it won't.
>> > > >>
>> > > >> It will satisfy my item #2 in the following way:
>> > > >> I can duplicate your pre-commit build for Windows and add an input
>> > > >> parameter, which would let people run the build on their patches
>> > > >> chosen from local machine rather than attaching them to Jiras.
>> > > >>
>> > > >> Thanks,
>> > > >> --Konstantin
>> > > >>
>> > > >> > In agile terms, you are the Owner of these requirements.  Please
>> give
>> > > me
>> > > >> > owner feedback as to whether my proposed work sounds like it will
>> > > >> > satisfy
>> > > >> > the requirements.
>> > > >> >
>> > > >> > Thank you,
>> > > >> > --Matt
>> > > >> >
>> > > >> >
>> > > >> > On Sun, Mar 3, 2013 at 12:16 PM, Konstantin Shvachko
>> > > >> > <sh...@gmail.com>
>> > > >> > wrote:
>> > > >> >>
>> > > >> >> Didn't I explain in details what I am asking for?
>> > > >> >>
>> > > >> >> Thanks,
>> > > >> >> --Konst
>> > > >> >>
>> > > >> >> On Sun, Mar 3, 2013 at 11:08 AM, Matt Foley <
>> mfoley@hortonworks.com>
>> > > >> >> wrote:
>> > > >> >> > Hi Konstantin,
>> > > >> >> > I'd like to point out two things:
>> > > >> >> > First, I already committed in this thread (email of Thu, Feb
>> 28,
>> > > 2013
>> > > >> >> > at
>> > > >> >> > 6:01 PM) to providing CI for Windows builds.  So please stop
>> acting
>> > > >> >> > like
>> > > >> >> > I'm
>> > > >> >> > resisting this idea or something.
>> > > >> >> > Second, you didn't answer my question, you just kvetched about
>> the
>> > > >> >> > phrasing.
>> > > >> >> > So I ask again:
>> > > >> >> >
>> > > >> >> > Will providing full "test-patch" integration (pre-commit build
>> and
>> > > >> >> > unit
>> > > >> >> > test
>> > > >> >> > triggered by Jira "Patch Available" state) satisfy your
>> request for
>> > > >> >> > functionality #1 and #2?  Yes or no, please.
>> > > >> >> >
>> > > >> >> > Thanks,
>> > > >> >> > --Matt
>> > > >> >> >
>> > > >> >> >
>> > > >> >> > On Sat, Mar 2, 2013 at 7:32 PM, Konstantin Shvachko
>> > > >> >> > <sh...@gmail.com>
>> > > >> >> > wrote:
>> > > >> >> >>
>> > > >> >> >> Hi Matt,
>> > > >> >> >>
>> > > >> >> >> On Sat, Mar 2, 2013 at 12:32 PM, Matt Foley <
>> > > mfoley@hortonworks.com>
>> > > >> >> >> wrote:
>> > > >> >> >> > Konstantin,
>> > > >> >> >> > I would like to explore what it would take to remove this
>> > > >> >> >> > perceived
>> > > >> >> >> > impediment --
>> > > >> >> >>
>> > > >> >> >> Glad you decided to explore. Thank you.
>> > > >> >> >>
>> > > >> >> >> > although I reserve the right to argue that this is not
>> > > >> >> >> > pre-requisite to merging the cross-platform support patch.
>> > > >> >> >>
>> > > >> >> >> It's your right indeed. So as mine to question what the
>> platform
>> > > >> >> >> support means for you, which I believe remained unclear.
>> > > >> >> >> I do not impede the change as you should have noticed. My
>> > > >> >> >> requirement
>> > > >> >> >> comes from my perception of the support, which means to me
>> exactly
>> > > >> >> >> two
>> > > >> >> >> things:
>> > > >> >> >> 1. The ability to recognise the code is broken for the
>> platform
>> > > >> >> >> 2. The ability to test new patches on the platform
>> > > >> >> >> The latter is problematic, as many noticed in this thread, for
>> > > those
>> > > >> >> >> whose customary environment does not include Windows.
>> > > >> >> >>
>> > > >> >> >> > If we implemented full "test-patch" support for Windows on
>> > > trunk,
>> > > >> >> >> > would
>> > > >> >> >> > that
>> > > >> >> >> > fulfill both your items #1 and #2?  Please note that:
>> > > >> >> >> > a) Pushing the "Patch Available" button in Jira shall cause
>> a
>> > > >> >> >> > pre-commit
>> > > >> >> >> > build to start within, I believe, 20 minutes.
>> > > >> >> >> > b) That build keeps logs for both java build and unit tests
>> for
>> > > >> >> >> > several
>> > > >> >> >> > days, that are accessible to all viewers.
>> > > >> >> >>
>> > > >> >> >> In item #1 I mostly asking for the nightly build, which is
>> simpler
>> > > >> >> >> than "test-patch". The latter would be ideal from the platform
>> > > >> >> >> support
>> > > >> >> >> viewpoint, but it is for the community to decide if we want
>> to add
>> > > >> >> >> extra +3 hours to the build.
>> > > >> >> >> Nightly build in my understanding is triggered by the timer
>> rather
>> > > >> >> >> than by Jira's "submit patch" button.  On Jenkins build
>> > > >> >> >> configuration
>> > > >> >> >> you can specify it under "Build periodically".
>> > > >> >> >>
>> > > >> >> >> > So, does this provide sufficient on-demand support that we
>> don't
>> > > >> >> >> > have
>> > > >> >> >> > to
>> > > >> >> >> > implement a whole new on-demand VM support structure of some
>> > > sort
>> > > >> >> >> > for
>> > > >> >> >> > #2
>> > > >> >> >> > (which would be an extraordinary and impractical demand)?
>> > > >> >> >>
>> > > >> >> >> I did not mention VMs. Item #2 means a build, which runs
>> > > >> >> >> "test-patch"
>> > > >> >> >> target with the file specified by a user (instead of a jira
>> > > >> >> >> attachment).
>> > > >> >> >> When user clicks "Build Now" link a box is displayed where the
>> > > user
>> > > >> >> >> can enter the file path containing the patch. This can be
>> > > specified
>> > > >> >> >> in
>> > > >> >> >> the Build Configuration under "This build is parameterized" by
>> > > >> >> >> choosing AddParameter / FileParameter. The build can run on
>> the
>> > > same
>> > > >> >> >> Windows machine as the nightly build.
>> > > >> >> >> Such build will let people test their patches for Windows on
>> > > Jenkins
>> > > >> >> >> if they don't posses a license for the right version of
>> Windows.
>> > > >> >> >> I hope this will not turn into extraordinary or impractical
>> > > effort.
>> > > >> >> >>
>> > > >> >> >> Thanks,
>> > > >> >> >> --Konst
>> > > >> >> >>
>> > > >> >> >> > Thanks,
>> > > >> >> >> > --Matt
>> > > >> >> >> >
>> > > >> >> >> >
>> > > >> >> >> > On Fri, Mar 1, 2013 at 12:18 PM, Konstantin Shvachko
>> > > >> >> >> > <sh...@gmail.com>
>> > > >> >> >> > wrote:
>> > > >> >> >> >>
>> > > >> >> >> >> -1
>> > > >> >> >> >> We should have a CI infrastructure in place before we can
>> > > commit
>> > > >> >> >> >> to
>> > > >> >> >> >> supporting Windows platform.
>> > > >> >> >> >>
>> > > >> >> >> >> Eric is right Win/Cygwin was supported since day one.
>> > > >> >> >> >> I had a Windows box under my desk running nightly builds
>> back
>> > > in
>> > > >> >> >> >> 2006-07.
>> > > >> >> >> >> People were irritated but I was filing windows bugs until
>> 0.22
>> > > >> >> >> >> release.
>> > > >> >> >> >> Times changing and I am glad to see wider support for Win
>> > > >> >> >> >> platform.
>> > > >> >> >> >>
>> > > >> >> >> >> But in order to make it work you guys need to put the CI
>> > > process
>> > > >> >> >> >> in
>> > > >> >> >> >> place
>> > > >> >> >> >>
>> > > >> >> >> >> 1. windows jenkins build: could be nightly or PreCommit.
>> > > >> >> >> >> - Nightly would mean that changes can be committed to trunk
>> > > based
>> > > >> >> >> >> on
>> > > >> >> >> >> linux PreCommit build. And people will file bugs if the
>> change
>> > > >> >> >> >> broke
>> > > >> >> >> >> Windows nightly build.
>> > > >> >> >> >> - PreCommit-win build will mean automatic reporting failed
>> > > tests
>> > > >> >> >> >> to
>> > > >> >> >> >> respective jira blocking commits the same way as it is now
>> with
>> > > >> >> >> >> linux
>> > > >> >> >> >> PreCommit builds.
>> > > >> >> >> >> We should discuss which way is more efficient for
>> developers.
>> > > >> >> >> >>
>> > > >> >> >> >> 2. On-demand-windows Jenkins build.
>> > > >> >> >> >> I see it as a build to which I can attach my patch and the
>> > > build
>> > > >> >> >> >> will
>> > > >> >> >> >> run my changes on a dedicated windows box.
>> > > >> >> >> >> That way people can test their changes without having
>> personal
>> > > >> >> >> >> windows
>> > > >> >> >> >> nodes.
>> > > >> >> >> >>
>> > > >> >> >> >> I think this is the minimal set of requirement for us to be
>> > > able
>> > > >> >> >> >> to
>> > > >> >> >> >> commit to the new platform.
>> > > >> >> >> >> Right now I see only one windows related build
>> > > >> >> >> >> https://builds.apache.org/view/Hadoop/job/Hadoop-1-win/
>> > > >> >> >> >> Which was failing since Sept 8, 2012 and did not run in the
>> > > last
>> > > >> >> >> >> month.
>> > > >> >> >> >>
>> > > >> >> >> >> Thanks,
>> > > >> >> >> >> --Konst
>> > > >> >> >> >>
>> > > >> >> >> >> On Thu, Feb 28, 2013 at 8:47 PM, Eric Baldeschwieler
>> > > >> >> >> >> <er...@hortonworks.com> wrote:
>> > > >> >> >> >> > +1 (non-binding)
>> > > >> >> >> >> >
>> > > >> >> >> >> > A few of observations:
>> > > >> >> >> >> >
>> > > >> >> >> >> > - Windows has actually been a supported platform for
>> Hadoop
>> > > >> >> >> >> > since
>> > > >> >> >> >> > 0.1
>> > > >> >> >> >> > .
>> > > >> >> >> >> > Doug championed supporting windows then and we've
>> continued
>> > > to
>> > > >> >> >> >> > do
>> > > >> >> >> >> > it
>> > > >> >> >> >> > with
>> > > >> >> >> >> > varying vigor over time.  To my knowledge we've never
>> made a
>> > > >> >> >> >> > decision
>> > > >> >> >> >> > to
>> > > >> >> >> >> > drop windows support.  The change here is improving our
>> > > support
>> > > >> >> >> >> > and
>> > > >> >> >> >> > dropping
>> > > >> >> >> >> > the requirement of cigwin.  We had Nutch windows users
>> on the
>> > > >> >> >> >> > list
>> > > >> >> >> >> > in
>> > > >> >> >> >> > 2006
>> > > >> >> >> >> > and we've been supporting windows FS requirements since
>> > > >> >> >> >> > inception.
>> > > >> >> >> >> >
>> > > >> >> >> >> > - A little pragmatism will go a long way.  As a community
>> > > we've
>> > > >> >> >> >> > got
>> > > >> >> >> >> > to
>> > > >> >> >> >> > stay committed to keeping hadoop simple (so it does work
>> on
>> > > >> >> >> >> > many
>> > > >> >> >> >> > platforms)
>> > > >> >> >> >> > and extending it to take advantage of key emerging
>> > > OS/hardware
>> > > >> >> >> >> > features,
>> > > >> >> >> >> > such as containers, new FSs, virtualization, flash ...
>>  We
>> > > >> >> >> >> > should
>> > > >> >> >> >> > all
>> > > >> >> >> >> > plan
>> > > >> >> >> >> > to let new features & optimizations emerge that don't
>> work
>> > > >> >> >> >> > everywhere, if
>> > > >> >> >> >> > they are compelling and central to hadoop's mission of
>> being
>> > > >> >> >> >> > THE
>> > > >> >> >> >> > best
>> > > >> >> >> >> > fabric
>> > > >> >> >> >> > for storing and processing big data.
>> > > >> >> >> >> >
>> > > >> >> >> >> > - A UI project like KDE has to deal with the MANY
>> differences
>> > > >> >> >> >> > between
>> > > >> >> >> >> > windows and linux UI APIs.  Hadoop faces no such complex
>> > > >> >> >> >> > challenge
>> > > >> >> >> >> > and hence
>> > > >> >> >> >> > can be maintained from a single codeline IMO.  It is
>> mostly
>> > > >> >> >> >> > abstracted from
>> > > >> >> >> >> > the OS APIs via Java and our design choices.  Where it
>> is not
>> > > >> >> >> >> > we
>> > > >> >> >> >> > can
>> > > >> >> >> >> > continue to add plugable abstractions.
>> > > >> >> >> >> >
>> > > >> >> >> >
>> > > >> >> >> >
>> > > >> >> >
>> > > >> >> >
>> > > >> >
>> > > >> >
>> > > >
>> > > >
>> > >
>>
>
>
>
> --
> Best regards,
>
>    - Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> (via Tom White)

Re: [Vote] Merge branch-trunk-win to trunk

[Konstantin Shvachko <sh...@gmail.com>]

Andrew, this used to be on all -dev lists. Let's keep it that way.

To the point.
Does this mean that people are silently porting windows changes to branch-2?
New features on a branch should be voted first, no?

Thanks,
--Konstantin


On Mon, Mar 25, 2013 at 1:36 PM, Andrew Purtell <ap...@apache.org> wrote:
> Noticed this too. Simply a 'public' modifier is missing, but it's unclear
> how this could not have been caught prior to check-in.
>
>
> On Mon, Mar 25, 2013 at 9:17 PM, Konstantin Boudnik <co...@apache.org> wrote:
>
>> It doesn't look like any progress has been done on the ticket below in the
>> last 3 weeks. And now branch-2 can't be compiled because of
>>
>>
>> hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSShell.java:[895,15]
>> WINDOWS is not public in org.apache.hadoop.fs.Path; cannot be accessed from
>> outside package
>>
>> That's exactly why I was -1'ing this...
>>   Cos
>>
>> On Mon, Mar 04, 2013 at 05:41PM, Matt Foley wrote:
>> > Thanks, gentlemen.  I've opened and taken responsibility for
>> > https://issues.apache.org/jira/browse/HADOOP-9359.  Giri Kesavan has
>> agreed
>> > to help with the parts that require Jenkins admin access.
>> >
>> > Thanks,
>> > --Matt
>> >
>> >
>> >
>> > On Mon, Mar 4, 2013 at 5:00 PM, Konstantin Shvachko <
>> shv.hadoop@gmail.com>wrote:
>> >
>> > > +1 on the merge.
>> > >
>> > > I am glad we agreed.
>> > > Having Jira to track the CI effort is a good idea.
>> > >
>> > > Thanks,
>> > > --Konstantin
>> > >
>> > > On Mon, Mar 4, 2013 at 3:29 PM, Matt Foley <mf...@hortonworks.com>
>> wrote:
>> > > > Thanks.  I agree Windows -1's in test-patch should not block commits.
>> > > >
>> > > > --Matt
>> > > >
>> > > >
>> > > >
>> > > > On Mon, Mar 4, 2013 at 2:30 PM, Konstantin Shvachko <
>> > > shv.hadoop@gmail.com>
>> > > > wrote:
>> > > >>
>> > > >> On Mon, Mar 4, 2013 at 12:22 PM, Matt Foley <mfoley@hortonworks.com
>> >
>> > > >> wrote:
>> > > >> > Konstantine, you have voted -1, and stated some requirements
>> before
>> > > >> > you'll
>> > > >> > withdraw that -1.  As I plan to do work to fulfill those
>> > > requirements, I
>> > > >> > want to make sure that what I'm proposing will, in fact, satisfy
>> you.
>> > > >> > That's why I'm asking, if we implement full "test-patch"
>> integration
>> > > for
>> > > >> > Windows, does it seem to you that that would provide adequate
>> support?
>> > > >>
>> > > >> Yes.
>> > > >>
>> > > >> > I have learned not to presume that my interpretation is correct.
>>  My
>> > > >> > interpretation of item #1 is that test-patch provides pre-commit
>> > > build,
>> > > >> > so
>> > > >> > it would satisfy item #1.  But rather than assuming that I am
>> > > >> > interpreting
>> > > >> > it correctly, I simply want your agreement that it would, or if
>> not,
>> > > >> > clarification why it won't.
>> > > >>
>> > > >> I agree it will satisfy my item #1.
>> > > >> I did not agree in my previous email, but I changed my mind based on
>> > > >> the latest discussion. I have to explain why now.
>> > > >> I was proposing nightly build because I did not want pre-commit
>> build
>> > > >> for Windows block commits to Linux. But if people are fine just
>> ignoring
>> > > >> -1s for the Windows part of the build it should be good.
>> > > >>
>> > > >> > Regarding item #2, it is also my interpretation that test-patch
>> > > provides
>> > > >> > an
>> > > >> > on-demand (perhaps 20-minutes deferred) Jenkins build and unit
>> test,
>> > > >> > with
>> > > >> > logs available to the developer, so it would satisfy item #2.  But
>> > > >> > rather
>> > > >> > than assuming that I am interpreting it correctly, I simply want
>> your
>> > > >> > agreement that it would, or if not, clarification why it won't.
>> > > >>
>> > > >> It will satisfy my item #2 in the following way:
>> > > >> I can duplicate your pre-commit build for Windows and add an input
>> > > >> parameter, which would let people run the build on their patches
>> > > >> chosen from local machine rather than attaching them to Jiras.
>> > > >>
>> > > >> Thanks,
>> > > >> --Konstantin
>> > > >>
>> > > >> > In agile terms, you are the Owner of these requirements.  Please
>> give
>> > > me
>> > > >> > owner feedback as to whether my proposed work sounds like it will
>> > > >> > satisfy
>> > > >> > the requirements.
>> > > >> >
>> > > >> > Thank you,
>> > > >> > --Matt
>> > > >> >
>> > > >> >
>> > > >> > On Sun, Mar 3, 2013 at 12:16 PM, Konstantin Shvachko
>> > > >> > <sh...@gmail.com>
>> > > >> > wrote:
>> > > >> >>
>> > > >> >> Didn't I explain in details what I am asking for?
>> > > >> >>
>> > > >> >> Thanks,
>> > > >> >> --Konst
>> > > >> >>
>> > > >> >> On Sun, Mar 3, 2013 at 11:08 AM, Matt Foley <
>> mfoley@hortonworks.com>
>> > > >> >> wrote:
>> > > >> >> > Hi Konstantin,
>> > > >> >> > I'd like to point out two things:
>> > > >> >> > First, I already committed in this thread (email of Thu, Feb
>> 28,
>> > > 2013
>> > > >> >> > at
>> > > >> >> > 6:01 PM) to providing CI for Windows builds.  So please stop
>> acting
>> > > >> >> > like
>> > > >> >> > I'm
>> > > >> >> > resisting this idea or something.
>> > > >> >> > Second, you didn't answer my question, you just kvetched about
>> the
>> > > >> >> > phrasing.
>> > > >> >> > So I ask again:
>> > > >> >> >
>> > > >> >> > Will providing full "test-patch" integration (pre-commit build
>> and
>> > > >> >> > unit
>> > > >> >> > test
>> > > >> >> > triggered by Jira "Patch Available" state) satisfy your
>> request for
>> > > >> >> > functionality #1 and #2?  Yes or no, please.
>> > > >> >> >
>> > > >> >> > Thanks,
>> > > >> >> > --Matt
>> > > >> >> >
>> > > >> >> >
>> > > >> >> > On Sat, Mar 2, 2013 at 7:32 PM, Konstantin Shvachko
>> > > >> >> > <sh...@gmail.com>
>> > > >> >> > wrote:
>> > > >> >> >>
>> > > >> >> >> Hi Matt,
>> > > >> >> >>
>> > > >> >> >> On Sat, Mar 2, 2013 at 12:32 PM, Matt Foley <
>> > > mfoley@hortonworks.com>
>> > > >> >> >> wrote:
>> > > >> >> >> > Konstantin,
>> > > >> >> >> > I would like to explore what it would take to remove this
>> > > >> >> >> > perceived
>> > > >> >> >> > impediment --
>> > > >> >> >>
>> > > >> >> >> Glad you decided to explore. Thank you.
>> > > >> >> >>
>> > > >> >> >> > although I reserve the right to argue that this is not
>> > > >> >> >> > pre-requisite to merging the cross-platform support patch.
>> > > >> >> >>
>> > > >> >> >> It's your right indeed. So as mine to question what the
>> platform
>> > > >> >> >> support means for you, which I believe remained unclear.
>> > > >> >> >> I do not impede the change as you should have noticed. My
>> > > >> >> >> requirement
>> > > >> >> >> comes from my perception of the support, which means to me
>> exactly
>> > > >> >> >> two
>> > > >> >> >> things:
>> > > >> >> >> 1. The ability to recognise the code is broken for the
>> platform
>> > > >> >> >> 2. The ability to test new patches on the platform
>> > > >> >> >> The latter is problematic, as many noticed in this thread, for
>> > > those
>> > > >> >> >> whose customary environment does not include Windows.
>> > > >> >> >>
>> > > >> >> >> > If we implemented full "test-patch" support for Windows on
>> > > trunk,
>> > > >> >> >> > would
>> > > >> >> >> > that
>> > > >> >> >> > fulfill both your items #1 and #2?  Please note that:
>> > > >> >> >> > a) Pushing the "Patch Available" button in Jira shall cause
>> a
>> > > >> >> >> > pre-commit
>> > > >> >> >> > build to start within, I believe, 20 minutes.
>> > > >> >> >> > b) That build keeps logs for both java build and unit tests
>> for
>> > > >> >> >> > several
>> > > >> >> >> > days, that are accessible to all viewers.
>> > > >> >> >>
>> > > >> >> >> In item #1 I mostly asking for the nightly build, which is
>> simpler
>> > > >> >> >> than "test-patch". The latter would be ideal from the platform
>> > > >> >> >> support
>> > > >> >> >> viewpoint, but it is for the community to decide if we want
>> to add
>> > > >> >> >> extra +3 hours to the build.
>> > > >> >> >> Nightly build in my understanding is triggered by the timer
>> rather
>> > > >> >> >> than by Jira's "submit patch" button.  On Jenkins build
>> > > >> >> >> configuration
>> > > >> >> >> you can specify it under "Build periodically".
>> > > >> >> >>
>> > > >> >> >> > So, does this provide sufficient on-demand support that we
>> don't
>> > > >> >> >> > have
>> > > >> >> >> > to
>> > > >> >> >> > implement a whole new on-demand VM support structure of some
>> > > sort
>> > > >> >> >> > for
>> > > >> >> >> > #2
>> > > >> >> >> > (which would be an extraordinary and impractical demand)?
>> > > >> >> >>
>> > > >> >> >> I did not mention VMs. Item #2 means a build, which runs
>> > > >> >> >> "test-patch"
>> > > >> >> >> target with the file specified by a user (instead of a jira
>> > > >> >> >> attachment).
>> > > >> >> >> When user clicks "Build Now" link a box is displayed where the
>> > > user
>> > > >> >> >> can enter the file path containing the patch. This can be
>> > > specified
>> > > >> >> >> in
>> > > >> >> >> the Build Configuration under "This build is parameterized" by
>> > > >> >> >> choosing AddParameter / FileParameter. The build can run on
>> the
>> > > same
>> > > >> >> >> Windows machine as the nightly build.
>> > > >> >> >> Such build will let people test their patches for Windows on
>> > > Jenkins
>> > > >> >> >> if they don't posses a license for the right version of
>> Windows.
>> > > >> >> >> I hope this will not turn into extraordinary or impractical
>> > > effort.
>> > > >> >> >>
>> > > >> >> >> Thanks,
>> > > >> >> >> --Konst
>> > > >> >> >>
>> > > >> >> >> > Thanks,
>> > > >> >> >> > --Matt
>> > > >> >> >> >
>> > > >> >> >> >
>> > > >> >> >> > On Fri, Mar 1, 2013 at 12:18 PM, Konstantin Shvachko
>> > > >> >> >> > <sh...@gmail.com>
>> > > >> >> >> > wrote:
>> > > >> >> >> >>
>> > > >> >> >> >> -1
>> > > >> >> >> >> We should have a CI infrastructure in place before we can
>> > > commit
>> > > >> >> >> >> to
>> > > >> >> >> >> supporting Windows platform.
>> > > >> >> >> >>
>> > > >> >> >> >> Eric is right Win/Cygwin was supported since day one.
>> > > >> >> >> >> I had a Windows box under my desk running nightly builds
>> back
>> > > in
>> > > >> >> >> >> 2006-07.
>> > > >> >> >> >> People were irritated but I was filing windows bugs until
>> 0.22
>> > > >> >> >> >> release.
>> > > >> >> >> >> Times changing and I am glad to see wider support for Win
>> > > >> >> >> >> platform.
>> > > >> >> >> >>
>> > > >> >> >> >> But in order to make it work you guys need to put the CI
>> > > process
>> > > >> >> >> >> in
>> > > >> >> >> >> place
>> > > >> >> >> >>
>> > > >> >> >> >> 1. windows jenkins build: could be nightly or PreCommit.
>> > > >> >> >> >> - Nightly would mean that changes can be committed to trunk
>> > > based
>> > > >> >> >> >> on
>> > > >> >> >> >> linux PreCommit build. And people will file bugs if the
>> change
>> > > >> >> >> >> broke
>> > > >> >> >> >> Windows nightly build.
>> > > >> >> >> >> - PreCommit-win build will mean automatic reporting failed
>> > > tests
>> > > >> >> >> >> to
>> > > >> >> >> >> respective jira blocking commits the same way as it is now
>> with
>> > > >> >> >> >> linux
>> > > >> >> >> >> PreCommit builds.
>> > > >> >> >> >> We should discuss which way is more efficient for
>> developers.
>> > > >> >> >> >>
>> > > >> >> >> >> 2. On-demand-windows Jenkins build.
>> > > >> >> >> >> I see it as a build to which I can attach my patch and the
>> > > build
>> > > >> >> >> >> will
>> > > >> >> >> >> run my changes on a dedicated windows box.
>> > > >> >> >> >> That way people can test their changes without having
>> personal
>> > > >> >> >> >> windows
>> > > >> >> >> >> nodes.
>> > > >> >> >> >>
>> > > >> >> >> >> I think this is the minimal set of requirement for us to be
>> > > able
>> > > >> >> >> >> to
>> > > >> >> >> >> commit to the new platform.
>> > > >> >> >> >> Right now I see only one windows related build
>> > > >> >> >> >> https://builds.apache.org/view/Hadoop/job/Hadoop-1-win/
>> > > >> >> >> >> Which was failing since Sept 8, 2012 and did not run in the
>> > > last
>> > > >> >> >> >> month.
>> > > >> >> >> >>
>> > > >> >> >> >> Thanks,
>> > > >> >> >> >> --Konst
>> > > >> >> >> >>
>> > > >> >> >> >> On Thu, Feb 28, 2013 at 8:47 PM, Eric Baldeschwieler
>> > > >> >> >> >> <er...@hortonworks.com> wrote:
>> > > >> >> >> >> > +1 (non-binding)
>> > > >> >> >> >> >
>> > > >> >> >> >> > A few of observations:
>> > > >> >> >> >> >
>> > > >> >> >> >> > - Windows has actually been a supported platform for
>> Hadoop
>> > > >> >> >> >> > since
>> > > >> >> >> >> > 0.1
>> > > >> >> >> >> > .
>> > > >> >> >> >> > Doug championed supporting windows then and we've
>> continued
>> > > to
>> > > >> >> >> >> > do
>> > > >> >> >> >> > it
>> > > >> >> >> >> > with
>> > > >> >> >> >> > varying vigor over time.  To my knowledge we've never
>> made a
>> > > >> >> >> >> > decision
>> > > >> >> >> >> > to
>> > > >> >> >> >> > drop windows support.  The change here is improving our
>> > > support
>> > > >> >> >> >> > and
>> > > >> >> >> >> > dropping
>> > > >> >> >> >> > the requirement of cigwin.  We had Nutch windows users
>> on the
>> > > >> >> >> >> > list
>> > > >> >> >> >> > in
>> > > >> >> >> >> > 2006
>> > > >> >> >> >> > and we've been supporting windows FS requirements since
>> > > >> >> >> >> > inception.
>> > > >> >> >> >> >
>> > > >> >> >> >> > - A little pragmatism will go a long way.  As a community
>> > > we've
>> > > >> >> >> >> > got
>> > > >> >> >> >> > to
>> > > >> >> >> >> > stay committed to keeping hadoop simple (so it does work
>> on
>> > > >> >> >> >> > many
>> > > >> >> >> >> > platforms)
>> > > >> >> >> >> > and extending it to take advantage of key emerging
>> > > OS/hardware
>> > > >> >> >> >> > features,
>> > > >> >> >> >> > such as containers, new FSs, virtualization, flash ...
>>  We
>> > > >> >> >> >> > should
>> > > >> >> >> >> > all
>> > > >> >> >> >> > plan
>> > > >> >> >> >> > to let new features & optimizations emerge that don't
>> work
>> > > >> >> >> >> > everywhere, if
>> > > >> >> >> >> > they are compelling and central to hadoop's mission of
>> being
>> > > >> >> >> >> > THE
>> > > >> >> >> >> > best
>> > > >> >> >> >> > fabric
>> > > >> >> >> >> > for storing and processing big data.
>> > > >> >> >> >> >
>> > > >> >> >> >> > - A UI project like KDE has to deal with the MANY
>> differences
>> > > >> >> >> >> > between
>> > > >> >> >> >> > windows and linux UI APIs.  Hadoop faces no such complex
>> > > >> >> >> >> > challenge
>> > > >> >> >> >> > and hence
>> > > >> >> >> >> > can be maintained from a single codeline IMO.  It is
>> mostly
>> > > >> >> >> >> > abstracted from
>> > > >> >> >> >> > the OS APIs via Java and our design choices.  Where it
>> is not
>> > > >> >> >> >> > we
>> > > >> >> >> >> > can
>> > > >> >> >> >> > continue to add plugable abstractions.
>> > > >> >> >> >> >
>> > > >> >> >> >
>> > > >> >> >> >
>> > > >> >> >
>> > > >> >> >
>> > > >> >
>> > > >> >
>> > > >
>> > > >
>> > >
>>
>
>
>
> --
> Best regards,
>
>    - Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> (via Tom White)

Re: [Vote] Merge branch-trunk-win to trunk

[Konstantin Shvachko <sh...@gmail.com>]

Andrew, this used to be on all -dev lists. Let's keep it that way.

To the point.
Does this mean that people are silently porting windows changes to branch-2?
New features on a branch should be voted first, no?

Thanks,
--Konstantin


On Mon, Mar 25, 2013 at 1:36 PM, Andrew Purtell <ap...@apache.org> wrote:
> Noticed this too. Simply a 'public' modifier is missing, but it's unclear
> how this could not have been caught prior to check-in.
>
>
> On Mon, Mar 25, 2013 at 9:17 PM, Konstantin Boudnik <co...@apache.org> wrote:
>
>> It doesn't look like any progress has been done on the ticket below in the
>> last 3 weeks. And now branch-2 can't be compiled because of
>>
>>
>> hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSShell.java:[895,15]
>> WINDOWS is not public in org.apache.hadoop.fs.Path; cannot be accessed from
>> outside package
>>
>> That's exactly why I was -1'ing this...
>>   Cos
>>
>> On Mon, Mar 04, 2013 at 05:41PM, Matt Foley wrote:
>> > Thanks, gentlemen.  I've opened and taken responsibility for
>> > https://issues.apache.org/jira/browse/HADOOP-9359.  Giri Kesavan has
>> agreed
>> > to help with the parts that require Jenkins admin access.
>> >
>> > Thanks,
>> > --Matt
>> >
>> >
>> >
>> > On Mon, Mar 4, 2013 at 5:00 PM, Konstantin Shvachko <
>> shv.hadoop@gmail.com>wrote:
>> >
>> > > +1 on the merge.
>> > >
>> > > I am glad we agreed.
>> > > Having Jira to track the CI effort is a good idea.
>> > >
>> > > Thanks,
>> > > --Konstantin
>> > >
>> > > On Mon, Mar 4, 2013 at 3:29 PM, Matt Foley <mf...@hortonworks.com>
>> wrote:
>> > > > Thanks.  I agree Windows -1's in test-patch should not block commits.
>> > > >
>> > > > --Matt
>> > > >
>> > > >
>> > > >
>> > > > On Mon, Mar 4, 2013 at 2:30 PM, Konstantin Shvachko <
>> > > shv.hadoop@gmail.com>
>> > > > wrote:
>> > > >>
>> > > >> On Mon, Mar 4, 2013 at 12:22 PM, Matt Foley <mfoley@hortonworks.com
>> >
>> > > >> wrote:
>> > > >> > Konstantine, you have voted -1, and stated some requirements
>> before
>> > > >> > you'll
>> > > >> > withdraw that -1.  As I plan to do work to fulfill those
>> > > requirements, I
>> > > >> > want to make sure that what I'm proposing will, in fact, satisfy
>> you.
>> > > >> > That's why I'm asking, if we implement full "test-patch"
>> integration
>> > > for
>> > > >> > Windows, does it seem to you that that would provide adequate
>> support?
>> > > >>
>> > > >> Yes.
>> > > >>
>> > > >> > I have learned not to presume that my interpretation is correct.
>>  My
>> > > >> > interpretation of item #1 is that test-patch provides pre-commit
>> > > build,
>> > > >> > so
>> > > >> > it would satisfy item #1.  But rather than assuming that I am
>> > > >> > interpreting
>> > > >> > it correctly, I simply want your agreement that it would, or if
>> not,
>> > > >> > clarification why it won't.
>> > > >>
>> > > >> I agree it will satisfy my item #1.
>> > > >> I did not agree in my previous email, but I changed my mind based on
>> > > >> the latest discussion. I have to explain why now.
>> > > >> I was proposing nightly build because I did not want pre-commit
>> build
>> > > >> for Windows block commits to Linux. But if people are fine just
>> ignoring
>> > > >> -1s for the Windows part of the build it should be good.
>> > > >>
>> > > >> > Regarding item #2, it is also my interpretation that test-patch
>> > > provides
>> > > >> > an
>> > > >> > on-demand (perhaps 20-minutes deferred) Jenkins build and unit
>> test,
>> > > >> > with
>> > > >> > logs available to the developer, so it would satisfy item #2.  But
>> > > >> > rather
>> > > >> > than assuming that I am interpreting it correctly, I simply want
>> your
>> > > >> > agreement that it would, or if not, clarification why it won't.
>> > > >>
>> > > >> It will satisfy my item #2 in the following way:
>> > > >> I can duplicate your pre-commit build for Windows and add an input
>> > > >> parameter, which would let people run the build on their patches
>> > > >> chosen from local machine rather than attaching them to Jiras.
>> > > >>
>> > > >> Thanks,
>> > > >> --Konstantin
>> > > >>
>> > > >> > In agile terms, you are the Owner of these requirements.  Please
>> give
>> > > me
>> > > >> > owner feedback as to whether my proposed work sounds like it will
>> > > >> > satisfy
>> > > >> > the requirements.
>> > > >> >
>> > > >> > Thank you,
>> > > >> > --Matt
>> > > >> >
>> > > >> >
>> > > >> > On Sun, Mar 3, 2013 at 12:16 PM, Konstantin Shvachko
>> > > >> > <sh...@gmail.com>
>> > > >> > wrote:
>> > > >> >>
>> > > >> >> Didn't I explain in details what I am asking for?
>> > > >> >>
>> > > >> >> Thanks,
>> > > >> >> --Konst
>> > > >> >>
>> > > >> >> On Sun, Mar 3, 2013 at 11:08 AM, Matt Foley <
>> mfoley@hortonworks.com>
>> > > >> >> wrote:
>> > > >> >> > Hi Konstantin,
>> > > >> >> > I'd like to point out two things:
>> > > >> >> > First, I already committed in this thread (email of Thu, Feb
>> 28,
>> > > 2013
>> > > >> >> > at
>> > > >> >> > 6:01 PM) to providing CI for Windows builds.  So please stop
>> acting
>> > > >> >> > like
>> > > >> >> > I'm
>> > > >> >> > resisting this idea or something.
>> > > >> >> > Second, you didn't answer my question, you just kvetched about
>> the
>> > > >> >> > phrasing.
>> > > >> >> > So I ask again:
>> > > >> >> >
>> > > >> >> > Will providing full "test-patch" integration (pre-commit build
>> and
>> > > >> >> > unit
>> > > >> >> > test
>> > > >> >> > triggered by Jira "Patch Available" state) satisfy your
>> request for
>> > > >> >> > functionality #1 and #2?  Yes or no, please.
>> > > >> >> >
>> > > >> >> > Thanks,
>> > > >> >> > --Matt
>> > > >> >> >
>> > > >> >> >
>> > > >> >> > On Sat, Mar 2, 2013 at 7:32 PM, Konstantin Shvachko
>> > > >> >> > <sh...@gmail.com>
>> > > >> >> > wrote:
>> > > >> >> >>
>> > > >> >> >> Hi Matt,
>> > > >> >> >>
>> > > >> >> >> On Sat, Mar 2, 2013 at 12:32 PM, Matt Foley <
>> > > mfoley@hortonworks.com>
>> > > >> >> >> wrote:
>> > > >> >> >> > Konstantin,
>> > > >> >> >> > I would like to explore what it would take to remove this
>> > > >> >> >> > perceived
>> > > >> >> >> > impediment --
>> > > >> >> >>
>> > > >> >> >> Glad you decided to explore. Thank you.
>> > > >> >> >>
>> > > >> >> >> > although I reserve the right to argue that this is not
>> > > >> >> >> > pre-requisite to merging the cross-platform support patch.
>> > > >> >> >>
>> > > >> >> >> It's your right indeed. So as mine to question what the
>> platform
>> > > >> >> >> support means for you, which I believe remained unclear.
>> > > >> >> >> I do not impede the change as you should have noticed. My
>> > > >> >> >> requirement
>> > > >> >> >> comes from my perception of the support, which means to me
>> exactly
>> > > >> >> >> two
>> > > >> >> >> things:
>> > > >> >> >> 1. The ability to recognise the code is broken for the
>> platform
>> > > >> >> >> 2. The ability to test new patches on the platform
>> > > >> >> >> The latter is problematic, as many noticed in this thread, for
>> > > those
>> > > >> >> >> whose customary environment does not include Windows.
>> > > >> >> >>
>> > > >> >> >> > If we implemented full "test-patch" support for Windows on
>> > > trunk,
>> > > >> >> >> > would
>> > > >> >> >> > that
>> > > >> >> >> > fulfill both your items #1 and #2?  Please note that:
>> > > >> >> >> > a) Pushing the "Patch Available" button in Jira shall cause
>> a
>> > > >> >> >> > pre-commit
>> > > >> >> >> > build to start within, I believe, 20 minutes.
>> > > >> >> >> > b) That build keeps logs for both java build and unit tests
>> for
>> > > >> >> >> > several
>> > > >> >> >> > days, that are accessible to all viewers.
>> > > >> >> >>
>> > > >> >> >> In item #1 I mostly asking for the nightly build, which is
>> simpler
>> > > >> >> >> than "test-patch". The latter would be ideal from the platform
>> > > >> >> >> support
>> > > >> >> >> viewpoint, but it is for the community to decide if we want
>> to add
>> > > >> >> >> extra +3 hours to the build.
>> > > >> >> >> Nightly build in my understanding is triggered by the timer
>> rather
>> > > >> >> >> than by Jira's "submit patch" button.  On Jenkins build
>> > > >> >> >> configuration
>> > > >> >> >> you can specify it under "Build periodically".
>> > > >> >> >>
>> > > >> >> >> > So, does this provide sufficient on-demand support that we
>> don't
>> > > >> >> >> > have
>> > > >> >> >> > to
>> > > >> >> >> > implement a whole new on-demand VM support structure of some
>> > > sort
>> > > >> >> >> > for
>> > > >> >> >> > #2
>> > > >> >> >> > (which would be an extraordinary and impractical demand)?
>> > > >> >> >>
>> > > >> >> >> I did not mention VMs. Item #2 means a build, which runs
>> > > >> >> >> "test-patch"
>> > > >> >> >> target with the file specified by a user (instead of a jira
>> > > >> >> >> attachment).
>> > > >> >> >> When user clicks "Build Now" link a box is displayed where the
>> > > user
>> > > >> >> >> can enter the file path containing the patch. This can be
>> > > specified
>> > > >> >> >> in
>> > > >> >> >> the Build Configuration under "This build is parameterized" by
>> > > >> >> >> choosing AddParameter / FileParameter. The build can run on
>> the
>> > > same
>> > > >> >> >> Windows machine as the nightly build.
>> > > >> >> >> Such build will let people test their patches for Windows on
>> > > Jenkins
>> > > >> >> >> if they don't posses a license for the right version of
>> Windows.
>> > > >> >> >> I hope this will not turn into extraordinary or impractical
>> > > effort.
>> > > >> >> >>
>> > > >> >> >> Thanks,
>> > > >> >> >> --Konst
>> > > >> >> >>
>> > > >> >> >> > Thanks,
>> > > >> >> >> > --Matt
>> > > >> >> >> >
>> > > >> >> >> >
>> > > >> >> >> > On Fri, Mar 1, 2013 at 12:18 PM, Konstantin Shvachko
>> > > >> >> >> > <sh...@gmail.com>
>> > > >> >> >> > wrote:
>> > > >> >> >> >>
>> > > >> >> >> >> -1
>> > > >> >> >> >> We should have a CI infrastructure in place before we can
>> > > commit
>> > > >> >> >> >> to
>> > > >> >> >> >> supporting Windows platform.
>> > > >> >> >> >>
>> > > >> >> >> >> Eric is right Win/Cygwin was supported since day one.
>> > > >> >> >> >> I had a Windows box under my desk running nightly builds
>> back
>> > > in
>> > > >> >> >> >> 2006-07.
>> > > >> >> >> >> People were irritated but I was filing windows bugs until
>> 0.22
>> > > >> >> >> >> release.
>> > > >> >> >> >> Times changing and I am glad to see wider support for Win
>> > > >> >> >> >> platform.
>> > > >> >> >> >>
>> > > >> >> >> >> But in order to make it work you guys need to put the CI
>> > > process
>> > > >> >> >> >> in
>> > > >> >> >> >> place
>> > > >> >> >> >>
>> > > >> >> >> >> 1. windows jenkins build: could be nightly or PreCommit.
>> > > >> >> >> >> - Nightly would mean that changes can be committed to trunk
>> > > based
>> > > >> >> >> >> on
>> > > >> >> >> >> linux PreCommit build. And people will file bugs if the
>> change
>> > > >> >> >> >> broke
>> > > >> >> >> >> Windows nightly build.
>> > > >> >> >> >> - PreCommit-win build will mean automatic reporting failed
>> > > tests
>> > > >> >> >> >> to
>> > > >> >> >> >> respective jira blocking commits the same way as it is now
>> with
>> > > >> >> >> >> linux
>> > > >> >> >> >> PreCommit builds.
>> > > >> >> >> >> We should discuss which way is more efficient for
>> developers.
>> > > >> >> >> >>
>> > > >> >> >> >> 2. On-demand-windows Jenkins build.
>> > > >> >> >> >> I see it as a build to which I can attach my patch and the
>> > > build
>> > > >> >> >> >> will
>> > > >> >> >> >> run my changes on a dedicated windows box.
>> > > >> >> >> >> That way people can test their changes without having
>> personal
>> > > >> >> >> >> windows
>> > > >> >> >> >> nodes.
>> > > >> >> >> >>
>> > > >> >> >> >> I think this is the minimal set of requirement for us to be
>> > > able
>> > > >> >> >> >> to
>> > > >> >> >> >> commit to the new platform.
>> > > >> >> >> >> Right now I see only one windows related build
>> > > >> >> >> >> https://builds.apache.org/view/Hadoop/job/Hadoop-1-win/
>> > > >> >> >> >> Which was failing since Sept 8, 2012 and did not run in the
>> > > last
>> > > >> >> >> >> month.
>> > > >> >> >> >>
>> > > >> >> >> >> Thanks,
>> > > >> >> >> >> --Konst
>> > > >> >> >> >>
>> > > >> >> >> >> On Thu, Feb 28, 2013 at 8:47 PM, Eric Baldeschwieler
>> > > >> >> >> >> <er...@hortonworks.com> wrote:
>> > > >> >> >> >> > +1 (non-binding)
>> > > >> >> >> >> >
>> > > >> >> >> >> > A few of observations:
>> > > >> >> >> >> >
>> > > >> >> >> >> > - Windows has actually been a supported platform for
>> Hadoop
>> > > >> >> >> >> > since
>> > > >> >> >> >> > 0.1
>> > > >> >> >> >> > .
>> > > >> >> >> >> > Doug championed supporting windows then and we've
>> continued
>> > > to
>> > > >> >> >> >> > do
>> > > >> >> >> >> > it
>> > > >> >> >> >> > with
>> > > >> >> >> >> > varying vigor over time.  To my knowledge we've never
>> made a
>> > > >> >> >> >> > decision
>> > > >> >> >> >> > to
>> > > >> >> >> >> > drop windows support.  The change here is improving our
>> > > support
>> > > >> >> >> >> > and
>> > > >> >> >> >> > dropping
>> > > >> >> >> >> > the requirement of cigwin.  We had Nutch windows users
>> on the
>> > > >> >> >> >> > list
>> > > >> >> >> >> > in
>> > > >> >> >> >> > 2006
>> > > >> >> >> >> > and we've been supporting windows FS requirements since
>> > > >> >> >> >> > inception.
>> > > >> >> >> >> >
>> > > >> >> >> >> > - A little pragmatism will go a long way.  As a community
>> > > we've
>> > > >> >> >> >> > got
>> > > >> >> >> >> > to
>> > > >> >> >> >> > stay committed to keeping hadoop simple (so it does work
>> on
>> > > >> >> >> >> > many
>> > > >> >> >> >> > platforms)
>> > > >> >> >> >> > and extending it to take advantage of key emerging
>> > > OS/hardware
>> > > >> >> >> >> > features,
>> > > >> >> >> >> > such as containers, new FSs, virtualization, flash ...
>>  We
>> > > >> >> >> >> > should
>> > > >> >> >> >> > all
>> > > >> >> >> >> > plan
>> > > >> >> >> >> > to let new features & optimizations emerge that don't
>> work
>> > > >> >> >> >> > everywhere, if
>> > > >> >> >> >> > they are compelling and central to hadoop's mission of
>> being
>> > > >> >> >> >> > THE
>> > > >> >> >> >> > best
>> > > >> >> >> >> > fabric
>> > > >> >> >> >> > for storing and processing big data.
>> > > >> >> >> >> >
>> > > >> >> >> >> > - A UI project like KDE has to deal with the MANY
>> differences
>> > > >> >> >> >> > between
>> > > >> >> >> >> > windows and linux UI APIs.  Hadoop faces no such complex
>> > > >> >> >> >> > challenge
>> > > >> >> >> >> > and hence
>> > > >> >> >> >> > can be maintained from a single codeline IMO.  It is
>> mostly
>> > > >> >> >> >> > abstracted from
>> > > >> >> >> >> > the OS APIs via Java and our design choices.  Where it
>> is not
>> > > >> >> >> >> > we
>> > > >> >> >> >> > can
>> > > >> >> >> >> > continue to add plugable abstractions.
>> > > >> >> >> >> >
>> > > >> >> >> >
>> > > >> >> >> >
>> > > >> >> >
>> > > >> >> >
>> > > >> >
>> > > >> >
>> > > >
>> > > >
>> > >
>>
>
>
>
> --
> Best regards,
>
>    - Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> (via Tom White)

Re: [Vote] Merge branch-trunk-win to trunk

[Konstantin Shvachko <sh...@gmail.com>]

Andrew, this used to be on all -dev lists. Let's keep it that way.

To the point.
Does this mean that people are silently porting windows changes to branch-2?
New features on a branch should be voted first, no?

Thanks,
--Konstantin


On Mon, Mar 25, 2013 at 1:36 PM, Andrew Purtell <ap...@apache.org> wrote:
> Noticed this too. Simply a 'public' modifier is missing, but it's unclear
> how this could not have been caught prior to check-in.
>
>
> On Mon, Mar 25, 2013 at 9:17 PM, Konstantin Boudnik <co...@apache.org> wrote:
>
>> It doesn't look like any progress has been done on the ticket below in the
>> last 3 weeks. And now branch-2 can't be compiled because of
>>
>>
>> hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSShell.java:[895,15]
>> WINDOWS is not public in org.apache.hadoop.fs.Path; cannot be accessed from
>> outside package
>>
>> That's exactly why I was -1'ing this...
>>   Cos
>>
>> On Mon, Mar 04, 2013 at 05:41PM, Matt Foley wrote:
>> > Thanks, gentlemen.  I've opened and taken responsibility for
>> > https://issues.apache.org/jira/browse/HADOOP-9359.  Giri Kesavan has
>> agreed
>> > to help with the parts that require Jenkins admin access.
>> >
>> > Thanks,
>> > --Matt
>> >
>> >
>> >
>> > On Mon, Mar 4, 2013 at 5:00 PM, Konstantin Shvachko <
>> shv.hadoop@gmail.com>wrote:
>> >
>> > > +1 on the merge.
>> > >
>> > > I am glad we agreed.
>> > > Having Jira to track the CI effort is a good idea.
>> > >
>> > > Thanks,
>> > > --Konstantin
>> > >
>> > > On Mon, Mar 4, 2013 at 3:29 PM, Matt Foley <mf...@hortonworks.com>
>> wrote:
>> > > > Thanks.  I agree Windows -1's in test-patch should not block commits.
>> > > >
>> > > > --Matt
>> > > >
>> > > >
>> > > >
>> > > > On Mon, Mar 4, 2013 at 2:30 PM, Konstantin Shvachko <
>> > > shv.hadoop@gmail.com>
>> > > > wrote:
>> > > >>
>> > > >> On Mon, Mar 4, 2013 at 12:22 PM, Matt Foley <mfoley@hortonworks.com
>> >
>> > > >> wrote:
>> > > >> > Konstantine, you have voted -1, and stated some requirements
>> before
>> > > >> > you'll
>> > > >> > withdraw that -1.  As I plan to do work to fulfill those
>> > > requirements, I
>> > > >> > want to make sure that what I'm proposing will, in fact, satisfy
>> you.
>> > > >> > That's why I'm asking, if we implement full "test-patch"
>> integration
>> > > for
>> > > >> > Windows, does it seem to you that that would provide adequate
>> support?
>> > > >>
>> > > >> Yes.
>> > > >>
>> > > >> > I have learned not to presume that my interpretation is correct.
>>  My
>> > > >> > interpretation of item #1 is that test-patch provides pre-commit
>> > > build,
>> > > >> > so
>> > > >> > it would satisfy item #1.  But rather than assuming that I am
>> > > >> > interpreting
>> > > >> > it correctly, I simply want your agreement that it would, or if
>> not,
>> > > >> > clarification why it won't.
>> > > >>
>> > > >> I agree it will satisfy my item #1.
>> > > >> I did not agree in my previous email, but I changed my mind based on
>> > > >> the latest discussion. I have to explain why now.
>> > > >> I was proposing nightly build because I did not want pre-commit
>> build
>> > > >> for Windows block commits to Linux. But if people are fine just
>> ignoring
>> > > >> -1s for the Windows part of the build it should be good.
>> > > >>
>> > > >> > Regarding item #2, it is also my interpretation that test-patch
>> > > provides
>> > > >> > an
>> > > >> > on-demand (perhaps 20-minutes deferred) Jenkins build and unit
>> test,
>> > > >> > with
>> > > >> > logs available to the developer, so it would satisfy item #2.  But
>> > > >> > rather
>> > > >> > than assuming that I am interpreting it correctly, I simply want
>> your
>> > > >> > agreement that it would, or if not, clarification why it won't.
>> > > >>
>> > > >> It will satisfy my item #2 in the following way:
>> > > >> I can duplicate your pre-commit build for Windows and add an input
>> > > >> parameter, which would let people run the build on their patches
>> > > >> chosen from local machine rather than attaching them to Jiras.
>> > > >>
>> > > >> Thanks,
>> > > >> --Konstantin
>> > > >>
>> > > >> > In agile terms, you are the Owner of these requirements.  Please
>> give
>> > > me
>> > > >> > owner feedback as to whether my proposed work sounds like it will
>> > > >> > satisfy
>> > > >> > the requirements.
>> > > >> >
>> > > >> > Thank you,
>> > > >> > --Matt
>> > > >> >
>> > > >> >
>> > > >> > On Sun, Mar 3, 2013 at 12:16 PM, Konstantin Shvachko
>> > > >> > <sh...@gmail.com>
>> > > >> > wrote:
>> > > >> >>
>> > > >> >> Didn't I explain in details what I am asking for?
>> > > >> >>
>> > > >> >> Thanks,
>> > > >> >> --Konst
>> > > >> >>
>> > > >> >> On Sun, Mar 3, 2013 at 11:08 AM, Matt Foley <
>> mfoley@hortonworks.com>
>> > > >> >> wrote:
>> > > >> >> > Hi Konstantin,
>> > > >> >> > I'd like to point out two things:
>> > > >> >> > First, I already committed in this thread (email of Thu, Feb
>> 28,
>> > > 2013
>> > > >> >> > at
>> > > >> >> > 6:01 PM) to providing CI for Windows builds.  So please stop
>> acting
>> > > >> >> > like
>> > > >> >> > I'm
>> > > >> >> > resisting this idea or something.
>> > > >> >> > Second, you didn't answer my question, you just kvetched about
>> the
>> > > >> >> > phrasing.
>> > > >> >> > So I ask again:
>> > > >> >> >
>> > > >> >> > Will providing full "test-patch" integration (pre-commit build
>> and
>> > > >> >> > unit
>> > > >> >> > test
>> > > >> >> > triggered by Jira "Patch Available" state) satisfy your
>> request for
>> > > >> >> > functionality #1 and #2?  Yes or no, please.
>> > > >> >> >
>> > > >> >> > Thanks,
>> > > >> >> > --Matt
>> > > >> >> >
>> > > >> >> >
>> > > >> >> > On Sat, Mar 2, 2013 at 7:32 PM, Konstantin Shvachko
>> > > >> >> > <sh...@gmail.com>
>> > > >> >> > wrote:
>> > > >> >> >>
>> > > >> >> >> Hi Matt,
>> > > >> >> >>
>> > > >> >> >> On Sat, Mar 2, 2013 at 12:32 PM, Matt Foley <
>> > > mfoley@hortonworks.com>
>> > > >> >> >> wrote:
>> > > >> >> >> > Konstantin,
>> > > >> >> >> > I would like to explore what it would take to remove this
>> > > >> >> >> > perceived
>> > > >> >> >> > impediment --
>> > > >> >> >>
>> > > >> >> >> Glad you decided to explore. Thank you.
>> > > >> >> >>
>> > > >> >> >> > although I reserve the right to argue that this is not
>> > > >> >> >> > pre-requisite to merging the cross-platform support patch.
>> > > >> >> >>
>> > > >> >> >> It's your right indeed. So as mine to question what the
>> platform
>> > > >> >> >> support means for you, which I believe remained unclear.
>> > > >> >> >> I do not impede the change as you should have noticed. My
>> > > >> >> >> requirement
>> > > >> >> >> comes from my perception of the support, which means to me
>> exactly
>> > > >> >> >> two
>> > > >> >> >> things:
>> > > >> >> >> 1. The ability to recognise the code is broken for the
>> platform
>> > > >> >> >> 2. The ability to test new patches on the platform
>> > > >> >> >> The latter is problematic, as many noticed in this thread, for
>> > > those
>> > > >> >> >> whose customary environment does not include Windows.
>> > > >> >> >>
>> > > >> >> >> > If we implemented full "test-patch" support for Windows on
>> > > trunk,
>> > > >> >> >> > would
>> > > >> >> >> > that
>> > > >> >> >> > fulfill both your items #1 and #2?  Please note that:
>> > > >> >> >> > a) Pushing the "Patch Available" button in Jira shall cause
>> a
>> > > >> >> >> > pre-commit
>> > > >> >> >> > build to start within, I believe, 20 minutes.
>> > > >> >> >> > b) That build keeps logs for both java build and unit tests
>> for
>> > > >> >> >> > several
>> > > >> >> >> > days, that are accessible to all viewers.
>> > > >> >> >>
>> > > >> >> >> In item #1 I mostly asking for the nightly build, which is
>> simpler
>> > > >> >> >> than "test-patch". The latter would be ideal from the platform
>> > > >> >> >> support
>> > > >> >> >> viewpoint, but it is for the community to decide if we want
>> to add
>> > > >> >> >> extra +3 hours to the build.
>> > > >> >> >> Nightly build in my understanding is triggered by the timer
>> rather
>> > > >> >> >> than by Jira's "submit patch" button.  On Jenkins build
>> > > >> >> >> configuration
>> > > >> >> >> you can specify it under "Build periodically".
>> > > >> >> >>
>> > > >> >> >> > So, does this provide sufficient on-demand support that we
>> don't
>> > > >> >> >> > have
>> > > >> >> >> > to
>> > > >> >> >> > implement a whole new on-demand VM support structure of some
>> > > sort
>> > > >> >> >> > for
>> > > >> >> >> > #2
>> > > >> >> >> > (which would be an extraordinary and impractical demand)?
>> > > >> >> >>
>> > > >> >> >> I did not mention VMs. Item #2 means a build, which runs
>> > > >> >> >> "test-patch"
>> > > >> >> >> target with the file specified by a user (instead of a jira
>> > > >> >> >> attachment).
>> > > >> >> >> When user clicks "Build Now" link a box is displayed where the
>> > > user
>> > > >> >> >> can enter the file path containing the patch. This can be
>> > > specified
>> > > >> >> >> in
>> > > >> >> >> the Build Configuration under "This build is parameterized" by
>> > > >> >> >> choosing AddParameter / FileParameter. The build can run on
>> the
>> > > same
>> > > >> >> >> Windows machine as the nightly build.
>> > > >> >> >> Such build will let people test their patches for Windows on
>> > > Jenkins
>> > > >> >> >> if they don't posses a license for the right version of
>> Windows.
>> > > >> >> >> I hope this will not turn into extraordinary or impractical
>> > > effort.
>> > > >> >> >>
>> > > >> >> >> Thanks,
>> > > >> >> >> --Konst
>> > > >> >> >>
>> > > >> >> >> > Thanks,
>> > > >> >> >> > --Matt
>> > > >> >> >> >
>> > > >> >> >> >
>> > > >> >> >> > On Fri, Mar 1, 2013 at 12:18 PM, Konstantin Shvachko
>> > > >> >> >> > <sh...@gmail.com>
>> > > >> >> >> > wrote:
>> > > >> >> >> >>
>> > > >> >> >> >> -1
>> > > >> >> >> >> We should have a CI infrastructure in place before we can
>> > > commit
>> > > >> >> >> >> to
>> > > >> >> >> >> supporting Windows platform.
>> > > >> >> >> >>
>> > > >> >> >> >> Eric is right Win/Cygwin was supported since day one.
>> > > >> >> >> >> I had a Windows box under my desk running nightly builds
>> back
>> > > in
>> > > >> >> >> >> 2006-07.
>> > > >> >> >> >> People were irritated but I was filing windows bugs until
>> 0.22
>> > > >> >> >> >> release.
>> > > >> >> >> >> Times changing and I am glad to see wider support for Win
>> > > >> >> >> >> platform.
>> > > >> >> >> >>
>> > > >> >> >> >> But in order to make it work you guys need to put the CI
>> > > process
>> > > >> >> >> >> in
>> > > >> >> >> >> place
>> > > >> >> >> >>
>> > > >> >> >> >> 1. windows jenkins build: could be nightly or PreCommit.
>> > > >> >> >> >> - Nightly would mean that changes can be committed to trunk
>> > > based
>> > > >> >> >> >> on
>> > > >> >> >> >> linux PreCommit build. And people will file bugs if the
>> change
>> > > >> >> >> >> broke
>> > > >> >> >> >> Windows nightly build.
>> > > >> >> >> >> - PreCommit-win build will mean automatic reporting failed
>> > > tests
>> > > >> >> >> >> to
>> > > >> >> >> >> respective jira blocking commits the same way as it is now
>> with
>> > > >> >> >> >> linux
>> > > >> >> >> >> PreCommit builds.
>> > > >> >> >> >> We should discuss which way is more efficient for
>> developers.
>> > > >> >> >> >>
>> > > >> >> >> >> 2. On-demand-windows Jenkins build.
>> > > >> >> >> >> I see it as a build to which I can attach my patch and the
>> > > build
>> > > >> >> >> >> will
>> > > >> >> >> >> run my changes on a dedicated windows box.
>> > > >> >> >> >> That way people can test their changes without having
>> personal
>> > > >> >> >> >> windows
>> > > >> >> >> >> nodes.
>> > > >> >> >> >>
>> > > >> >> >> >> I think this is the minimal set of requirement for us to be
>> > > able
>> > > >> >> >> >> to
>> > > >> >> >> >> commit to the new platform.
>> > > >> >> >> >> Right now I see only one windows related build
>> > > >> >> >> >> https://builds.apache.org/view/Hadoop/job/Hadoop-1-win/
>> > > >> >> >> >> Which was failing since Sept 8, 2012 and did not run in the
>> > > last
>> > > >> >> >> >> month.
>> > > >> >> >> >>
>> > > >> >> >> >> Thanks,
>> > > >> >> >> >> --Konst
>> > > >> >> >> >>
>> > > >> >> >> >> On Thu, Feb 28, 2013 at 8:47 PM, Eric Baldeschwieler
>> > > >> >> >> >> <er...@hortonworks.com> wrote:
>> > > >> >> >> >> > +1 (non-binding)
>> > > >> >> >> >> >
>> > > >> >> >> >> > A few of observations:
>> > > >> >> >> >> >
>> > > >> >> >> >> > - Windows has actually been a supported platform for
>> Hadoop
>> > > >> >> >> >> > since
>> > > >> >> >> >> > 0.1
>> > > >> >> >> >> > .
>> > > >> >> >> >> > Doug championed supporting windows then and we've
>> continued
>> > > to
>> > > >> >> >> >> > do
>> > > >> >> >> >> > it
>> > > >> >> >> >> > with
>> > > >> >> >> >> > varying vigor over time.  To my knowledge we've never
>> made a
>> > > >> >> >> >> > decision
>> > > >> >> >> >> > to
>> > > >> >> >> >> > drop windows support.  The change here is improving our
>> > > support
>> > > >> >> >> >> > and
>> > > >> >> >> >> > dropping
>> > > >> >> >> >> > the requirement of cigwin.  We had Nutch windows users
>> on the
>> > > >> >> >> >> > list
>> > > >> >> >> >> > in
>> > > >> >> >> >> > 2006
>> > > >> >> >> >> > and we've been supporting windows FS requirements since
>> > > >> >> >> >> > inception.
>> > > >> >> >> >> >
>> > > >> >> >> >> > - A little pragmatism will go a long way.  As a community
>> > > we've
>> > > >> >> >> >> > got
>> > > >> >> >> >> > to
>> > > >> >> >> >> > stay committed to keeping hadoop simple (so it does work
>> on
>> > > >> >> >> >> > many
>> > > >> >> >> >> > platforms)
>> > > >> >> >> >> > and extending it to take advantage of key emerging
>> > > OS/hardware
>> > > >> >> >> >> > features,
>> > > >> >> >> >> > such as containers, new FSs, virtualization, flash ...
>>  We
>> > > >> >> >> >> > should
>> > > >> >> >> >> > all
>> > > >> >> >> >> > plan
>> > > >> >> >> >> > to let new features & optimizations emerge that don't
>> work
>> > > >> >> >> >> > everywhere, if
>> > > >> >> >> >> > they are compelling and central to hadoop's mission of
>> being
>> > > >> >> >> >> > THE
>> > > >> >> >> >> > best
>> > > >> >> >> >> > fabric
>> > > >> >> >> >> > for storing and processing big data.
>> > > >> >> >> >> >
>> > > >> >> >> >> > - A UI project like KDE has to deal with the MANY
>> differences
>> > > >> >> >> >> > between
>> > > >> >> >> >> > windows and linux UI APIs.  Hadoop faces no such complex
>> > > >> >> >> >> > challenge
>> > > >> >> >> >> > and hence
>> > > >> >> >> >> > can be maintained from a single codeline IMO.  It is
>> mostly
>> > > >> >> >> >> > abstracted from
>> > > >> >> >> >> > the OS APIs via Java and our design choices.  Where it
>> is not
>> > > >> >> >> >> > we
>> > > >> >> >> >> > can
>> > > >> >> >> >> > continue to add plugable abstractions.
>> > > >> >> >> >> >
>> > > >> >> >> >
>> > > >> >> >> >
>> > > >> >> >
>> > > >> >> >
>> > > >> >
>> > > >> >
>> > > >
>> > > >
>> > >
>>
>
>
>
> --
> Best regards,
>
>    - Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> (via Tom White)

Re: [Vote] Merge branch-trunk-win to trunk

[Andrew Purtell <ap...@apache.org>]

Noticed this too. Simply a 'public' modifier is missing, but it's unclear
how this could not have been caught prior to check-in.


On Mon, Mar 25, 2013 at 9:17 PM, Konstantin Boudnik <co...@apache.org> wrote:

> It doesn't look like any progress has been done on the ticket below in the
> last 3 weeks. And now branch-2 can't be compiled because of
>
>
> hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSShell.java:[895,15]
> WINDOWS is not public in org.apache.hadoop.fs.Path; cannot be accessed from
> outside package
>
> That's exactly why I was -1'ing this...
>   Cos
>
> On Mon, Mar 04, 2013 at 05:41PM, Matt Foley wrote:
> > Thanks, gentlemen.  I've opened and taken responsibility for
> > https://issues.apache.org/jira/browse/HADOOP-9359.  Giri Kesavan has
> agreed
> > to help with the parts that require Jenkins admin access.
> >
> > Thanks,
> > --Matt
> >
> >
> >
> > On Mon, Mar 4, 2013 at 5:00 PM, Konstantin Shvachko <
> shv.hadoop@gmail.com>wrote:
> >
> > > +1 on the merge.
> > >
> > > I am glad we agreed.
> > > Having Jira to track the CI effort is a good idea.
> > >
> > > Thanks,
> > > --Konstantin
> > >
> > > On Mon, Mar 4, 2013 at 3:29 PM, Matt Foley <mf...@hortonworks.com>
> wrote:
> > > > Thanks.  I agree Windows -1's in test-patch should not block commits.
> > > >
> > > > --Matt
> > > >
> > > >
> > > >
> > > > On Mon, Mar 4, 2013 at 2:30 PM, Konstantin Shvachko <
> > > shv.hadoop@gmail.com>
> > > > wrote:
> > > >>
> > > >> On Mon, Mar 4, 2013 at 12:22 PM, Matt Foley <mfoley@hortonworks.com
> >
> > > >> wrote:
> > > >> > Konstantine, you have voted -1, and stated some requirements
> before
> > > >> > you'll
> > > >> > withdraw that -1.  As I plan to do work to fulfill those
> > > requirements, I
> > > >> > want to make sure that what I'm proposing will, in fact, satisfy
> you.
> > > >> > That's why I'm asking, if we implement full "test-patch"
> integration
> > > for
> > > >> > Windows, does it seem to you that that would provide adequate
> support?
> > > >>
> > > >> Yes.
> > > >>
> > > >> > I have learned not to presume that my interpretation is correct.
>  My
> > > >> > interpretation of item #1 is that test-patch provides pre-commit
> > > build,
> > > >> > so
> > > >> > it would satisfy item #1.  But rather than assuming that I am
> > > >> > interpreting
> > > >> > it correctly, I simply want your agreement that it would, or if
> not,
> > > >> > clarification why it won't.
> > > >>
> > > >> I agree it will satisfy my item #1.
> > > >> I did not agree in my previous email, but I changed my mind based on
> > > >> the latest discussion. I have to explain why now.
> > > >> I was proposing nightly build because I did not want pre-commit
> build
> > > >> for Windows block commits to Linux. But if people are fine just
> ignoring
> > > >> -1s for the Windows part of the build it should be good.
> > > >>
> > > >> > Regarding item #2, it is also my interpretation that test-patch
> > > provides
> > > >> > an
> > > >> > on-demand (perhaps 20-minutes deferred) Jenkins build and unit
> test,
> > > >> > with
> > > >> > logs available to the developer, so it would satisfy item #2.  But
> > > >> > rather
> > > >> > than assuming that I am interpreting it correctly, I simply want
> your
> > > >> > agreement that it would, or if not, clarification why it won't.
> > > >>
> > > >> It will satisfy my item #2 in the following way:
> > > >> I can duplicate your pre-commit build for Windows and add an input
> > > >> parameter, which would let people run the build on their patches
> > > >> chosen from local machine rather than attaching them to Jiras.
> > > >>
> > > >> Thanks,
> > > >> --Konstantin
> > > >>
> > > >> > In agile terms, you are the Owner of these requirements.  Please
> give
> > > me
> > > >> > owner feedback as to whether my proposed work sounds like it will
> > > >> > satisfy
> > > >> > the requirements.
> > > >> >
> > > >> > Thank you,
> > > >> > --Matt
> > > >> >
> > > >> >
> > > >> > On Sun, Mar 3, 2013 at 12:16 PM, Konstantin Shvachko
> > > >> > <sh...@gmail.com>
> > > >> > wrote:
> > > >> >>
> > > >> >> Didn't I explain in details what I am asking for?
> > > >> >>
> > > >> >> Thanks,
> > > >> >> --Konst
> > > >> >>
> > > >> >> On Sun, Mar 3, 2013 at 11:08 AM, Matt Foley <
> mfoley@hortonworks.com>
> > > >> >> wrote:
> > > >> >> > Hi Konstantin,
> > > >> >> > I'd like to point out two things:
> > > >> >> > First, I already committed in this thread (email of Thu, Feb
> 28,
> > > 2013
> > > >> >> > at
> > > >> >> > 6:01 PM) to providing CI for Windows builds.  So please stop
> acting
> > > >> >> > like
> > > >> >> > I'm
> > > >> >> > resisting this idea or something.
> > > >> >> > Second, you didn't answer my question, you just kvetched about
> the
> > > >> >> > phrasing.
> > > >> >> > So I ask again:
> > > >> >> >
> > > >> >> > Will providing full "test-patch" integration (pre-commit build
> and
> > > >> >> > unit
> > > >> >> > test
> > > >> >> > triggered by Jira "Patch Available" state) satisfy your
> request for
> > > >> >> > functionality #1 and #2?  Yes or no, please.
> > > >> >> >
> > > >> >> > Thanks,
> > > >> >> > --Matt
> > > >> >> >
> > > >> >> >
> > > >> >> > On Sat, Mar 2, 2013 at 7:32 PM, Konstantin Shvachko
> > > >> >> > <sh...@gmail.com>
> > > >> >> > wrote:
> > > >> >> >>
> > > >> >> >> Hi Matt,
> > > >> >> >>
> > > >> >> >> On Sat, Mar 2, 2013 at 12:32 PM, Matt Foley <
> > > mfoley@hortonworks.com>
> > > >> >> >> wrote:
> > > >> >> >> > Konstantin,
> > > >> >> >> > I would like to explore what it would take to remove this
> > > >> >> >> > perceived
> > > >> >> >> > impediment --
> > > >> >> >>
> > > >> >> >> Glad you decided to explore. Thank you.
> > > >> >> >>
> > > >> >> >> > although I reserve the right to argue that this is not
> > > >> >> >> > pre-requisite to merging the cross-platform support patch.
> > > >> >> >>
> > > >> >> >> It's your right indeed. So as mine to question what the
> platform
> > > >> >> >> support means for you, which I believe remained unclear.
> > > >> >> >> I do not impede the change as you should have noticed. My
> > > >> >> >> requirement
> > > >> >> >> comes from my perception of the support, which means to me
> exactly
> > > >> >> >> two
> > > >> >> >> things:
> > > >> >> >> 1. The ability to recognise the code is broken for the
> platform
> > > >> >> >> 2. The ability to test new patches on the platform
> > > >> >> >> The latter is problematic, as many noticed in this thread, for
> > > those
> > > >> >> >> whose customary environment does not include Windows.
> > > >> >> >>
> > > >> >> >> > If we implemented full "test-patch" support for Windows on
> > > trunk,
> > > >> >> >> > would
> > > >> >> >> > that
> > > >> >> >> > fulfill both your items #1 and #2?  Please note that:
> > > >> >> >> > a) Pushing the "Patch Available" button in Jira shall cause
> a
> > > >> >> >> > pre-commit
> > > >> >> >> > build to start within, I believe, 20 minutes.
> > > >> >> >> > b) That build keeps logs for both java build and unit tests
> for
> > > >> >> >> > several
> > > >> >> >> > days, that are accessible to all viewers.
> > > >> >> >>
> > > >> >> >> In item #1 I mostly asking for the nightly build, which is
> simpler
> > > >> >> >> than "test-patch". The latter would be ideal from the platform
> > > >> >> >> support
> > > >> >> >> viewpoint, but it is for the community to decide if we want
> to add
> > > >> >> >> extra +3 hours to the build.
> > > >> >> >> Nightly build in my understanding is triggered by the timer
> rather
> > > >> >> >> than by Jira's "submit patch" button.  On Jenkins build
> > > >> >> >> configuration
> > > >> >> >> you can specify it under "Build periodically".
> > > >> >> >>
> > > >> >> >> > So, does this provide sufficient on-demand support that we
> don't
> > > >> >> >> > have
> > > >> >> >> > to
> > > >> >> >> > implement a whole new on-demand VM support structure of some
> > > sort
> > > >> >> >> > for
> > > >> >> >> > #2
> > > >> >> >> > (which would be an extraordinary and impractical demand)?
> > > >> >> >>
> > > >> >> >> I did not mention VMs. Item #2 means a build, which runs
> > > >> >> >> "test-patch"
> > > >> >> >> target with the file specified by a user (instead of a jira
> > > >> >> >> attachment).
> > > >> >> >> When user clicks "Build Now" link a box is displayed where the
> > > user
> > > >> >> >> can enter the file path containing the patch. This can be
> > > specified
> > > >> >> >> in
> > > >> >> >> the Build Configuration under "This build is parameterized" by
> > > >> >> >> choosing AddParameter / FileParameter. The build can run on
> the
> > > same
> > > >> >> >> Windows machine as the nightly build.
> > > >> >> >> Such build will let people test their patches for Windows on
> > > Jenkins
> > > >> >> >> if they don't posses a license for the right version of
> Windows.
> > > >> >> >> I hope this will not turn into extraordinary or impractical
> > > effort.
> > > >> >> >>
> > > >> >> >> Thanks,
> > > >> >> >> --Konst
> > > >> >> >>
> > > >> >> >> > Thanks,
> > > >> >> >> > --Matt
> > > >> >> >> >
> > > >> >> >> >
> > > >> >> >> > On Fri, Mar 1, 2013 at 12:18 PM, Konstantin Shvachko
> > > >> >> >> > <sh...@gmail.com>
> > > >> >> >> > wrote:
> > > >> >> >> >>
> > > >> >> >> >> -1
> > > >> >> >> >> We should have a CI infrastructure in place before we can
> > > commit
> > > >> >> >> >> to
> > > >> >> >> >> supporting Windows platform.
> > > >> >> >> >>
> > > >> >> >> >> Eric is right Win/Cygwin was supported since day one.
> > > >> >> >> >> I had a Windows box under my desk running nightly builds
> back
> > > in
> > > >> >> >> >> 2006-07.
> > > >> >> >> >> People were irritated but I was filing windows bugs until
> 0.22
> > > >> >> >> >> release.
> > > >> >> >> >> Times changing and I am glad to see wider support for Win
> > > >> >> >> >> platform.
> > > >> >> >> >>
> > > >> >> >> >> But in order to make it work you guys need to put the CI
> > > process
> > > >> >> >> >> in
> > > >> >> >> >> place
> > > >> >> >> >>
> > > >> >> >> >> 1. windows jenkins build: could be nightly or PreCommit.
> > > >> >> >> >> - Nightly would mean that changes can be committed to trunk
> > > based
> > > >> >> >> >> on
> > > >> >> >> >> linux PreCommit build. And people will file bugs if the
> change
> > > >> >> >> >> broke
> > > >> >> >> >> Windows nightly build.
> > > >> >> >> >> - PreCommit-win build will mean automatic reporting failed
> > > tests
> > > >> >> >> >> to
> > > >> >> >> >> respective jira blocking commits the same way as it is now
> with
> > > >> >> >> >> linux
> > > >> >> >> >> PreCommit builds.
> > > >> >> >> >> We should discuss which way is more efficient for
> developers.
> > > >> >> >> >>
> > > >> >> >> >> 2. On-demand-windows Jenkins build.
> > > >> >> >> >> I see it as a build to which I can attach my patch and the
> > > build
> > > >> >> >> >> will
> > > >> >> >> >> run my changes on a dedicated windows box.
> > > >> >> >> >> That way people can test their changes without having
> personal
> > > >> >> >> >> windows
> > > >> >> >> >> nodes.
> > > >> >> >> >>
> > > >> >> >> >> I think this is the minimal set of requirement for us to be
> > > able
> > > >> >> >> >> to
> > > >> >> >> >> commit to the new platform.
> > > >> >> >> >> Right now I see only one windows related build
> > > >> >> >> >> https://builds.apache.org/view/Hadoop/job/Hadoop-1-win/
> > > >> >> >> >> Which was failing since Sept 8, 2012 and did not run in the
> > > last
> > > >> >> >> >> month.
> > > >> >> >> >>
> > > >> >> >> >> Thanks,
> > > >> >> >> >> --Konst
> > > >> >> >> >>
> > > >> >> >> >> On Thu, Feb 28, 2013 at 8:47 PM, Eric Baldeschwieler
> > > >> >> >> >> <er...@hortonworks.com> wrote:
> > > >> >> >> >> > +1 (non-binding)
> > > >> >> >> >> >
> > > >> >> >> >> > A few of observations:
> > > >> >> >> >> >
> > > >> >> >> >> > - Windows has actually been a supported platform for
> Hadoop
> > > >> >> >> >> > since
> > > >> >> >> >> > 0.1
> > > >> >> >> >> > .
> > > >> >> >> >> > Doug championed supporting windows then and we've
> continued
> > > to
> > > >> >> >> >> > do
> > > >> >> >> >> > it
> > > >> >> >> >> > with
> > > >> >> >> >> > varying vigor over time.  To my knowledge we've never
> made a
> > > >> >> >> >> > decision
> > > >> >> >> >> > to
> > > >> >> >> >> > drop windows support.  The change here is improving our
> > > support
> > > >> >> >> >> > and
> > > >> >> >> >> > dropping
> > > >> >> >> >> > the requirement of cigwin.  We had Nutch windows users
> on the
> > > >> >> >> >> > list
> > > >> >> >> >> > in
> > > >> >> >> >> > 2006
> > > >> >> >> >> > and we've been supporting windows FS requirements since
> > > >> >> >> >> > inception.
> > > >> >> >> >> >
> > > >> >> >> >> > - A little pragmatism will go a long way.  As a community
> > > we've
> > > >> >> >> >> > got
> > > >> >> >> >> > to
> > > >> >> >> >> > stay committed to keeping hadoop simple (so it does work
> on
> > > >> >> >> >> > many
> > > >> >> >> >> > platforms)
> > > >> >> >> >> > and extending it to take advantage of key emerging
> > > OS/hardware
> > > >> >> >> >> > features,
> > > >> >> >> >> > such as containers, new FSs, virtualization, flash ...
>  We
> > > >> >> >> >> > should
> > > >> >> >> >> > all
> > > >> >> >> >> > plan
> > > >> >> >> >> > to let new features & optimizations emerge that don't
> work
> > > >> >> >> >> > everywhere, if
> > > >> >> >> >> > they are compelling and central to hadoop's mission of
> being
> > > >> >> >> >> > THE
> > > >> >> >> >> > best
> > > >> >> >> >> > fabric
> > > >> >> >> >> > for storing and processing big data.
> > > >> >> >> >> >
> > > >> >> >> >> > - A UI project like KDE has to deal with the MANY
> differences
> > > >> >> >> >> > between
> > > >> >> >> >> > windows and linux UI APIs.  Hadoop faces no such complex
> > > >> >> >> >> > challenge
> > > >> >> >> >> > and hence
> > > >> >> >> >> > can be maintained from a single codeline IMO.  It is
> mostly
> > > >> >> >> >> > abstracted from
> > > >> >> >> >> > the OS APIs via Java and our design choices.  Where it
> is not
> > > >> >> >> >> > we
> > > >> >> >> >> > can
> > > >> >> >> >> > continue to add plugable abstractions.
> > > >> >> >> >> >
> > > >> >> >> >
> > > >> >> >> >
> > > >> >> >
> > > >> >> >
> > > >> >
> > > >> >
> > > >
> > > >
> > >
>



-- 
Best regards,

   - Andy

Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)

Re: [Vote] Merge branch-trunk-win to trunk

[Suresh Srinivas <su...@hortonworks.com>]

Cos,

There is progress being made on that ticket. Also it has nothing to do with
that.

Please follow the discussion here and why this happened due to an invalid
commit that was reverted -
https://issues.apache.org/jira/browse/HDFS-4615?focusedCommentId=13612650&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13612650

Regards,
Suresh


On Mon, Mar 25, 2013 at 1:17 PM, Konstantin Boudnik <co...@apache.org> wrote:

> It doesn't look like any progress has been done on the ticket below in the
> last 3 weeks. And now branch-2 can't be compiled because of
>
>
> hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSShell.java:[895,15]
> WINDOWS is not public in org.apache.hadoop.fs.Path; cannot be accessed from
> outside package
>
> That's exactly why I was -1'ing this...
>   Cos
>
> On Mon, Mar 04, 2013 at 05:41PM, Matt Foley wrote:
> > Thanks, gentlemen.  I've opened and taken responsibility for
> > https://issues.apache.org/jira/browse/HADOOP-9359.  Giri Kesavan has
> agreed
> > to help with the parts that require Jenkins admin access.
> >
> > Thanks,
> > --Matt
> >
> >
> >
> > On Mon, Mar 4, 2013 at 5:00 PM, Konstantin Shvachko <
> shv.hadoop@gmail.com>wrote:
> >
> > > +1 on the merge.
> > >
> > > I am glad we agreed.
> > > Having Jira to track the CI effort is a good idea.
> > >
> > > Thanks,
> > > --Konstantin
> > >
> > > On Mon, Mar 4, 2013 at 3:29 PM, Matt Foley <mf...@hortonworks.com>
> wrote:
> > > > Thanks.  I agree Windows -1's in test-patch should not block commits.
> > > >
> > > > --Matt
> > > >
> > > >
> > > >
> > > > On Mon, Mar 4, 2013 at 2:30 PM, Konstantin Shvachko <
> > > shv.hadoop@gmail.com>
> > > > wrote:
> > > >>
> > > >> On Mon, Mar 4, 2013 at 12:22 PM, Matt Foley <mfoley@hortonworks.com
> >
> > > >> wrote:
> > > >> > Konstantine, you have voted -1, and stated some requirements
> before
> > > >> > you'll
> > > >> > withdraw that -1.  As I plan to do work to fulfill those
> > > requirements, I
> > > >> > want to make sure that what I'm proposing will, in fact, satisfy
> you.
> > > >> > That's why I'm asking, if we implement full "test-patch"
> integration
> > > for
> > > >> > Windows, does it seem to you that that would provide adequate
> support?
> > > >>
> > > >> Yes.
> > > >>
> > > >> > I have learned not to presume that my interpretation is correct.
>  My
> > > >> > interpretation of item #1 is that test-patch provides pre-commit
> > > build,
> > > >> > so
> > > >> > it would satisfy item #1.  But rather than assuming that I am
> > > >> > interpreting
> > > >> > it correctly, I simply want your agreement that it would, or if
> not,
> > > >> > clarification why it won't.
> > > >>
> > > >> I agree it will satisfy my item #1.
> > > >> I did not agree in my previous email, but I changed my mind based on
> > > >> the latest discussion. I have to explain why now.
> > > >> I was proposing nightly build because I did not want pre-commit
> build
> > > >> for Windows block commits to Linux. But if people are fine just
> ignoring
> > > >> -1s for the Windows part of the build it should be good.
> > > >>
> > > >> > Regarding item #2, it is also my interpretation that test-patch
> > > provides
> > > >> > an
> > > >> > on-demand (perhaps 20-minutes deferred) Jenkins build and unit
> test,
> > > >> > with
> > > >> > logs available to the developer, so it would satisfy item #2.  But
> > > >> > rather
> > > >> > than assuming that I am interpreting it correctly, I simply want
> your
> > > >> > agreement that it would, or if not, clarification why it won't.
> > > >>
> > > >> It will satisfy my item #2 in the following way:
> > > >> I can duplicate your pre-commit build for Windows and add an input
> > > >> parameter, which would let people run the build on their patches
> > > >> chosen from local machine rather than attaching them to Jiras.
> > > >>
> > > >> Thanks,
> > > >> --Konstantin
> > > >>
> > > >> > In agile terms, you are the Owner of these requirements.  Please
> give
> > > me
> > > >> > owner feedback as to whether my proposed work sounds like it will
> > > >> > satisfy
> > > >> > the requirements.
> > > >> >
> > > >> > Thank you,
> > > >> > --Matt
> > > >> >
> > > >> >
> > > >> > On Sun, Mar 3, 2013 at 12:16 PM, Konstantin Shvachko
> > > >> > <sh...@gmail.com>
> > > >> > wrote:
> > > >> >>
> > > >> >> Didn't I explain in details what I am asking for?
> > > >> >>
> > > >> >> Thanks,
> > > >> >> --Konst
> > > >> >>
> > > >> >> On Sun, Mar 3, 2013 at 11:08 AM, Matt Foley <
> mfoley@hortonworks.com>
> > > >> >> wrote:
> > > >> >> > Hi Konstantin,
> > > >> >> > I'd like to point out two things:
> > > >> >> > First, I already committed in this thread (email of Thu, Feb
> 28,
> > > 2013
> > > >> >> > at
> > > >> >> > 6:01 PM) to providing CI for Windows builds.  So please stop
> acting
> > > >> >> > like
> > > >> >> > I'm
> > > >> >> > resisting this idea or something.
> > > >> >> > Second, you didn't answer my question, you just kvetched about
> the
> > > >> >> > phrasing.
> > > >> >> > So I ask again:
> > > >> >> >
> > > >> >> > Will providing full "test-patch" integration (pre-commit build
> and
> > > >> >> > unit
> > > >> >> > test
> > > >> >> > triggered by Jira "Patch Available" state) satisfy your
> request for
> > > >> >> > functionality #1 and #2?  Yes or no, please.
> > > >> >> >
> > > >> >> > Thanks,
> > > >> >> > --Matt
> > > >> >> >
> > > >> >> >
> > > >> >> > On Sat, Mar 2, 2013 at 7:32 PM, Konstantin Shvachko
> > > >> >> > <sh...@gmail.com>
> > > >> >> > wrote:
> > > >> >> >>
> > > >> >> >> Hi Matt,
> > > >> >> >>
> > > >> >> >> On Sat, Mar 2, 2013 at 12:32 PM, Matt Foley <
> > > mfoley@hortonworks.com>
> > > >> >> >> wrote:
> > > >> >> >> > Konstantin,
> > > >> >> >> > I would like to explore what it would take to remove this
> > > >> >> >> > perceived
> > > >> >> >> > impediment --
> > > >> >> >>
> > > >> >> >> Glad you decided to explore. Thank you.
> > > >> >> >>
> > > >> >> >> > although I reserve the right to argue that this is not
> > > >> >> >> > pre-requisite to merging the cross-platform support patch.
> > > >> >> >>
> > > >> >> >> It's your right indeed. So as mine to question what the
> platform
> > > >> >> >> support means for you, which I believe remained unclear.
> > > >> >> >> I do not impede the change as you should have noticed. My
> > > >> >> >> requirement
> > > >> >> >> comes from my perception of the support, which means to me
> exactly
> > > >> >> >> two
> > > >> >> >> things:
> > > >> >> >> 1. The ability to recognise the code is broken for the
> platform
> > > >> >> >> 2. The ability to test new patches on the platform
> > > >> >> >> The latter is problematic, as many noticed in this thread, for
> > > those
> > > >> >> >> whose customary environment does not include Windows.
> > > >> >> >>
> > > >> >> >> > If we implemented full "test-patch" support for Windows on
> > > trunk,
> > > >> >> >> > would
> > > >> >> >> > that
> > > >> >> >> > fulfill both your items #1 and #2?  Please note that:
> > > >> >> >> > a) Pushing the "Patch Available" button in Jira shall cause
> a
> > > >> >> >> > pre-commit
> > > >> >> >> > build to start within, I believe, 20 minutes.
> > > >> >> >> > b) That build keeps logs for both java build and unit tests
> for
> > > >> >> >> > several
> > > >> >> >> > days, that are accessible to all viewers.
> > > >> >> >>
> > > >> >> >> In item #1 I mostly asking for the nightly build, which is
> simpler
> > > >> >> >> than "test-patch". The latter would be ideal from the platform
> > > >> >> >> support
> > > >> >> >> viewpoint, but it is for the community to decide if we want
> to add
> > > >> >> >> extra +3 hours to the build.
> > > >> >> >> Nightly build in my understanding is triggered by the timer
> rather
> > > >> >> >> than by Jira's "submit patch" button.  On Jenkins build
> > > >> >> >> configuration
> > > >> >> >> you can specify it under "Build periodically".
> > > >> >> >>
> > > >> >> >> > So, does this provide sufficient on-demand support that we
> don't
> > > >> >> >> > have
> > > >> >> >> > to
> > > >> >> >> > implement a whole new on-demand VM support structure of some
> > > sort
> > > >> >> >> > for
> > > >> >> >> > #2
> > > >> >> >> > (which would be an extraordinary and impractical demand)?
> > > >> >> >>
> > > >> >> >> I did not mention VMs. Item #2 means a build, which runs
> > > >> >> >> "test-patch"
> > > >> >> >> target with the file specified by a user (instead of a jira
> > > >> >> >> attachment).
> > > >> >> >> When user clicks "Build Now" link a box is displayed where the
> > > user
> > > >> >> >> can enter the file path containing the patch. This can be
> > > specified
> > > >> >> >> in
> > > >> >> >> the Build Configuration under "This build is parameterized" by
> > > >> >> >> choosing AddParameter / FileParameter. The build can run on
> the
> > > same
> > > >> >> >> Windows machine as the nightly build.
> > > >> >> >> Such build will let people test their patches for Windows on
> > > Jenkins
> > > >> >> >> if they don't posses a license for the right version of
> Windows.
> > > >> >> >> I hope this will not turn into extraordinary or impractical
> > > effort.
> > > >> >> >>
> > > >> >> >> Thanks,
> > > >> >> >> --Konst
> > > >> >> >>
> > > >> >> >> > Thanks,
> > > >> >> >> > --Matt
> > > >> >> >> >
> > > >> >> >> >
> > > >> >> >> > On Fri, Mar 1, 2013 at 12:18 PM, Konstantin Shvachko
> > > >> >> >> > <sh...@gmail.com>
> > > >> >> >> > wrote:
> > > >> >> >> >>
> > > >> >> >> >> -1
> > > >> >> >> >> We should have a CI infrastructure in place before we can
> > > commit
> > > >> >> >> >> to
> > > >> >> >> >> supporting Windows platform.
> > > >> >> >> >>
> > > >> >> >> >> Eric is right Win/Cygwin was supported since day one.
> > > >> >> >> >> I had a Windows box under my desk running nightly builds
> back
> > > in
> > > >> >> >> >> 2006-07.
> > > >> >> >> >> People were irritated but I was filing windows bugs until
> 0.22
> > > >> >> >> >> release.
> > > >> >> >> >> Times changing and I am glad to see wider support for Win
> > > >> >> >> >> platform.
> > > >> >> >> >>
> > > >> >> >> >> But in order to make it work you guys need to put the CI
> > > process
> > > >> >> >> >> in
> > > >> >> >> >> place
> > > >> >> >> >>
> > > >> >> >> >> 1. windows jenkins build: could be nightly or PreCommit.
> > > >> >> >> >> - Nightly would mean that changes can be committed to trunk
> > > based
> > > >> >> >> >> on
> > > >> >> >> >> linux PreCommit build. And people will file bugs if the
> change
> > > >> >> >> >> broke
> > > >> >> >> >> Windows nightly build.
> > > >> >> >> >> - PreCommit-win build will mean automatic reporting failed
> > > tests
> > > >> >> >> >> to
> > > >> >> >> >> respective jira blocking commits the same way as it is now
> with
> > > >> >> >> >> linux
> > > >> >> >> >> PreCommit builds.
> > > >> >> >> >> We should discuss which way is more efficient for
> developers.
> > > >> >> >> >>
> > > >> >> >> >> 2. On-demand-windows Jenkins build.
> > > >> >> >> >> I see it as a build to which I can attach my patch and the
> > > build
> > > >> >> >> >> will
> > > >> >> >> >> run my changes on a dedicated windows box.
> > > >> >> >> >> That way people can test their changes without having
> personal
> > > >> >> >> >> windows
> > > >> >> >> >> nodes.
> > > >> >> >> >>
> > > >> >> >> >> I think this is the minimal set of requirement for us to be
> > > able
> > > >> >> >> >> to
> > > >> >> >> >> commit to the new platform.
> > > >> >> >> >> Right now I see only one windows related build
> > > >> >> >> >> https://builds.apache.org/view/Hadoop/job/Hadoop-1-win/
> > > >> >> >> >> Which was failing since Sept 8, 2012 and did not run in the
> > > last
> > > >> >> >> >> month.
> > > >> >> >> >>
> > > >> >> >> >> Thanks,
> > > >> >> >> >> --Konst
> > > >> >> >> >>
> > > >> >> >> >> On Thu, Feb 28, 2013 at 8:47 PM, Eric Baldeschwieler
> > > >> >> >> >> <er...@hortonworks.com> wrote:
> > > >> >> >> >> > +1 (non-binding)
> > > >> >> >> >> >
> > > >> >> >> >> > A few of observations:
> > > >> >> >> >> >
> > > >> >> >> >> > - Windows has actually been a supported platform for
> Hadoop
> > > >> >> >> >> > since
> > > >> >> >> >> > 0.1
> > > >> >> >> >> > .
> > > >> >> >> >> > Doug championed supporting windows then and we've
> continued
> > > to
> > > >> >> >> >> > do
> > > >> >> >> >> > it
> > > >> >> >> >> > with
> > > >> >> >> >> > varying vigor over time.  To my knowledge we've never
> made a
> > > >> >> >> >> > decision
> > > >> >> >> >> > to
> > > >> >> >> >> > drop windows support.  The change here is improving our
> > > support
> > > >> >> >> >> > and
> > > >> >> >> >> > dropping
> > > >> >> >> >> > the requirement of cigwin.  We had Nutch windows users
> on the
> > > >> >> >> >> > list
> > > >> >> >> >> > in
> > > >> >> >> >> > 2006
> > > >> >> >> >> > and we've been supporting windows FS requirements since
> > > >> >> >> >> > inception.
> > > >> >> >> >> >
> > > >> >> >> >> > - A little pragmatism will go a long way.  As a community
> > > we've
> > > >> >> >> >> > got
> > > >> >> >> >> > to
> > > >> >> >> >> > stay committed to keeping hadoop simple (so it does work
> on
> > > >> >> >> >> > many
> > > >> >> >> >> > platforms)
> > > >> >> >> >> > and extending it to take advantage of key emerging
> > > OS/hardware
> > > >> >> >> >> > features,
> > > >> >> >> >> > such as containers, new FSs, virtualization, flash ...
>  We
> > > >> >> >> >> > should
> > > >> >> >> >> > all
> > > >> >> >> >> > plan
> > > >> >> >> >> > to let new features & optimizations emerge that don't
> work
> > > >> >> >> >> > everywhere, if
> > > >> >> >> >> > they are compelling and central to hadoop's mission of
> being
> > > >> >> >> >> > THE
> > > >> >> >> >> > best
> > > >> >> >> >> > fabric
> > > >> >> >> >> > for storing and processing big data.
> > > >> >> >> >> >
> > > >> >> >> >> > - A UI project like KDE has to deal with the MANY
> differences
> > > >> >> >> >> > between
> > > >> >> >> >> > windows and linux UI APIs.  Hadoop faces no such complex
> > > >> >> >> >> > challenge
> > > >> >> >> >> > and hence
> > > >> >> >> >> > can be maintained from a single codeline IMO.  It is
> mostly
> > > >> >> >> >> > abstracted from
> > > >> >> >> >> > the OS APIs via Java and our design choices.  Where it
> is not
> > > >> >> >> >> > we
> > > >> >> >> >> > can
> > > >> >> >> >> > continue to add plugable abstractions.
> > > >> >> >> >> >
> > > >> >> >> >
> > > >> >> >> >
> > > >> >> >
> > > >> >> >
> > > >> >
> > > >> >
> > > >
> > > >
> > >
>



-- 
http://hortonworks.com/download/

Re: [Vote] Merge branch-trunk-win to trunk

[Konstantin Boudnik <co...@apache.org>]

Yes, you are right of course - the mis-merged commit is the cause. Thanks for
pointing this out!

I think it would be beneficial if we had branch-2 on going build in the
Jenkins. I will go ahead and create one tonight.

Cos

On Mon, Mar 25, 2013 at 05:09PM, Suresh Srinivas wrote:
> Adding other mailing lists I missed earlier.
> 
> Cos,
> 
> There is progress being made on that ticket. Also it has nothing to do with
> that.
> 
> Please follow the discussion here and why this happened due to an invalid
> commit that was reverted -
> https://issues.apache.org/jira/browse/HDFS-4615?focusedCommentId=13612650&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13612650
> 
> Regards,
> Suresh
> 
> 
> On Mon, Mar 25, 2013 at 1:17 PM, Konstantin Boudnik <co...@apache.org> wrote:
> 
> > It doesn't look like any progress has been done on the ticket below in the
> > last 3 weeks. And now branch-2 can't be compiled because of
> >
> >
> > hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSShell.java:[895,15]
> > WINDOWS is not public in org.apache.hadoop.fs.Path; cannot be accessed from
> > outside package
> >
> > That's exactly why I was -1'ing this...
> >   Cos
> >
> > On Mon, Mar 04, 2013 at 05:41PM, Matt Foley wrote:
> > > Thanks, gentlemen.  I've opened and taken responsibility for
> > > https://issues.apache.org/jira/browse/HADOOP-9359.  Giri Kesavan has
> > agreed
> > > to help with the parts that require Jenkins admin access.
> > >
> > > Thanks,
> > > --Matt
> > >
> > >
> > >
> > > On Mon, Mar 4, 2013 at 5:00 PM, Konstantin Shvachko <
> > shv.hadoop@gmail.com>wrote:
> > >
> > > > +1 on the merge.
> > > >
> > > > I am glad we agreed.
> > > > Having Jira to track the CI effort is a good idea.
> > > >
> > > > Thanks,
> > > > --Konstantin
> > > >
> > > > On Mon, Mar 4, 2013 at 3:29 PM, Matt Foley <mf...@hortonworks.com>
> > wrote:
> > > > > Thanks.  I agree Windows -1's in test-patch should not block commits.
> > > > >
> > > > > --Matt
> > > > >
> > > > >
> > > > >
> > > > > On Mon, Mar 4, 2013 at 2:30 PM, Konstantin Shvachko <
> > > > shv.hadoop@gmail.com>
> > > > > wrote:
> > > > >>
> > > > >> On Mon, Mar 4, 2013 at 12:22 PM, Matt Foley <mfoley@hortonworks.com
> > >
> > > > >> wrote:
> > > > >> > Konstantine, you have voted -1, and stated some requirements
> > before
> > > > >> > you'll
> > > > >> > withdraw that -1.  As I plan to do work to fulfill those
> > > > requirements, I
> > > > >> > want to make sure that what I'm proposing will, in fact, satisfy
> > you.
> > > > >> > That's why I'm asking, if we implement full "test-patch"
> > integration
> > > > for
> > > > >> > Windows, does it seem to you that that would provide adequate
> > support?
> > > > >>
> > > > >> Yes.
> > > > >>
> > > > >> > I have learned not to presume that my interpretation is correct.
> >  My
> > > > >> > interpretation of item #1 is that test-patch provides pre-commit
> > > > build,
> > > > >> > so
> > > > >> > it would satisfy item #1.  But rather than assuming that I am
> > > > >> > interpreting
> > > > >> > it correctly, I simply want your agreement that it would, or if
> > not,
> > > > >> > clarification why it won't.
> > > > >>
> > > > >> I agree it will satisfy my item #1.
> > > > >> I did not agree in my previous email, but I changed my mind based on
> > > > >> the latest discussion. I have to explain why now.
> > > > >> I was proposing nightly build because I did not want pre-commit
> > build
> > > > >> for Windows block commits to Linux. But if people are fine just
> > ignoring
> > > > >> -1s for the Windows part of the build it should be good.
> > > > >>
> > > > >> > Regarding item #2, it is also my interpretation that test-patch
> > > > provides
> > > > >> > an
> > > > >> > on-demand (perhaps 20-minutes deferred) Jenkins build and unit
> > test,
> > > > >> > with
> > > > >> > logs available to the developer, so it would satisfy item #2.  But
> > > > >> > rather
> > > > >> > than assuming that I am interpreting it correctly, I simply want
> > your
> > > > >> > agreement that it would, or if not, clarification why it won't.
> > > > >>
> > > > >> It will satisfy my item #2 in the following way:
> > > > >> I can duplicate your pre-commit build for Windows and add an input
> > > > >> parameter, which would let people run the build on their patches
> > > > >> chosen from local machine rather than attaching them to Jiras.
> > > > >>
> > > > >> Thanks,
> > > > >> --Konstantin
> > > > >>
> > > > >> > In agile terms, you are the Owner of these requirements.  Please
> > give
> > > > me
> > > > >> > owner feedback as to whether my proposed work sounds like it will
> > > > >> > satisfy
> > > > >> > the requirements.
> > > > >> >
> > > > >> > Thank you,
> > > > >> > --Matt
> > > > >> >
> > > > >> >
> > > > >> > On Sun, Mar 3, 2013 at 12:16 PM, Konstantin Shvachko
> > > > >> > <sh...@gmail.com>
> > > > >> > wrote:
> > > > >> >>
> > > > >> >> Didn't I explain in details what I am asking for?
> > > > >> >>
> > > > >> >> Thanks,
> > > > >> >> --Konst
> > > > >> >>
> > > > >> >> On Sun, Mar 3, 2013 at 11:08 AM, Matt Foley <
> > mfoley@hortonworks.com>
> > > > >> >> wrote:
> > > > >> >> > Hi Konstantin,
> > > > >> >> > I'd like to point out two things:
> > > > >> >> > First, I already committed in this thread (email of Thu, Feb
> > 28,
> > > > 2013
> > > > >> >> > at
> > > > >> >> > 6:01 PM) to providing CI for Windows builds.  So please stop
> > acting
> > > > >> >> > like
> > > > >> >> > I'm
> > > > >> >> > resisting this idea or something.
> > > > >> >> > Second, you didn't answer my question, you just kvetched about
> > the
> > > > >> >> > phrasing.
> > > > >> >> > So I ask again:
> > > > >> >> >
> > > > >> >> > Will providing full "test-patch" integration (pre-commit build
> > and
> > > > >> >> > unit
> > > > >> >> > test
> > > > >> >> > triggered by Jira "Patch Available" state) satisfy your
> > request for
> > > > >> >> > functionality #1 and #2?  Yes or no, please.
> > > > >> >> >
> > > > >> >> > Thanks,
> > > > >> >> > --Matt
> > > > >> >> >
> > > > >> >> >
> > > > >> >> > On Sat, Mar 2, 2013 at 7:32 PM, Konstantin Shvachko
> > > > >> >> > <sh...@gmail.com>
> > > > >> >> > wrote:
> > > > >> >> >>
> > > > >> >> >> Hi Matt,
> > > > >> >> >>
> > > > >> >> >> On Sat, Mar 2, 2013 at 12:32 PM, Matt Foley <
> > > > mfoley@hortonworks.com>
> > > > >> >> >> wrote:
> > > > >> >> >> > Konstantin,
> > > > >> >> >> > I would like to explore what it would take to remove this
> > > > >> >> >> > perceived
> > > > >> >> >> > impediment --
> > > > >> >> >>
> > > > >> >> >> Glad you decided to explore. Thank you.
> > > > >> >> >>
> > > > >> >> >> > although I reserve the right to argue that this is not
> > > > >> >> >> > pre-requisite to merging the cross-platform support patch.
> > > > >> >> >>
> > > > >> >> >> It's your right indeed. So as mine to question what the
> > platform
> > > > >> >> >> support means for you, which I believe remained unclear.
> > > > >> >> >> I do not impede the change as you should have noticed. My
> > > > >> >> >> requirement
> > > > >> >> >> comes from my perception of the support, which means to me
> > exactly
> > > > >> >> >> two
> > > > >> >> >> things:
> > > > >> >> >> 1. The ability to recognise the code is broken for the
> > platform
> > > > >> >> >> 2. The ability to test new patches on the platform
> > > > >> >> >> The latter is problematic, as many noticed in this thread, for
> > > > those
> > > > >> >> >> whose customary environment does not include Windows.
> > > > >> >> >>
> > > > >> >> >> > If we implemented full "test-patch" support for Windows on
> > > > trunk,
> > > > >> >> >> > would
> > > > >> >> >> > that
> > > > >> >> >> > fulfill both your items #1 and #2?  Please note that:
> > > > >> >> >> > a) Pushing the "Patch Available" button in Jira shall cause
> > a
> > > > >> >> >> > pre-commit
> > > > >> >> >> > build to start within, I believe, 20 minutes.
> > > > >> >> >> > b) That build keeps logs for both java build and unit tests
> > for
> > > > >> >> >> > several
> > > > >> >> >> > days, that are accessible to all viewers.
> > > > >> >> >>
> > > > >> >> >> In item #1 I mostly asking for the nightly build, which is
> > simpler
> > > > >> >> >> than "test-patch". The latter would be ideal from the platform
> > > > >> >> >> support
> > > > >> >> >> viewpoint, but it is for the community to decide if we want
> > to add
> > > > >> >> >> extra +3 hours to the build.
> > > > >> >> >> Nightly build in my understanding is triggered by the timer
> > rather
> > > > >> >> >> than by Jira's "submit patch" button.  On Jenkins build
> > > > >> >> >> configuration
> > > > >> >> >> you can specify it under "Build periodically".
> > > > >> >> >>
> > > > >> >> >> > So, does this provide sufficient on-demand support that we
> > don't
> > > > >> >> >> > have
> > > > >> >> >> > to
> > > > >> >> >> > implement a whole new on-demand VM support structure of some
> > > > sort
> > > > >> >> >> > for
> > > > >> >> >> > #2
> > > > >> >> >> > (which would be an extraordinary and impractical demand)?
> > > > >> >> >>
> > > > >> >> >> I did not mention VMs. Item #2 means a build, which runs
> > > > >> >> >> "test-patch"
> > > > >> >> >> target with the file specified by a user (instead of a jira
> > > > >> >> >> attachment).
> > > > >> >> >> When user clicks "Build Now" link a box is displayed where the
> > > > user
> > > > >> >> >> can enter the file path containing the patch. This can be
> > > > specified
> > > > >> >> >> in
> > > > >> >> >> the Build Configuration under "This build is parameterized" by
> > > > >> >> >> choosing AddParameter / FileParameter. The build can run on
> > the
> > > > same
> > > > >> >> >> Windows machine as the nightly build.
> > > > >> >> >> Such build will let people test their patches for Windows on
> > > > Jenkins
> > > > >> >> >> if they don't posses a license for the right version of
> > Windows.
> > > > >> >> >> I hope this will not turn into extraordinary or impractical
> > > > effort.
> > > > >> >> >>
> > > > >> >> >> Thanks,
> > > > >> >> >> --Konst
> > > > >> >> >>
> > > > >> >> >> > Thanks,
> > > > >> >> >> > --Matt
> > > > >> >> >> >
> > > > >> >> >> >
> > > > >> >> >> > On Fri, Mar 1, 2013 at 12:18 PM, Konstantin Shvachko
> > > > >> >> >> > <sh...@gmail.com>
> > > > >> >> >> > wrote:
> > > > >> >> >> >>
> > > > >> >> >> >> -1
> > > > >> >> >> >> We should have a CI infrastructure in place before we can
> > > > commit
> > > > >> >> >> >> to
> > > > >> >> >> >> supporting Windows platform.
> > > > >> >> >> >>
> > > > >> >> >> >> Eric is right Win/Cygwin was supported since day one.
> > > > >> >> >> >> I had a Windows box under my desk running nightly builds
> > back
> > > > in
> > > > >> >> >> >> 2006-07.
> > > > >> >> >> >> People were irritated but I was filing windows bugs until
> > 0.22
> > > > >> >> >> >> release.
> > > > >> >> >> >> Times changing and I am glad to see wider support for Win
> > > > >> >> >> >> platform.
> > > > >> >> >> >>
> > > > >> >> >> >> But in order to make it work you guys need to put the CI
> > > > process
> > > > >> >> >> >> in
> > > > >> >> >> >> place
> > > > >> >> >> >>
> > > > >> >> >> >> 1. windows jenkins build: could be nightly or PreCommit.
> > > > >> >> >> >> - Nightly would mean that changes can be committed to trunk
> > > > based
> > > > >> >> >> >> on
> > > > >> >> >> >> linux PreCommit build. And people will file bugs if the
> > change
> > > > >> >> >> >> broke
> > > > >> >> >> >> Windows nightly build.
> > > > >> >> >> >> - PreCommit-win build will mean automatic reporting failed
> > > > tests
> > > > >> >> >> >> to
> > > > >> >> >> >> respective jira blocking commits the same way as it is now
> > with
> > > > >> >> >> >> linux
> > > > >> >> >> >> PreCommit builds.
> > > > >> >> >> >> We should discuss which way is more efficient for
> > developers.
> > > > >> >> >> >>
> > > > >> >> >> >> 2. On-demand-windows Jenkins build.
> > > > >> >> >> >> I see it as a build to which I can attach my patch and the
> > > > build
> > > > >> >> >> >> will
> > > > >> >> >> >> run my changes on a dedicated windows box.
> > > > >> >> >> >> That way people can test their changes without having
> > personal
> > > > >> >> >> >> windows
> > > > >> >> >> >> nodes.
> > > > >> >> >> >>
> > > > >> >> >> >> I think this is the minimal set of requirement for us to be
> > > > able
> > > > >> >> >> >> to
> > > > >> >> >> >> commit to the new platform.
> > > > >> >> >> >> Right now I see only one windows related build
> > > > >> >> >> >> https://builds.apache.org/view/Hadoop/job/Hadoop-1-win/
> > > > >> >> >> >> Which was failing since Sept 8, 2012 and did not run in the
> > > > last
> > > > >> >> >> >> month.
> > > > >> >> >> >>
> > > > >> >> >> >> Thanks,
> > > > >> >> >> >> --Konst
> > > > >> >> >> >>
> > > > >> >> >> >> On Thu, Feb 28, 2013 at 8:47 PM, Eric Baldeschwieler
> > > > >> >> >> >> <er...@hortonworks.com> wrote:
> > > > >> >> >> >> > +1 (non-binding)
> > > > >> >> >> >> >
> > > > >> >> >> >> > A few of observations:
> > > > >> >> >> >> >
> > > > >> >> >> >> > - Windows has actually been a supported platform for
> > Hadoop
> > > > >> >> >> >> > since
> > > > >> >> >> >> > 0.1
> > > > >> >> >> >> > .
> > > > >> >> >> >> > Doug championed supporting windows then and we've
> > continued
> > > > to
> > > > >> >> >> >> > do
> > > > >> >> >> >> > it
> > > > >> >> >> >> > with
> > > > >> >> >> >> > varying vigor over time.  To my knowledge we've never
> > made a
> > > > >> >> >> >> > decision
> > > > >> >> >> >> > to
> > > > >> >> >> >> > drop windows support.  The change here is improving our
> > > > support
> > > > >> >> >> >> > and
> > > > >> >> >> >> > dropping
> > > > >> >> >> >> > the requirement of cigwin.  We had Nutch windows users
> > on the
> > > > >> >> >> >> > list
> > > > >> >> >> >> > in
> > > > >> >> >> >> > 2006
> > > > >> >> >> >> > and we've been supporting windows FS requirements since
> > > > >> >> >> >> > inception.
> > > > >> >> >> >> >
> > > > >> >> >> >> > - A little pragmatism will go a long way.  As a community
> > > > we've
> > > > >> >> >> >> > got
> > > > >> >> >> >> > to
> > > > >> >> >> >> > stay committed to keeping hadoop simple (so it does work
> > on
> > > > >> >> >> >> > many
> > > > >> >> >> >> > platforms)
> > > > >> >> >> >> > and extending it to take advantage of key emerging
> > > > OS/hardware
> > > > >> >> >> >> > features,
> > > > >> >> >> >> > such as containers, new FSs, virtualization, flash ...
> >  We
> > > > >> >> >> >> > should
> > > > >> >> >> >> > all
> > > > >> >> >> >> > plan
> > > > >> >> >> >> > to let new features & optimizations emerge that don't
> > work
> > > > >> >> >> >> > everywhere, if
> > > > >> >> >> >> > they are compelling and central to hadoop's mission of
> > being
> > > > >> >> >> >> > THE
> > > > >> >> >> >> > best
> > > > >> >> >> >> > fabric
> > > > >> >> >> >> > for storing and processing big data.
> > > > >> >> >> >> >
> > > > >> >> >> >> > - A UI project like KDE has to deal with the MANY
> > differences
> > > > >> >> >> >> > between
> > > > >> >> >> >> > windows and linux UI APIs.  Hadoop faces no such complex
> > > > >> >> >> >> > challenge
> > > > >> >> >> >> > and hence
> > > > >> >> >> >> > can be maintained from a single codeline IMO.  It is
> > mostly
> > > > >> >> >> >> > abstracted from
> > > > >> >> >> >> > the OS APIs via Java and our design choices.  Where it
> > is not
> > > > >> >> >> >> > we
> > > > >> >> >> >> > can
> > > > >> >> >> >> > continue to add plugable abstractions.
> > > > >> >> >> >> >
> > > > >> >> >> >
> > > > >> >> >> >
> > > > >> >> >
> > > > >> >> >
> > > > >> >
> > > > >> >
> > > > >
> > > > >
> > > >
> >
> 
> 
> 
> -- 
> http://hortonworks.com/download/

Re: [Vote] Merge branch-trunk-win to trunk

[Konstantin Boudnik <co...@apache.org>]

Yes, you are right of course - the mis-merged commit is the cause. Thanks for
pointing this out!

I think it would be beneficial if we had branch-2 on going build in the
Jenkins. I will go ahead and create one tonight.

Cos

On Mon, Mar 25, 2013 at 05:09PM, Suresh Srinivas wrote:
> Adding other mailing lists I missed earlier.
> 
> Cos,
> 
> There is progress being made on that ticket. Also it has nothing to do with
> that.
> 
> Please follow the discussion here and why this happened due to an invalid
> commit that was reverted -
> https://issues.apache.org/jira/browse/HDFS-4615?focusedCommentId=13612650&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13612650
> 
> Regards,
> Suresh
> 
> 
> On Mon, Mar 25, 2013 at 1:17 PM, Konstantin Boudnik <co...@apache.org> wrote:
> 
> > It doesn't look like any progress has been done on the ticket below in the
> > last 3 weeks. And now branch-2 can't be compiled because of
> >
> >
> > hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSShell.java:[895,15]
> > WINDOWS is not public in org.apache.hadoop.fs.Path; cannot be accessed from
> > outside package
> >
> > That's exactly why I was -1'ing this...
> >   Cos
> >
> > On Mon, Mar 04, 2013 at 05:41PM, Matt Foley wrote:
> > > Thanks, gentlemen.  I've opened and taken responsibility for
> > > https://issues.apache.org/jira/browse/HADOOP-9359.  Giri Kesavan has
> > agreed
> > > to help with the parts that require Jenkins admin access.
> > >
> > > Thanks,
> > > --Matt
> > >
> > >
> > >
> > > On Mon, Mar 4, 2013 at 5:00 PM, Konstantin Shvachko <
> > shv.hadoop@gmail.com>wrote:
> > >
> > > > +1 on the merge.
> > > >
> > > > I am glad we agreed.
> > > > Having Jira to track the CI effort is a good idea.
> > > >
> > > > Thanks,
> > > > --Konstantin
> > > >
> > > > On Mon, Mar 4, 2013 at 3:29 PM, Matt Foley <mf...@hortonworks.com>
> > wrote:
> > > > > Thanks.  I agree Windows -1's in test-patch should not block commits.
> > > > >
> > > > > --Matt
> > > > >
> > > > >
> > > > >
> > > > > On Mon, Mar 4, 2013 at 2:30 PM, Konstantin Shvachko <
> > > > shv.hadoop@gmail.com>
> > > > > wrote:
> > > > >>
> > > > >> On Mon, Mar 4, 2013 at 12:22 PM, Matt Foley <mfoley@hortonworks.com
> > >
> > > > >> wrote:
> > > > >> > Konstantine, you have voted -1, and stated some requirements
> > before
> > > > >> > you'll
> > > > >> > withdraw that -1.  As I plan to do work to fulfill those
> > > > requirements, I
> > > > >> > want to make sure that what I'm proposing will, in fact, satisfy
> > you.
> > > > >> > That's why I'm asking, if we implement full "test-patch"
> > integration
> > > > for
> > > > >> > Windows, does it seem to you that that would provide adequate
> > support?
> > > > >>
> > > > >> Yes.
> > > > >>
> > > > >> > I have learned not to presume that my interpretation is correct.
> >  My
> > > > >> > interpretation of item #1 is that test-patch provides pre-commit
> > > > build,
> > > > >> > so
> > > > >> > it would satisfy item #1.  But rather than assuming that I am
> > > > >> > interpreting
> > > > >> > it correctly, I simply want your agreement that it would, or if
> > not,
> > > > >> > clarification why it won't.
> > > > >>
> > > > >> I agree it will satisfy my item #1.
> > > > >> I did not agree in my previous email, but I changed my mind based on
> > > > >> the latest discussion. I have to explain why now.
> > > > >> I was proposing nightly build because I did not want pre-commit
> > build
> > > > >> for Windows block commits to Linux. But if people are fine just
> > ignoring
> > > > >> -1s for the Windows part of the build it should be good.
> > > > >>
> > > > >> > Regarding item #2, it is also my interpretation that test-patch
> > > > provides
> > > > >> > an
> > > > >> > on-demand (perhaps 20-minutes deferred) Jenkins build and unit
> > test,
> > > > >> > with
> > > > >> > logs available to the developer, so it would satisfy item #2.  But
> > > > >> > rather
> > > > >> > than assuming that I am interpreting it correctly, I simply want
> > your
> > > > >> > agreement that it would, or if not, clarification why it won't.
> > > > >>
> > > > >> It will satisfy my item #2 in the following way:
> > > > >> I can duplicate your pre-commit build for Windows and add an input
> > > > >> parameter, which would let people run the build on their patches
> > > > >> chosen from local machine rather than attaching them to Jiras.
> > > > >>
> > > > >> Thanks,
> > > > >> --Konstantin
> > > > >>
> > > > >> > In agile terms, you are the Owner of these requirements.  Please
> > give
> > > > me
> > > > >> > owner feedback as to whether my proposed work sounds like it will
> > > > >> > satisfy
> > > > >> > the requirements.
> > > > >> >
> > > > >> > Thank you,
> > > > >> > --Matt
> > > > >> >
> > > > >> >
> > > > >> > On Sun, Mar 3, 2013 at 12:16 PM, Konstantin Shvachko
> > > > >> > <sh...@gmail.com>
> > > > >> > wrote:
> > > > >> >>
> > > > >> >> Didn't I explain in details what I am asking for?
> > > > >> >>
> > > > >> >> Thanks,
> > > > >> >> --Konst
> > > > >> >>
> > > > >> >> On Sun, Mar 3, 2013 at 11:08 AM, Matt Foley <
> > mfoley@hortonworks.com>
> > > > >> >> wrote:
> > > > >> >> > Hi Konstantin,
> > > > >> >> > I'd like to point out two things:
> > > > >> >> > First, I already committed in this thread (email of Thu, Feb
> > 28,
> > > > 2013
> > > > >> >> > at
> > > > >> >> > 6:01 PM) to providing CI for Windows builds.  So please stop
> > acting
> > > > >> >> > like
> > > > >> >> > I'm
> > > > >> >> > resisting this idea or something.
> > > > >> >> > Second, you didn't answer my question, you just kvetched about
> > the
> > > > >> >> > phrasing.
> > > > >> >> > So I ask again:
> > > > >> >> >
> > > > >> >> > Will providing full "test-patch" integration (pre-commit build
> > and
> > > > >> >> > unit
> > > > >> >> > test
> > > > >> >> > triggered by Jira "Patch Available" state) satisfy your
> > request for
> > > > >> >> > functionality #1 and #2?  Yes or no, please.
> > > > >> >> >
> > > > >> >> > Thanks,
> > > > >> >> > --Matt
> > > > >> >> >
> > > > >> >> >
> > > > >> >> > On Sat, Mar 2, 2013 at 7:32 PM, Konstantin Shvachko
> > > > >> >> > <sh...@gmail.com>
> > > > >> >> > wrote:
> > > > >> >> >>
> > > > >> >> >> Hi Matt,
> > > > >> >> >>
> > > > >> >> >> On Sat, Mar 2, 2013 at 12:32 PM, Matt Foley <
> > > > mfoley@hortonworks.com>
> > > > >> >> >> wrote:
> > > > >> >> >> > Konstantin,
> > > > >> >> >> > I would like to explore what it would take to remove this
> > > > >> >> >> > perceived
> > > > >> >> >> > impediment --
> > > > >> >> >>
> > > > >> >> >> Glad you decided to explore. Thank you.
> > > > >> >> >>
> > > > >> >> >> > although I reserve the right to argue that this is not
> > > > >> >> >> > pre-requisite to merging the cross-platform support patch.
> > > > >> >> >>
> > > > >> >> >> It's your right indeed. So as mine to question what the
> > platform
> > > > >> >> >> support means for you, which I believe remained unclear.
> > > > >> >> >> I do not impede the change as you should have noticed. My
> > > > >> >> >> requirement
> > > > >> >> >> comes from my perception of the support, which means to me
> > exactly
> > > > >> >> >> two
> > > > >> >> >> things:
> > > > >> >> >> 1. The ability to recognise the code is broken for the
> > platform
> > > > >> >> >> 2. The ability to test new patches on the platform
> > > > >> >> >> The latter is problematic, as many noticed in this thread, for
> > > > those
> > > > >> >> >> whose customary environment does not include Windows.
> > > > >> >> >>
> > > > >> >> >> > If we implemented full "test-patch" support for Windows on
> > > > trunk,
> > > > >> >> >> > would
> > > > >> >> >> > that
> > > > >> >> >> > fulfill both your items #1 and #2?  Please note that:
> > > > >> >> >> > a) Pushing the "Patch Available" button in Jira shall cause
> > a
> > > > >> >> >> > pre-commit
> > > > >> >> >> > build to start within, I believe, 20 minutes.
> > > > >> >> >> > b) That build keeps logs for both java build and unit tests
> > for
> > > > >> >> >> > several
> > > > >> >> >> > days, that are accessible to all viewers.
> > > > >> >> >>
> > > > >> >> >> In item #1 I mostly asking for the nightly build, which is
> > simpler
> > > > >> >> >> than "test-patch". The latter would be ideal from the platform
> > > > >> >> >> support
> > > > >> >> >> viewpoint, but it is for the community to decide if we want
> > to add
> > > > >> >> >> extra +3 hours to the build.
> > > > >> >> >> Nightly build in my understanding is triggered by the timer
> > rather
> > > > >> >> >> than by Jira's "submit patch" button.  On Jenkins build
> > > > >> >> >> configuration
> > > > >> >> >> you can specify it under "Build periodically".
> > > > >> >> >>
> > > > >> >> >> > So, does this provide sufficient on-demand support that we
> > don't
> > > > >> >> >> > have
> > > > >> >> >> > to
> > > > >> >> >> > implement a whole new on-demand VM support structure of some
> > > > sort
> > > > >> >> >> > for
> > > > >> >> >> > #2
> > > > >> >> >> > (which would be an extraordinary and impractical demand)?
> > > > >> >> >>
> > > > >> >> >> I did not mention VMs. Item #2 means a build, which runs
> > > > >> >> >> "test-patch"
> > > > >> >> >> target with the file specified by a user (instead of a jira
> > > > >> >> >> attachment).
> > > > >> >> >> When user clicks "Build Now" link a box is displayed where the
> > > > user
> > > > >> >> >> can enter the file path containing the patch. This can be
> > > > specified
> > > > >> >> >> in
> > > > >> >> >> the Build Configuration under "This build is parameterized" by
> > > > >> >> >> choosing AddParameter / FileParameter. The build can run on
> > the
> > > > same
> > > > >> >> >> Windows machine as the nightly build.
> > > > >> >> >> Such build will let people test their patches for Windows on
> > > > Jenkins
> > > > >> >> >> if they don't posses a license for the right version of
> > Windows.
> > > > >> >> >> I hope this will not turn into extraordinary or impractical
> > > > effort.
> > > > >> >> >>
> > > > >> >> >> Thanks,
> > > > >> >> >> --Konst
> > > > >> >> >>
> > > > >> >> >> > Thanks,
> > > > >> >> >> > --Matt
> > > > >> >> >> >
> > > > >> >> >> >
> > > > >> >> >> > On Fri, Mar 1, 2013 at 12:18 PM, Konstantin Shvachko
> > > > >> >> >> > <sh...@gmail.com>
> > > > >> >> >> > wrote:
> > > > >> >> >> >>
> > > > >> >> >> >> -1
> > > > >> >> >> >> We should have a CI infrastructure in place before we can
> > > > commit
> > > > >> >> >> >> to
> > > > >> >> >> >> supporting Windows platform.
> > > > >> >> >> >>
> > > > >> >> >> >> Eric is right Win/Cygwin was supported since day one.
> > > > >> >> >> >> I had a Windows box under my desk running nightly builds
> > back
> > > > in
> > > > >> >> >> >> 2006-07.
> > > > >> >> >> >> People were irritated but I was filing windows bugs until
> > 0.22
> > > > >> >> >> >> release.
> > > > >> >> >> >> Times changing and I am glad to see wider support for Win
> > > > >> >> >> >> platform.
> > > > >> >> >> >>
> > > > >> >> >> >> But in order to make it work you guys need to put the CI
> > > > process
> > > > >> >> >> >> in
> > > > >> >> >> >> place
> > > > >> >> >> >>
> > > > >> >> >> >> 1. windows jenkins build: could be nightly or PreCommit.
> > > > >> >> >> >> - Nightly would mean that changes can be committed to trunk
> > > > based
> > > > >> >> >> >> on
> > > > >> >> >> >> linux PreCommit build. And people will file bugs if the
> > change
> > > > >> >> >> >> broke
> > > > >> >> >> >> Windows nightly build.
> > > > >> >> >> >> - PreCommit-win build will mean automatic reporting failed
> > > > tests
> > > > >> >> >> >> to
> > > > >> >> >> >> respective jira blocking commits the same way as it is now
> > with
> > > > >> >> >> >> linux
> > > > >> >> >> >> PreCommit builds.
> > > > >> >> >> >> We should discuss which way is more efficient for
> > developers.
> > > > >> >> >> >>
> > > > >> >> >> >> 2. On-demand-windows Jenkins build.
> > > > >> >> >> >> I see it as a build to which I can attach my patch and the
> > > > build
> > > > >> >> >> >> will
> > > > >> >> >> >> run my changes on a dedicated windows box.
> > > > >> >> >> >> That way people can test their changes without having
> > personal
> > > > >> >> >> >> windows
> > > > >> >> >> >> nodes.
> > > > >> >> >> >>
> > > > >> >> >> >> I think this is the minimal set of requirement for us to be
> > > > able
> > > > >> >> >> >> to
> > > > >> >> >> >> commit to the new platform.
> > > > >> >> >> >> Right now I see only one windows related build
> > > > >> >> >> >> https://builds.apache.org/view/Hadoop/job/Hadoop-1-win/
> > > > >> >> >> >> Which was failing since Sept 8, 2012 and did not run in the
> > > > last
> > > > >> >> >> >> month.
> > > > >> >> >> >>
> > > > >> >> >> >> Thanks,
> > > > >> >> >> >> --Konst
> > > > >> >> >> >>
> > > > >> >> >> >> On Thu, Feb 28, 2013 at 8:47 PM, Eric Baldeschwieler
> > > > >> >> >> >> <er...@hortonworks.com> wrote:
> > > > >> >> >> >> > +1 (non-binding)
> > > > >> >> >> >> >
> > > > >> >> >> >> > A few of observations:
> > > > >> >> >> >> >
> > > > >> >> >> >> > - Windows has actually been a supported platform for
> > Hadoop
> > > > >> >> >> >> > since
> > > > >> >> >> >> > 0.1
> > > > >> >> >> >> > .
> > > > >> >> >> >> > Doug championed supporting windows then and we've
> > continued
> > > > to
> > > > >> >> >> >> > do
> > > > >> >> >> >> > it
> > > > >> >> >> >> > with
> > > > >> >> >> >> > varying vigor over time.  To my knowledge we've never
> > made a
> > > > >> >> >> >> > decision
> > > > >> >> >> >> > to
> > > > >> >> >> >> > drop windows support.  The change here is improving our
> > > > support
> > > > >> >> >> >> > and
> > > > >> >> >> >> > dropping
> > > > >> >> >> >> > the requirement of cigwin.  We had Nutch windows users
> > on the
> > > > >> >> >> >> > list
> > > > >> >> >> >> > in
> > > > >> >> >> >> > 2006
> > > > >> >> >> >> > and we've been supporting windows FS requirements since
> > > > >> >> >> >> > inception.
> > > > >> >> >> >> >
> > > > >> >> >> >> > - A little pragmatism will go a long way.  As a community
> > > > we've
> > > > >> >> >> >> > got
> > > > >> >> >> >> > to
> > > > >> >> >> >> > stay committed to keeping hadoop simple (so it does work
> > on
> > > > >> >> >> >> > many
> > > > >> >> >> >> > platforms)
> > > > >> >> >> >> > and extending it to take advantage of key emerging
> > > > OS/hardware
> > > > >> >> >> >> > features,
> > > > >> >> >> >> > such as containers, new FSs, virtualization, flash ...
> >  We
> > > > >> >> >> >> > should
> > > > >> >> >> >> > all
> > > > >> >> >> >> > plan
> > > > >> >> >> >> > to let new features & optimizations emerge that don't
> > work
> > > > >> >> >> >> > everywhere, if
> > > > >> >> >> >> > they are compelling and central to hadoop's mission of
> > being
> > > > >> >> >> >> > THE
> > > > >> >> >> >> > best
> > > > >> >> >> >> > fabric
> > > > >> >> >> >> > for storing and processing big data.
> > > > >> >> >> >> >
> > > > >> >> >> >> > - A UI project like KDE has to deal with the MANY
> > differences
> > > > >> >> >> >> > between
> > > > >> >> >> >> > windows and linux UI APIs.  Hadoop faces no such complex
> > > > >> >> >> >> > challenge
> > > > >> >> >> >> > and hence
> > > > >> >> >> >> > can be maintained from a single codeline IMO.  It is
> > mostly
> > > > >> >> >> >> > abstracted from
> > > > >> >> >> >> > the OS APIs via Java and our design choices.  Where it
> > is not
> > > > >> >> >> >> > we
> > > > >> >> >> >> > can
> > > > >> >> >> >> > continue to add plugable abstractions.
> > > > >> >> >> >> >
> > > > >> >> >> >
> > > > >> >> >> >
> > > > >> >> >
> > > > >> >> >
> > > > >> >
> > > > >> >
> > > > >
> > > > >
> > > >
> >
> 
> 
> 
> -- 
> http://hortonworks.com/download/

Re: [Vote] Merge branch-trunk-win to trunk

[Suresh Srinivas <su...@hortonworks.com>]

Adding other mailing lists I missed earlier.


Cos,

There is progress being made on that ticket. Also it has nothing to do with
that.

Please follow the discussion here and why this happened due to an invalid
commit that was reverted -
https://issues.apache.org/jira/browse/HDFS-4615?focusedCommentId=13612650&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13612650

Regards,
Suresh


On Mon, Mar 25, 2013 at 1:17 PM, Konstantin Boudnik <co...@apache.org> wrote:

> It doesn't look like any progress has been done on the ticket below in the
> last 3 weeks. And now branch-2 can't be compiled because of
>
>
> hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSShell.java:[895,15]
> WINDOWS is not public in org.apache.hadoop.fs.Path; cannot be accessed from
> outside package
>
> That's exactly why I was -1'ing this...
>   Cos
>
> On Mon, Mar 04, 2013 at 05:41PM, Matt Foley wrote:
> > Thanks, gentlemen.  I've opened and taken responsibility for
> > https://issues.apache.org/jira/browse/HADOOP-9359.  Giri Kesavan has
> agreed
> > to help with the parts that require Jenkins admin access.
> >
> > Thanks,
> > --Matt
> >
> >
> >
> > On Mon, Mar 4, 2013 at 5:00 PM, Konstantin Shvachko <
> shv.hadoop@gmail.com>wrote:
> >
> > > +1 on the merge.
> > >
> > > I am glad we agreed.
> > > Having Jira to track the CI effort is a good idea.
> > >
> > > Thanks,
> > > --Konstantin
> > >
> > > On Mon, Mar 4, 2013 at 3:29 PM, Matt Foley <mf...@hortonworks.com>
> wrote:
> > > > Thanks.  I agree Windows -1's in test-patch should not block commits.
> > > >
> > > > --Matt
> > > >
> > > >
> > > >
> > > > On Mon, Mar 4, 2013 at 2:30 PM, Konstantin Shvachko <
> > > shv.hadoop@gmail.com>
> > > > wrote:
> > > >>
> > > >> On Mon, Mar 4, 2013 at 12:22 PM, Matt Foley <mfoley@hortonworks.com
> >
> > > >> wrote:
> > > >> > Konstantine, you have voted -1, and stated some requirements
> before
> > > >> > you'll
> > > >> > withdraw that -1.  As I plan to do work to fulfill those
> > > requirements, I
> > > >> > want to make sure that what I'm proposing will, in fact, satisfy
> you.
> > > >> > That's why I'm asking, if we implement full "test-patch"
> integration
> > > for
> > > >> > Windows, does it seem to you that that would provide adequate
> support?
> > > >>
> > > >> Yes.
> > > >>
> > > >> > I have learned not to presume that my interpretation is correct.
>  My
> > > >> > interpretation of item #1 is that test-patch provides pre-commit
> > > build,
> > > >> > so
> > > >> > it would satisfy item #1.  But rather than assuming that I am
> > > >> > interpreting
> > > >> > it correctly, I simply want your agreement that it would, or if
> not,
> > > >> > clarification why it won't.
> > > >>
> > > >> I agree it will satisfy my item #1.
> > > >> I did not agree in my previous email, but I changed my mind based on
> > > >> the latest discussion. I have to explain why now.
> > > >> I was proposing nightly build because I did not want pre-commit
> build
> > > >> for Windows block commits to Linux. But if people are fine just
> ignoring
> > > >> -1s for the Windows part of the build it should be good.
> > > >>
> > > >> > Regarding item #2, it is also my interpretation that test-patch
> > > provides
> > > >> > an
> > > >> > on-demand (perhaps 20-minutes deferred) Jenkins build and unit
> test,
> > > >> > with
> > > >> > logs available to the developer, so it would satisfy item #2.  But
> > > >> > rather
> > > >> > than assuming that I am interpreting it correctly, I simply want
> your
> > > >> > agreement that it would, or if not, clarification why it won't.
> > > >>
> > > >> It will satisfy my item #2 in the following way:
> > > >> I can duplicate your pre-commit build for Windows and add an input
> > > >> parameter, which would let people run the build on their patches
> > > >> chosen from local machine rather than attaching them to Jiras.
> > > >>
> > > >> Thanks,
> > > >> --Konstantin
> > > >>
> > > >> > In agile terms, you are the Owner of these requirements.  Please
> give
> > > me
> > > >> > owner feedback as to whether my proposed work sounds like it will
> > > >> > satisfy
> > > >> > the requirements.
> > > >> >
> > > >> > Thank you,
> > > >> > --Matt
> > > >> >
> > > >> >
> > > >> > On Sun, Mar 3, 2013 at 12:16 PM, Konstantin Shvachko
> > > >> > <sh...@gmail.com>
> > > >> > wrote:
> > > >> >>
> > > >> >> Didn't I explain in details what I am asking for?
> > > >> >>
> > > >> >> Thanks,
> > > >> >> --Konst
> > > >> >>
> > > >> >> On Sun, Mar 3, 2013 at 11:08 AM, Matt Foley <
> mfoley@hortonworks.com>
> > > >> >> wrote:
> > > >> >> > Hi Konstantin,
> > > >> >> > I'd like to point out two things:
> > > >> >> > First, I already committed in this thread (email of Thu, Feb
> 28,
> > > 2013
> > > >> >> > at
> > > >> >> > 6:01 PM) to providing CI for Windows builds.  So please stop
> acting
> > > >> >> > like
> > > >> >> > I'm
> > > >> >> > resisting this idea or something.
> > > >> >> > Second, you didn't answer my question, you just kvetched about
> the
> > > >> >> > phrasing.
> > > >> >> > So I ask again:
> > > >> >> >
> > > >> >> > Will providing full "test-patch" integration (pre-commit build
> and
> > > >> >> > unit
> > > >> >> > test
> > > >> >> > triggered by Jira "Patch Available" state) satisfy your
> request for
> > > >> >> > functionality #1 and #2?  Yes or no, please.
> > > >> >> >
> > > >> >> > Thanks,
> > > >> >> > --Matt
> > > >> >> >
> > > >> >> >
> > > >> >> > On Sat, Mar 2, 2013 at 7:32 PM, Konstantin Shvachko
> > > >> >> > <sh...@gmail.com>
> > > >> >> > wrote:
> > > >> >> >>
> > > >> >> >> Hi Matt,
> > > >> >> >>
> > > >> >> >> On Sat, Mar 2, 2013 at 12:32 PM, Matt Foley <
> > > mfoley@hortonworks.com>
> > > >> >> >> wrote:
> > > >> >> >> > Konstantin,
> > > >> >> >> > I would like to explore what it would take to remove this
> > > >> >> >> > perceived
> > > >> >> >> > impediment --
> > > >> >> >>
> > > >> >> >> Glad you decided to explore. Thank you.
> > > >> >> >>
> > > >> >> >> > although I reserve the right to argue that this is not
> > > >> >> >> > pre-requisite to merging the cross-platform support patch.
> > > >> >> >>
> > > >> >> >> It's your right indeed. So as mine to question what the
> platform
> > > >> >> >> support means for you, which I believe remained unclear.
> > > >> >> >> I do not impede the change as you should have noticed. My
> > > >> >> >> requirement
> > > >> >> >> comes from my perception of the support, which means to me
> exactly
> > > >> >> >> two
> > > >> >> >> things:
> > > >> >> >> 1. The ability to recognise the code is broken for the
> platform
> > > >> >> >> 2. The ability to test new patches on the platform
> > > >> >> >> The latter is problematic, as many noticed in this thread, for
> > > those
> > > >> >> >> whose customary environment does not include Windows.
> > > >> >> >>
> > > >> >> >> > If we implemented full "test-patch" support for Windows on
> > > trunk,
> > > >> >> >> > would
> > > >> >> >> > that
> > > >> >> >> > fulfill both your items #1 and #2?  Please note that:
> > > >> >> >> > a) Pushing the "Patch Available" button in Jira shall cause
> a
> > > >> >> >> > pre-commit
> > > >> >> >> > build to start within, I believe, 20 minutes.
> > > >> >> >> > b) That build keeps logs for both java build and unit tests
> for
> > > >> >> >> > several
> > > >> >> >> > days, that are accessible to all viewers.
> > > >> >> >>
> > > >> >> >> In item #1 I mostly asking for the nightly build, which is
> simpler
> > > >> >> >> than "test-patch". The latter would be ideal from the platform
> > > >> >> >> support
> > > >> >> >> viewpoint, but it is for the community to decide if we want
> to add
> > > >> >> >> extra +3 hours to the build.
> > > >> >> >> Nightly build in my understanding is triggered by the timer
> rather
> > > >> >> >> than by Jira's "submit patch" button.  On Jenkins build
> > > >> >> >> configuration
> > > >> >> >> you can specify it under "Build periodically".
> > > >> >> >>
> > > >> >> >> > So, does this provide sufficient on-demand support that we
> don't
> > > >> >> >> > have
> > > >> >> >> > to
> > > >> >> >> > implement a whole new on-demand VM support structure of some
> > > sort
> > > >> >> >> > for
> > > >> >> >> > #2
> > > >> >> >> > (which would be an extraordinary and impractical demand)?
> > > >> >> >>
> > > >> >> >> I did not mention VMs. Item #2 means a build, which runs
> > > >> >> >> "test-patch"
> > > >> >> >> target with the file specified by a user (instead of a jira
> > > >> >> >> attachment).
> > > >> >> >> When user clicks "Build Now" link a box is displayed where the
> > > user
> > > >> >> >> can enter the file path containing the patch. This can be
> > > specified
> > > >> >> >> in
> > > >> >> >> the Build Configuration under "This build is parameterized" by
> > > >> >> >> choosing AddParameter / FileParameter. The build can run on
> the
> > > same
> > > >> >> >> Windows machine as the nightly build.
> > > >> >> >> Such build will let people test their patches for Windows on
> > > Jenkins
> > > >> >> >> if they don't posses a license for the right version of
> Windows.
> > > >> >> >> I hope this will not turn into extraordinary or impractical
> > > effort.
> > > >> >> >>
> > > >> >> >> Thanks,
> > > >> >> >> --Konst
> > > >> >> >>
> > > >> >> >> > Thanks,
> > > >> >> >> > --Matt
> > > >> >> >> >
> > > >> >> >> >
> > > >> >> >> > On Fri, Mar 1, 2013 at 12:18 PM, Konstantin Shvachko
> > > >> >> >> > <sh...@gmail.com>
> > > >> >> >> > wrote:
> > > >> >> >> >>
> > > >> >> >> >> -1
> > > >> >> >> >> We should have a CI infrastructure in place before we can
> > > commit
> > > >> >> >> >> to
> > > >> >> >> >> supporting Windows platform.
> > > >> >> >> >>
> > > >> >> >> >> Eric is right Win/Cygwin was supported since day one.
> > > >> >> >> >> I had a Windows box under my desk running nightly builds
> back
> > > in
> > > >> >> >> >> 2006-07.
> > > >> >> >> >> People were irritated but I was filing windows bugs until
> 0.22
> > > >> >> >> >> release.
> > > >> >> >> >> Times changing and I am glad to see wider support for Win
> > > >> >> >> >> platform.
> > > >> >> >> >>
> > > >> >> >> >> But in order to make it work you guys need to put the CI
> > > process
> > > >> >> >> >> in
> > > >> >> >> >> place
> > > >> >> >> >>
> > > >> >> >> >> 1. windows jenkins build: could be nightly or PreCommit.
> > > >> >> >> >> - Nightly would mean that changes can be committed to trunk
> > > based
> > > >> >> >> >> on
> > > >> >> >> >> linux PreCommit build. And people will file bugs if the
> change
> > > >> >> >> >> broke
> > > >> >> >> >> Windows nightly build.
> > > >> >> >> >> - PreCommit-win build will mean automatic reporting failed
> > > tests
> > > >> >> >> >> to
> > > >> >> >> >> respective jira blocking commits the same way as it is now
> with
> > > >> >> >> >> linux
> > > >> >> >> >> PreCommit builds.
> > > >> >> >> >> We should discuss which way is more efficient for
> developers.
> > > >> >> >> >>
> > > >> >> >> >> 2. On-demand-windows Jenkins build.
> > > >> >> >> >> I see it as a build to which I can attach my patch and the
> > > build
> > > >> >> >> >> will
> > > >> >> >> >> run my changes on a dedicated windows box.
> > > >> >> >> >> That way people can test their changes without having
> personal
> > > >> >> >> >> windows
> > > >> >> >> >> nodes.
> > > >> >> >> >>
> > > >> >> >> >> I think this is the minimal set of requirement for us to be
> > > able
> > > >> >> >> >> to
> > > >> >> >> >> commit to the new platform.
> > > >> >> >> >> Right now I see only one windows related build
> > > >> >> >> >> https://builds.apache.org/view/Hadoop/job/Hadoop-1-win/
> > > >> >> >> >> Which was failing since Sept 8, 2012 and did not run in the
> > > last
> > > >> >> >> >> month.
> > > >> >> >> >>
> > > >> >> >> >> Thanks,
> > > >> >> >> >> --Konst
> > > >> >> >> >>
> > > >> >> >> >> On Thu, Feb 28, 2013 at 8:47 PM, Eric Baldeschwieler
> > > >> >> >> >> <er...@hortonworks.com> wrote:
> > > >> >> >> >> > +1 (non-binding)
> > > >> >> >> >> >
> > > >> >> >> >> > A few of observations:
> > > >> >> >> >> >
> > > >> >> >> >> > - Windows has actually been a supported platform for
> Hadoop
> > > >> >> >> >> > since
> > > >> >> >> >> > 0.1
> > > >> >> >> >> > .
> > > >> >> >> >> > Doug championed supporting windows then and we've
> continued
> > > to
> > > >> >> >> >> > do
> > > >> >> >> >> > it
> > > >> >> >> >> > with
> > > >> >> >> >> > varying vigor over time.  To my knowledge we've never
> made a
> > > >> >> >> >> > decision
> > > >> >> >> >> > to
> > > >> >> >> >> > drop windows support.  The change here is improving our
> > > support
> > > >> >> >> >> > and
> > > >> >> >> >> > dropping
> > > >> >> >> >> > the requirement of cigwin.  We had Nutch windows users
> on the
> > > >> >> >> >> > list
> > > >> >> >> >> > in
> > > >> >> >> >> > 2006
> > > >> >> >> >> > and we've been supporting windows FS requirements since
> > > >> >> >> >> > inception.
> > > >> >> >> >> >
> > > >> >> >> >> > - A little pragmatism will go a long way.  As a community
> > > we've
> > > >> >> >> >> > got
> > > >> >> >> >> > to
> > > >> >> >> >> > stay committed to keeping hadoop simple (so it does work
> on
> > > >> >> >> >> > many
> > > >> >> >> >> > platforms)
> > > >> >> >> >> > and extending it to take advantage of key emerging
> > > OS/hardware
> > > >> >> >> >> > features,
> > > >> >> >> >> > such as containers, new FSs, virtualization, flash ...
>  We
> > > >> >> >> >> > should
> > > >> >> >> >> > all
> > > >> >> >> >> > plan
> > > >> >> >> >> > to let new features & optimizations emerge that don't
> work
> > > >> >> >> >> > everywhere, if
> > > >> >> >> >> > they are compelling and central to hadoop's mission of
> being
> > > >> >> >> >> > THE
> > > >> >> >> >> > best
> > > >> >> >> >> > fabric
> > > >> >> >> >> > for storing and processing big data.
> > > >> >> >> >> >
> > > >> >> >> >> > - A UI project like KDE has to deal with the MANY
> differences
> > > >> >> >> >> > between
> > > >> >> >> >> > windows and linux UI APIs.  Hadoop faces no such complex
> > > >> >> >> >> > challenge
> > > >> >> >> >> > and hence
> > > >> >> >> >> > can be maintained from a single codeline IMO.  It is
> mostly
> > > >> >> >> >> > abstracted from
> > > >> >> >> >> > the OS APIs via Java and our design choices.  Where it
> is not
> > > >> >> >> >> > we
> > > >> >> >> >> > can
> > > >> >> >> >> > continue to add plugable abstractions.
> > > >> >> >> >> >
> > > >> >> >> >
> > > >> >> >> >
> > > >> >> >
> > > >> >> >
> > > >> >
> > > >> >
> > > >
> > > >
> > >
>



-- 
http://hortonworks.com/download/

Re: [Vote] Merge branch-trunk-win to trunk

[Suresh Srinivas <su...@hortonworks.com>]

Adding other mailing lists I missed earlier.


Cos,

There is progress being made on that ticket. Also it has nothing to do with
that.

Please follow the discussion here and why this happened due to an invalid
commit that was reverted -
https://issues.apache.org/jira/browse/HDFS-4615?focusedCommentId=13612650&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13612650

Regards,
Suresh


On Mon, Mar 25, 2013 at 1:17 PM, Konstantin Boudnik <co...@apache.org> wrote:

> It doesn't look like any progress has been done on the ticket below in the
> last 3 weeks. And now branch-2 can't be compiled because of
>
>
> hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSShell.java:[895,15]
> WINDOWS is not public in org.apache.hadoop.fs.Path; cannot be accessed from
> outside package
>
> That's exactly why I was -1'ing this...
>   Cos
>
> On Mon, Mar 04, 2013 at 05:41PM, Matt Foley wrote:
> > Thanks, gentlemen.  I've opened and taken responsibility for
> > https://issues.apache.org/jira/browse/HADOOP-9359.  Giri Kesavan has
> agreed
> > to help with the parts that require Jenkins admin access.
> >
> > Thanks,
> > --Matt
> >
> >
> >
> > On Mon, Mar 4, 2013 at 5:00 PM, Konstantin Shvachko <
> shv.hadoop@gmail.com>wrote:
> >
> > > +1 on the merge.
> > >
> > > I am glad we agreed.
> > > Having Jira to track the CI effort is a good idea.
> > >
> > > Thanks,
> > > --Konstantin
> > >
> > > On Mon, Mar 4, 2013 at 3:29 PM, Matt Foley <mf...@hortonworks.com>
> wrote:
> > > > Thanks.  I agree Windows -1's in test-patch should not block commits.
> > > >
> > > > --Matt
> > > >
> > > >
> > > >
> > > > On Mon, Mar 4, 2013 at 2:30 PM, Konstantin Shvachko <
> > > shv.hadoop@gmail.com>
> > > > wrote:
> > > >>
> > > >> On Mon, Mar 4, 2013 at 12:22 PM, Matt Foley <mfoley@hortonworks.com
> >
> > > >> wrote:
> > > >> > Konstantine, you have voted -1, and stated some requirements
> before
> > > >> > you'll
> > > >> > withdraw that -1.  As I plan to do work to fulfill those
> > > requirements, I
> > > >> > want to make sure that what I'm proposing will, in fact, satisfy
> you.
> > > >> > That's why I'm asking, if we implement full "test-patch"
> integration
> > > for
> > > >> > Windows, does it seem to you that that would provide adequate
> support?
> > > >>
> > > >> Yes.
> > > >>
> > > >> > I have learned not to presume that my interpretation is correct.
>  My
> > > >> > interpretation of item #1 is that test-patch provides pre-commit
> > > build,
> > > >> > so
> > > >> > it would satisfy item #1.  But rather than assuming that I am
> > > >> > interpreting
> > > >> > it correctly, I simply want your agreement that it would, or if
> not,
> > > >> > clarification why it won't.
> > > >>
> > > >> I agree it will satisfy my item #1.
> > > >> I did not agree in my previous email, but I changed my mind based on
> > > >> the latest discussion. I have to explain why now.
> > > >> I was proposing nightly build because I did not want pre-commit
> build
> > > >> for Windows block commits to Linux. But if people are fine just
> ignoring
> > > >> -1s for the Windows part of the build it should be good.
> > > >>
> > > >> > Regarding item #2, it is also my interpretation that test-patch
> > > provides
> > > >> > an
> > > >> > on-demand (perhaps 20-minutes deferred) Jenkins build and unit
> test,
> > > >> > with
> > > >> > logs available to the developer, so it would satisfy item #2.  But
> > > >> > rather
> > > >> > than assuming that I am interpreting it correctly, I simply want
> your
> > > >> > agreement that it would, or if not, clarification why it won't.
> > > >>
> > > >> It will satisfy my item #2 in the following way:
> > > >> I can duplicate your pre-commit build for Windows and add an input
> > > >> parameter, which would let people run the build on their patches
> > > >> chosen from local machine rather than attaching them to Jiras.
> > > >>
> > > >> Thanks,
> > > >> --Konstantin
> > > >>
> > > >> > In agile terms, you are the Owner of these requirements.  Please
> give
> > > me
> > > >> > owner feedback as to whether my proposed work sounds like it will
> > > >> > satisfy
> > > >> > the requirements.
> > > >> >
> > > >> > Thank you,
> > > >> > --Matt
> > > >> >
> > > >> >
> > > >> > On Sun, Mar 3, 2013 at 12:16 PM, Konstantin Shvachko
> > > >> > <sh...@gmail.com>
> > > >> > wrote:
> > > >> >>
> > > >> >> Didn't I explain in details what I am asking for?
> > > >> >>
> > > >> >> Thanks,
> > > >> >> --Konst
> > > >> >>
> > > >> >> On Sun, Mar 3, 2013 at 11:08 AM, Matt Foley <
> mfoley@hortonworks.com>
> > > >> >> wrote:
> > > >> >> > Hi Konstantin,
> > > >> >> > I'd like to point out two things:
> > > >> >> > First, I already committed in this thread (email of Thu, Feb
> 28,
> > > 2013
> > > >> >> > at
> > > >> >> > 6:01 PM) to providing CI for Windows builds.  So please stop
> acting
> > > >> >> > like
> > > >> >> > I'm
> > > >> >> > resisting this idea or something.
> > > >> >> > Second, you didn't answer my question, you just kvetched about
> the
> > > >> >> > phrasing.
> > > >> >> > So I ask again:
> > > >> >> >
> > > >> >> > Will providing full "test-patch" integration (pre-commit build
> and
> > > >> >> > unit
> > > >> >> > test
> > > >> >> > triggered by Jira "Patch Available" state) satisfy your
> request for
> > > >> >> > functionality #1 and #2?  Yes or no, please.
> > > >> >> >
> > > >> >> > Thanks,
> > > >> >> > --Matt
> > > >> >> >
> > > >> >> >
> > > >> >> > On Sat, Mar 2, 2013 at 7:32 PM, Konstantin Shvachko
> > > >> >> > <sh...@gmail.com>
> > > >> >> > wrote:
> > > >> >> >>
> > > >> >> >> Hi Matt,
> > > >> >> >>
> > > >> >> >> On Sat, Mar 2, 2013 at 12:32 PM, Matt Foley <
> > > mfoley@hortonworks.com>
> > > >> >> >> wrote:
> > > >> >> >> > Konstantin,
> > > >> >> >> > I would like to explore what it would take to remove this
> > > >> >> >> > perceived
> > > >> >> >> > impediment --
> > > >> >> >>
> > > >> >> >> Glad you decided to explore. Thank you.
> > > >> >> >>
> > > >> >> >> > although I reserve the right to argue that this is not
> > > >> >> >> > pre-requisite to merging the cross-platform support patch.
> > > >> >> >>
> > > >> >> >> It's your right indeed. So as mine to question what the
> platform
> > > >> >> >> support means for you, which I believe remained unclear.
> > > >> >> >> I do not impede the change as you should have noticed. My
> > > >> >> >> requirement
> > > >> >> >> comes from my perception of the support, which means to me
> exactly
> > > >> >> >> two
> > > >> >> >> things:
> > > >> >> >> 1. The ability to recognise the code is broken for the
> platform
> > > >> >> >> 2. The ability to test new patches on the platform
> > > >> >> >> The latter is problematic, as many noticed in this thread, for
> > > those
> > > >> >> >> whose customary environment does not include Windows.
> > > >> >> >>
> > > >> >> >> > If we implemented full "test-patch" support for Windows on
> > > trunk,
> > > >> >> >> > would
> > > >> >> >> > that
> > > >> >> >> > fulfill both your items #1 and #2?  Please note that:
> > > >> >> >> > a) Pushing the "Patch Available" button in Jira shall cause
> a
> > > >> >> >> > pre-commit
> > > >> >> >> > build to start within, I believe, 20 minutes.
> > > >> >> >> > b) That build keeps logs for both java build and unit tests
> for
> > > >> >> >> > several
> > > >> >> >> > days, that are accessible to all viewers.
> > > >> >> >>
> > > >> >> >> In item #1 I mostly asking for the nightly build, which is
> simpler
> > > >> >> >> than "test-patch". The latter would be ideal from the platform
> > > >> >> >> support
> > > >> >> >> viewpoint, but it is for the community to decide if we want
> to add
> > > >> >> >> extra +3 hours to the build.
> > > >> >> >> Nightly build in my understanding is triggered by the timer
> rather
> > > >> >> >> than by Jira's "submit patch" button.  On Jenkins build
> > > >> >> >> configuration
> > > >> >> >> you can specify it under "Build periodically".
> > > >> >> >>
> > > >> >> >> > So, does this provide sufficient on-demand support that we
> don't
> > > >> >> >> > have
> > > >> >> >> > to
> > > >> >> >> > implement a whole new on-demand VM support structure of some
> > > sort
> > > >> >> >> > for
> > > >> >> >> > #2
> > > >> >> >> > (which would be an extraordinary and impractical demand)?
> > > >> >> >>
> > > >> >> >> I did not mention VMs. Item #2 means a build, which runs
> > > >> >> >> "test-patch"
> > > >> >> >> target with the file specified by a user (instead of a jira
> > > >> >> >> attachment).
> > > >> >> >> When user clicks "Build Now" link a box is displayed where the
> > > user
> > > >> >> >> can enter the file path containing the patch. This can be
> > > specified
> > > >> >> >> in
> > > >> >> >> the Build Configuration under "This build is parameterized" by
> > > >> >> >> choosing AddParameter / FileParameter. The build can run on
> the
> > > same
> > > >> >> >> Windows machine as the nightly build.
> > > >> >> >> Such build will let people test their patches for Windows on
> > > Jenkins
> > > >> >> >> if they don't posses a license for the right version of
> Windows.
> > > >> >> >> I hope this will not turn into extraordinary or impractical
> > > effort.
> > > >> >> >>
> > > >> >> >> Thanks,
> > > >> >> >> --Konst
> > > >> >> >>
> > > >> >> >> > Thanks,
> > > >> >> >> > --Matt
> > > >> >> >> >
> > > >> >> >> >
> > > >> >> >> > On Fri, Mar 1, 2013 at 12:18 PM, Konstantin Shvachko
> > > >> >> >> > <sh...@gmail.com>
> > > >> >> >> > wrote:
> > > >> >> >> >>
> > > >> >> >> >> -1
> > > >> >> >> >> We should have a CI infrastructure in place before we can
> > > commit
> > > >> >> >> >> to
> > > >> >> >> >> supporting Windows platform.
> > > >> >> >> >>
> > > >> >> >> >> Eric is right Win/Cygwin was supported since day one.
> > > >> >> >> >> I had a Windows box under my desk running nightly builds
> back
> > > in
> > > >> >> >> >> 2006-07.
> > > >> >> >> >> People were irritated but I was filing windows bugs until
> 0.22
> > > >> >> >> >> release.
> > > >> >> >> >> Times changing and I am glad to see wider support for Win
> > > >> >> >> >> platform.
> > > >> >> >> >>
> > > >> >> >> >> But in order to make it work you guys need to put the CI
> > > process
> > > >> >> >> >> in
> > > >> >> >> >> place
> > > >> >> >> >>
> > > >> >> >> >> 1. windows jenkins build: could be nightly or PreCommit.
> > > >> >> >> >> - Nightly would mean that changes can be committed to trunk
> > > based
> > > >> >> >> >> on
> > > >> >> >> >> linux PreCommit build. And people will file bugs if the
> change
> > > >> >> >> >> broke
> > > >> >> >> >> Windows nightly build.
> > > >> >> >> >> - PreCommit-win build will mean automatic reporting failed
> > > tests
> > > >> >> >> >> to
> > > >> >> >> >> respective jira blocking commits the same way as it is now
> with
> > > >> >> >> >> linux
> > > >> >> >> >> PreCommit builds.
> > > >> >> >> >> We should discuss which way is more efficient for
> developers.
> > > >> >> >> >>
> > > >> >> >> >> 2. On-demand-windows Jenkins build.
> > > >> >> >> >> I see it as a build to which I can attach my patch and the
> > > build
> > > >> >> >> >> will
> > > >> >> >> >> run my changes on a dedicated windows box.
> > > >> >> >> >> That way people can test their changes without having
> personal
> > > >> >> >> >> windows
> > > >> >> >> >> nodes.
> > > >> >> >> >>
> > > >> >> >> >> I think this is the minimal set of requirement for us to be
> > > able
> > > >> >> >> >> to
> > > >> >> >> >> commit to the new platform.
> > > >> >> >> >> Right now I see only one windows related build
> > > >> >> >> >> https://builds.apache.org/view/Hadoop/job/Hadoop-1-win/
> > > >> >> >> >> Which was failing since Sept 8, 2012 and did not run in the
> > > last
> > > >> >> >> >> month.
> > > >> >> >> >>
> > > >> >> >> >> Thanks,
> > > >> >> >> >> --Konst
> > > >> >> >> >>
> > > >> >> >> >> On Thu, Feb 28, 2013 at 8:47 PM, Eric Baldeschwieler
> > > >> >> >> >> <er...@hortonworks.com> wrote:
> > > >> >> >> >> > +1 (non-binding)
> > > >> >> >> >> >
> > > >> >> >> >> > A few of observations:
> > > >> >> >> >> >
> > > >> >> >> >> > - Windows has actually been a supported platform for
> Hadoop
> > > >> >> >> >> > since
> > > >> >> >> >> > 0.1
> > > >> >> >> >> > .
> > > >> >> >> >> > Doug championed supporting windows then and we've
> continued
> > > to
> > > >> >> >> >> > do
> > > >> >> >> >> > it
> > > >> >> >> >> > with
> > > >> >> >> >> > varying vigor over time.  To my knowledge we've never
> made a
> > > >> >> >> >> > decision
> > > >> >> >> >> > to
> > > >> >> >> >> > drop windows support.  The change here is improving our
> > > support
> > > >> >> >> >> > and
> > > >> >> >> >> > dropping
> > > >> >> >> >> > the requirement of cigwin.  We had Nutch windows users
> on the
> > > >> >> >> >> > list
> > > >> >> >> >> > in
> > > >> >> >> >> > 2006
> > > >> >> >> >> > and we've been supporting windows FS requirements since
> > > >> >> >> >> > inception.
> > > >> >> >> >> >
> > > >> >> >> >> > - A little pragmatism will go a long way.  As a community
> > > we've
> > > >> >> >> >> > got
> > > >> >> >> >> > to
> > > >> >> >> >> > stay committed to keeping hadoop simple (so it does work
> on
> > > >> >> >> >> > many
> > > >> >> >> >> > platforms)
> > > >> >> >> >> > and extending it to take advantage of key emerging
> > > OS/hardware
> > > >> >> >> >> > features,
> > > >> >> >> >> > such as containers, new FSs, virtualization, flash ...
>  We
> > > >> >> >> >> > should
> > > >> >> >> >> > all
> > > >> >> >> >> > plan
> > > >> >> >> >> > to let new features & optimizations emerge that don't
> work
> > > >> >> >> >> > everywhere, if
> > > >> >> >> >> > they are compelling and central to hadoop's mission of
> being
> > > >> >> >> >> > THE
> > > >> >> >> >> > best
> > > >> >> >> >> > fabric
> > > >> >> >> >> > for storing and processing big data.
> > > >> >> >> >> >
> > > >> >> >> >> > - A UI project like KDE has to deal with the MANY
> differences
> > > >> >> >> >> > between
> > > >> >> >> >> > windows and linux UI APIs.  Hadoop faces no such complex
> > > >> >> >> >> > challenge
> > > >> >> >> >> > and hence
> > > >> >> >> >> > can be maintained from a single codeline IMO.  It is
> mostly
> > > >> >> >> >> > abstracted from
> > > >> >> >> >> > the OS APIs via Java and our design choices.  Where it
> is not
> > > >> >> >> >> > we
> > > >> >> >> >> > can
> > > >> >> >> >> > continue to add plugable abstractions.
> > > >> >> >> >> >
> > > >> >> >> >
> > > >> >> >> >
> > > >> >> >
> > > >> >> >
> > > >> >
> > > >> >
> > > >
> > > >
> > >
>



-- 
http://hortonworks.com/download/

Re: [Vote] Merge branch-trunk-win to trunk

[Suresh Srinivas <su...@hortonworks.com>]

Adding other mailing lists I missed earlier.


Cos,

There is progress being made on that ticket. Also it has nothing to do with
that.

Please follow the discussion here and why this happened due to an invalid
commit that was reverted -
https://issues.apache.org/jira/browse/HDFS-4615?focusedCommentId=13612650&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13612650

Regards,
Suresh


On Mon, Mar 25, 2013 at 1:17 PM, Konstantin Boudnik <co...@apache.org> wrote:

> It doesn't look like any progress has been done on the ticket below in the
> last 3 weeks. And now branch-2 can't be compiled because of
>
>
> hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSShell.java:[895,15]
> WINDOWS is not public in org.apache.hadoop.fs.Path; cannot be accessed from
> outside package
>
> That's exactly why I was -1'ing this...
>   Cos
>
> On Mon, Mar 04, 2013 at 05:41PM, Matt Foley wrote:
> > Thanks, gentlemen.  I've opened and taken responsibility for
> > https://issues.apache.org/jira/browse/HADOOP-9359.  Giri Kesavan has
> agreed
> > to help with the parts that require Jenkins admin access.
> >
> > Thanks,
> > --Matt
> >
> >
> >
> > On Mon, Mar 4, 2013 at 5:00 PM, Konstantin Shvachko <
> shv.hadoop@gmail.com>wrote:
> >
> > > +1 on the merge.
> > >
> > > I am glad we agreed.
> > > Having Jira to track the CI effort is a good idea.
> > >
> > > Thanks,
> > > --Konstantin
> > >
> > > On Mon, Mar 4, 2013 at 3:29 PM, Matt Foley <mf...@hortonworks.com>
> wrote:
> > > > Thanks.  I agree Windows -1's in test-patch should not block commits.
> > > >
> > > > --Matt
> > > >
> > > >
> > > >
> > > > On Mon, Mar 4, 2013 at 2:30 PM, Konstantin Shvachko <
> > > shv.hadoop@gmail.com>
> > > > wrote:
> > > >>
> > > >> On Mon, Mar 4, 2013 at 12:22 PM, Matt Foley <mfoley@hortonworks.com
> >
> > > >> wrote:
> > > >> > Konstantine, you have voted -1, and stated some requirements
> before
> > > >> > you'll
> > > >> > withdraw that -1.  As I plan to do work to fulfill those
> > > requirements, I
> > > >> > want to make sure that what I'm proposing will, in fact, satisfy
> you.
> > > >> > That's why I'm asking, if we implement full "test-patch"
> integration
> > > for
> > > >> > Windows, does it seem to you that that would provide adequate
> support?
> > > >>
> > > >> Yes.
> > > >>
> > > >> > I have learned not to presume that my interpretation is correct.
>  My
> > > >> > interpretation of item #1 is that test-patch provides pre-commit
> > > build,
> > > >> > so
> > > >> > it would satisfy item #1.  But rather than assuming that I am
> > > >> > interpreting
> > > >> > it correctly, I simply want your agreement that it would, or if
> not,
> > > >> > clarification why it won't.
> > > >>
> > > >> I agree it will satisfy my item #1.
> > > >> I did not agree in my previous email, but I changed my mind based on
> > > >> the latest discussion. I have to explain why now.
> > > >> I was proposing nightly build because I did not want pre-commit
> build
> > > >> for Windows block commits to Linux. But if people are fine just
> ignoring
> > > >> -1s for the Windows part of the build it should be good.
> > > >>
> > > >> > Regarding item #2, it is also my interpretation that test-patch
> > > provides
> > > >> > an
> > > >> > on-demand (perhaps 20-minutes deferred) Jenkins build and unit
> test,
> > > >> > with
> > > >> > logs available to the developer, so it would satisfy item #2.  But
> > > >> > rather
> > > >> > than assuming that I am interpreting it correctly, I simply want
> your
> > > >> > agreement that it would, or if not, clarification why it won't.
> > > >>
> > > >> It will satisfy my item #2 in the following way:
> > > >> I can duplicate your pre-commit build for Windows and add an input
> > > >> parameter, which would let people run the build on their patches
> > > >> chosen from local machine rather than attaching them to Jiras.
> > > >>
> > > >> Thanks,
> > > >> --Konstantin
> > > >>
> > > >> > In agile terms, you are the Owner of these requirements.  Please
> give
> > > me
> > > >> > owner feedback as to whether my proposed work sounds like it will
> > > >> > satisfy
> > > >> > the requirements.
> > > >> >
> > > >> > Thank you,
> > > >> > --Matt
> > > >> >
> > > >> >
> > > >> > On Sun, Mar 3, 2013 at 12:16 PM, Konstantin Shvachko
> > > >> > <sh...@gmail.com>
> > > >> > wrote:
> > > >> >>
> > > >> >> Didn't I explain in details what I am asking for?
> > > >> >>
> > > >> >> Thanks,
> > > >> >> --Konst
> > > >> >>
> > > >> >> On Sun, Mar 3, 2013 at 11:08 AM, Matt Foley <
> mfoley@hortonworks.com>
> > > >> >> wrote:
> > > >> >> > Hi Konstantin,
> > > >> >> > I'd like to point out two things:
> > > >> >> > First, I already committed in this thread (email of Thu, Feb
> 28,
> > > 2013
> > > >> >> > at
> > > >> >> > 6:01 PM) to providing CI for Windows builds.  So please stop
> acting
> > > >> >> > like
> > > >> >> > I'm
> > > >> >> > resisting this idea or something.
> > > >> >> > Second, you didn't answer my question, you just kvetched about
> the
> > > >> >> > phrasing.
> > > >> >> > So I ask again:
> > > >> >> >
> > > >> >> > Will providing full "test-patch" integration (pre-commit build
> and
> > > >> >> > unit
> > > >> >> > test
> > > >> >> > triggered by Jira "Patch Available" state) satisfy your
> request for
> > > >> >> > functionality #1 and #2?  Yes or no, please.
> > > >> >> >
> > > >> >> > Thanks,
> > > >> >> > --Matt
> > > >> >> >
> > > >> >> >
> > > >> >> > On Sat, Mar 2, 2013 at 7:32 PM, Konstantin Shvachko
> > > >> >> > <sh...@gmail.com>
> > > >> >> > wrote:
> > > >> >> >>
> > > >> >> >> Hi Matt,
> > > >> >> >>
> > > >> >> >> On Sat, Mar 2, 2013 at 12:32 PM, Matt Foley <
> > > mfoley@hortonworks.com>
> > > >> >> >> wrote:
> > > >> >> >> > Konstantin,
> > > >> >> >> > I would like to explore what it would take to remove this
> > > >> >> >> > perceived
> > > >> >> >> > impediment --
> > > >> >> >>
> > > >> >> >> Glad you decided to explore. Thank you.
> > > >> >> >>
> > > >> >> >> > although I reserve the right to argue that this is not
> > > >> >> >> > pre-requisite to merging the cross-platform support patch.
> > > >> >> >>
> > > >> >> >> It's your right indeed. So as mine to question what the
> platform
> > > >> >> >> support means for you, which I believe remained unclear.
> > > >> >> >> I do not impede the change as you should have noticed. My
> > > >> >> >> requirement
> > > >> >> >> comes from my perception of the support, which means to me
> exactly
> > > >> >> >> two
> > > >> >> >> things:
> > > >> >> >> 1. The ability to recognise the code is broken for the
> platform
> > > >> >> >> 2. The ability to test new patches on the platform
> > > >> >> >> The latter is problematic, as many noticed in this thread, for
> > > those
> > > >> >> >> whose customary environment does not include Windows.
> > > >> >> >>
> > > >> >> >> > If we implemented full "test-patch" support for Windows on
> > > trunk,
> > > >> >> >> > would
> > > >> >> >> > that
> > > >> >> >> > fulfill both your items #1 and #2?  Please note that:
> > > >> >> >> > a) Pushing the "Patch Available" button in Jira shall cause
> a
> > > >> >> >> > pre-commit
> > > >> >> >> > build to start within, I believe, 20 minutes.
> > > >> >> >> > b) That build keeps logs for both java build and unit tests
> for
> > > >> >> >> > several
> > > >> >> >> > days, that are accessible to all viewers.
> > > >> >> >>
> > > >> >> >> In item #1 I mostly asking for the nightly build, which is
> simpler
> > > >> >> >> than "test-patch". The latter would be ideal from the platform
> > > >> >> >> support
> > > >> >> >> viewpoint, but it is for the community to decide if we want
> to add
> > > >> >> >> extra +3 hours to the build.
> > > >> >> >> Nightly build in my understanding is triggered by the timer
> rather
> > > >> >> >> than by Jira's "submit patch" button.  On Jenkins build
> > > >> >> >> configuration
> > > >> >> >> you can specify it under "Build periodically".
> > > >> >> >>
> > > >> >> >> > So, does this provide sufficient on-demand support that we
> don't
> > > >> >> >> > have
> > > >> >> >> > to
> > > >> >> >> > implement a whole new on-demand VM support structure of some
> > > sort
> > > >> >> >> > for
> > > >> >> >> > #2
> > > >> >> >> > (which would be an extraordinary and impractical demand)?
> > > >> >> >>
> > > >> >> >> I did not mention VMs. Item #2 means a build, which runs
> > > >> >> >> "test-patch"
> > > >> >> >> target with the file specified by a user (instead of a jira
> > > >> >> >> attachment).
> > > >> >> >> When user clicks "Build Now" link a box is displayed where the
> > > user
> > > >> >> >> can enter the file path containing the patch. This can be
> > > specified
> > > >> >> >> in
> > > >> >> >> the Build Configuration under "This build is parameterized" by
> > > >> >> >> choosing AddParameter / FileParameter. The build can run on
> the
> > > same
> > > >> >> >> Windows machine as the nightly build.
> > > >> >> >> Such build will let people test their patches for Windows on
> > > Jenkins
> > > >> >> >> if they don't posses a license for the right version of
> Windows.
> > > >> >> >> I hope this will not turn into extraordinary or impractical
> > > effort.
> > > >> >> >>
> > > >> >> >> Thanks,
> > > >> >> >> --Konst
> > > >> >> >>
> > > >> >> >> > Thanks,
> > > >> >> >> > --Matt
> > > >> >> >> >
> > > >> >> >> >
> > > >> >> >> > On Fri, Mar 1, 2013 at 12:18 PM, Konstantin Shvachko
> > > >> >> >> > <sh...@gmail.com>
> > > >> >> >> > wrote:
> > > >> >> >> >>
> > > >> >> >> >> -1
> > > >> >> >> >> We should have a CI infrastructure in place before we can
> > > commit
> > > >> >> >> >> to
> > > >> >> >> >> supporting Windows platform.
> > > >> >> >> >>
> > > >> >> >> >> Eric is right Win/Cygwin was supported since day one.
> > > >> >> >> >> I had a Windows box under my desk running nightly builds
> back
> > > in
> > > >> >> >> >> 2006-07.
> > > >> >> >> >> People were irritated but I was filing windows bugs until
> 0.22
> > > >> >> >> >> release.
> > > >> >> >> >> Times changing and I am glad to see wider support for Win
> > > >> >> >> >> platform.
> > > >> >> >> >>
> > > >> >> >> >> But in order to make it work you guys need to put the CI
> > > process
> > > >> >> >> >> in
> > > >> >> >> >> place
> > > >> >> >> >>
> > > >> >> >> >> 1. windows jenkins build: could be nightly or PreCommit.
> > > >> >> >> >> - Nightly would mean that changes can be committed to trunk
> > > based
> > > >> >> >> >> on
> > > >> >> >> >> linux PreCommit build. And people will file bugs if the
> change
> > > >> >> >> >> broke
> > > >> >> >> >> Windows nightly build.
> > > >> >> >> >> - PreCommit-win build will mean automatic reporting failed
> > > tests
> > > >> >> >> >> to
> > > >> >> >> >> respective jira blocking commits the same way as it is now
> with
> > > >> >> >> >> linux
> > > >> >> >> >> PreCommit builds.
> > > >> >> >> >> We should discuss which way is more efficient for
> developers.
> > > >> >> >> >>
> > > >> >> >> >> 2. On-demand-windows Jenkins build.
> > > >> >> >> >> I see it as a build to which I can attach my patch and the
> > > build
> > > >> >> >> >> will
> > > >> >> >> >> run my changes on a dedicated windows box.
> > > >> >> >> >> That way people can test their changes without having
> personal
> > > >> >> >> >> windows
> > > >> >> >> >> nodes.
> > > >> >> >> >>
> > > >> >> >> >> I think this is the minimal set of requirement for us to be
> > > able
> > > >> >> >> >> to
> > > >> >> >> >> commit to the new platform.
> > > >> >> >> >> Right now I see only one windows related build
> > > >> >> >> >> https://builds.apache.org/view/Hadoop/job/Hadoop-1-win/
> > > >> >> >> >> Which was failing since Sept 8, 2012 and did not run in the
> > > last
> > > >> >> >> >> month.
> > > >> >> >> >>
> > > >> >> >> >> Thanks,
> > > >> >> >> >> --Konst
> > > >> >> >> >>
> > > >> >> >> >> On Thu, Feb 28, 2013 at 8:47 PM, Eric Baldeschwieler
> > > >> >> >> >> <er...@hortonworks.com> wrote:
> > > >> >> >> >> > +1 (non-binding)
> > > >> >> >> >> >
> > > >> >> >> >> > A few of observations:
> > > >> >> >> >> >
> > > >> >> >> >> > - Windows has actually been a supported platform for
> Hadoop
> > > >> >> >> >> > since
> > > >> >> >> >> > 0.1
> > > >> >> >> >> > .
> > > >> >> >> >> > Doug championed supporting windows then and we've
> continued
> > > to
> > > >> >> >> >> > do
> > > >> >> >> >> > it
> > > >> >> >> >> > with
> > > >> >> >> >> > varying vigor over time.  To my knowledge we've never
> made a
> > > >> >> >> >> > decision
> > > >> >> >> >> > to
> > > >> >> >> >> > drop windows support.  The change here is improving our
> > > support
> > > >> >> >> >> > and
> > > >> >> >> >> > dropping
> > > >> >> >> >> > the requirement of cigwin.  We had Nutch windows users
> on the
> > > >> >> >> >> > list
> > > >> >> >> >> > in
> > > >> >> >> >> > 2006
> > > >> >> >> >> > and we've been supporting windows FS requirements since
> > > >> >> >> >> > inception.
> > > >> >> >> >> >
> > > >> >> >> >> > - A little pragmatism will go a long way.  As a community
> > > we've
> > > >> >> >> >> > got
> > > >> >> >> >> > to
> > > >> >> >> >> > stay committed to keeping hadoop simple (so it does work
> on
> > > >> >> >> >> > many
> > > >> >> >> >> > platforms)
> > > >> >> >> >> > and extending it to take advantage of key emerging
> > > OS/hardware
> > > >> >> >> >> > features,
> > > >> >> >> >> > such as containers, new FSs, virtualization, flash ...
>  We
> > > >> >> >> >> > should
> > > >> >> >> >> > all
> > > >> >> >> >> > plan
> > > >> >> >> >> > to let new features & optimizations emerge that don't
> work
> > > >> >> >> >> > everywhere, if
> > > >> >> >> >> > they are compelling and central to hadoop's mission of
> being
> > > >> >> >> >> > THE
> > > >> >> >> >> > best
> > > >> >> >> >> > fabric
> > > >> >> >> >> > for storing and processing big data.
> > > >> >> >> >> >
> > > >> >> >> >> > - A UI project like KDE has to deal with the MANY
> differences
> > > >> >> >> >> > between
> > > >> >> >> >> > windows and linux UI APIs.  Hadoop faces no such complex
> > > >> >> >> >> > challenge
> > > >> >> >> >> > and hence
> > > >> >> >> >> > can be maintained from a single codeline IMO.  It is
> mostly
> > > >> >> >> >> > abstracted from
> > > >> >> >> >> > the OS APIs via Java and our design choices.  Where it
> is not
> > > >> >> >> >> > we
> > > >> >> >> >> > can
> > > >> >> >> >> > continue to add plugable abstractions.
> > > >> >> >> >> >
> > > >> >> >> >
> > > >> >> >> >
> > > >> >> >
> > > >> >> >
> > > >> >
> > > >> >
> > > >
> > > >
> > >
>



-- 
http://hortonworks.com/download/

Re: [Vote] Merge branch-trunk-win to trunk

[Suresh Srinivas <su...@hortonworks.com>]

Adding other mailing lists I missed earlier.


Cos,

There is progress being made on that ticket. Also it has nothing to do with
that.

Please follow the discussion here and why this happened due to an invalid
commit that was reverted -
https://issues.apache.org/jira/browse/HDFS-4615?focusedCommentId=13612650&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13612650

Regards,
Suresh


On Mon, Mar 25, 2013 at 1:17 PM, Konstantin Boudnik <co...@apache.org> wrote:

> It doesn't look like any progress has been done on the ticket below in the
> last 3 weeks. And now branch-2 can't be compiled because of
>
>
> hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSShell.java:[895,15]
> WINDOWS is not public in org.apache.hadoop.fs.Path; cannot be accessed from
> outside package
>
> That's exactly why I was -1'ing this...
>   Cos
>
> On Mon, Mar 04, 2013 at 05:41PM, Matt Foley wrote:
> > Thanks, gentlemen.  I've opened and taken responsibility for
> > https://issues.apache.org/jira/browse/HADOOP-9359.  Giri Kesavan has
> agreed
> > to help with the parts that require Jenkins admin access.
> >
> > Thanks,
> > --Matt
> >
> >
> >
> > On Mon, Mar 4, 2013 at 5:00 PM, Konstantin Shvachko <
> shv.hadoop@gmail.com>wrote:
> >
> > > +1 on the merge.
> > >
> > > I am glad we agreed.
> > > Having Jira to track the CI effort is a good idea.
> > >
> > > Thanks,
> > > --Konstantin
> > >
> > > On Mon, Mar 4, 2013 at 3:29 PM, Matt Foley <mf...@hortonworks.com>
> wrote:
> > > > Thanks.  I agree Windows -1's in test-patch should not block commits.
> > > >
> > > > --Matt
> > > >
> > > >
> > > >
> > > > On Mon, Mar 4, 2013 at 2:30 PM, Konstantin Shvachko <
> > > shv.hadoop@gmail.com>
> > > > wrote:
> > > >>
> > > >> On Mon, Mar 4, 2013 at 12:22 PM, Matt Foley <mfoley@hortonworks.com
> >
> > > >> wrote:
> > > >> > Konstantine, you have voted -1, and stated some requirements
> before
> > > >> > you'll
> > > >> > withdraw that -1.  As I plan to do work to fulfill those
> > > requirements, I
> > > >> > want to make sure that what I'm proposing will, in fact, satisfy
> you.
> > > >> > That's why I'm asking, if we implement full "test-patch"
> integration
> > > for
> > > >> > Windows, does it seem to you that that would provide adequate
> support?
> > > >>
> > > >> Yes.
> > > >>
> > > >> > I have learned not to presume that my interpretation is correct.
>  My
> > > >> > interpretation of item #1 is that test-patch provides pre-commit
> > > build,
> > > >> > so
> > > >> > it would satisfy item #1.  But rather than assuming that I am
> > > >> > interpreting
> > > >> > it correctly, I simply want your agreement that it would, or if
> not,
> > > >> > clarification why it won't.
> > > >>
> > > >> I agree it will satisfy my item #1.
> > > >> I did not agree in my previous email, but I changed my mind based on
> > > >> the latest discussion. I have to explain why now.
> > > >> I was proposing nightly build because I did not want pre-commit
> build
> > > >> for Windows block commits to Linux. But if people are fine just
> ignoring
> > > >> -1s for the Windows part of the build it should be good.
> > > >>
> > > >> > Regarding item #2, it is also my interpretation that test-patch
> > > provides
> > > >> > an
> > > >> > on-demand (perhaps 20-minutes deferred) Jenkins build and unit
> test,
> > > >> > with
> > > >> > logs available to the developer, so it would satisfy item #2.  But
> > > >> > rather
> > > >> > than assuming that I am interpreting it correctly, I simply want
> your
> > > >> > agreement that it would, or if not, clarification why it won't.
> > > >>
> > > >> It will satisfy my item #2 in the following way:
> > > >> I can duplicate your pre-commit build for Windows and add an input
> > > >> parameter, which would let people run the build on their patches
> > > >> chosen from local machine rather than attaching them to Jiras.
> > > >>
> > > >> Thanks,
> > > >> --Konstantin
> > > >>
> > > >> > In agile terms, you are the Owner of these requirements.  Please
> give
> > > me
> > > >> > owner feedback as to whether my proposed work sounds like it will
> > > >> > satisfy
> > > >> > the requirements.
> > > >> >
> > > >> > Thank you,
> > > >> > --Matt
> > > >> >
> > > >> >
> > > >> > On Sun, Mar 3, 2013 at 12:16 PM, Konstantin Shvachko
> > > >> > <sh...@gmail.com>
> > > >> > wrote:
> > > >> >>
> > > >> >> Didn't I explain in details what I am asking for?
> > > >> >>
> > > >> >> Thanks,
> > > >> >> --Konst
> > > >> >>
> > > >> >> On Sun, Mar 3, 2013 at 11:08 AM, Matt Foley <
> mfoley@hortonworks.com>
> > > >> >> wrote:
> > > >> >> > Hi Konstantin,
> > > >> >> > I'd like to point out two things:
> > > >> >> > First, I already committed in this thread (email of Thu, Feb
> 28,
> > > 2013
> > > >> >> > at
> > > >> >> > 6:01 PM) to providing CI for Windows builds.  So please stop
> acting
> > > >> >> > like
> > > >> >> > I'm
> > > >> >> > resisting this idea or something.
> > > >> >> > Second, you didn't answer my question, you just kvetched about
> the
> > > >> >> > phrasing.
> > > >> >> > So I ask again:
> > > >> >> >
> > > >> >> > Will providing full "test-patch" integration (pre-commit build
> and
> > > >> >> > unit
> > > >> >> > test
> > > >> >> > triggered by Jira "Patch Available" state) satisfy your
> request for
> > > >> >> > functionality #1 and #2?  Yes or no, please.
> > > >> >> >
> > > >> >> > Thanks,
> > > >> >> > --Matt
> > > >> >> >
> > > >> >> >
> > > >> >> > On Sat, Mar 2, 2013 at 7:32 PM, Konstantin Shvachko
> > > >> >> > <sh...@gmail.com>
> > > >> >> > wrote:
> > > >> >> >>
> > > >> >> >> Hi Matt,
> > > >> >> >>
> > > >> >> >> On Sat, Mar 2, 2013 at 12:32 PM, Matt Foley <
> > > mfoley@hortonworks.com>
> > > >> >> >> wrote:
> > > >> >> >> > Konstantin,
> > > >> >> >> > I would like to explore what it would take to remove this
> > > >> >> >> > perceived
> > > >> >> >> > impediment --
> > > >> >> >>
> > > >> >> >> Glad you decided to explore. Thank you.
> > > >> >> >>
> > > >> >> >> > although I reserve the right to argue that this is not
> > > >> >> >> > pre-requisite to merging the cross-platform support patch.
> > > >> >> >>
> > > >> >> >> It's your right indeed. So as mine to question what the
> platform
> > > >> >> >> support means for you, which I believe remained unclear.
> > > >> >> >> I do not impede the change as you should have noticed. My
> > > >> >> >> requirement
> > > >> >> >> comes from my perception of the support, which means to me
> exactly
> > > >> >> >> two
> > > >> >> >> things:
> > > >> >> >> 1. The ability to recognise the code is broken for the
> platform
> > > >> >> >> 2. The ability to test new patches on the platform
> > > >> >> >> The latter is problematic, as many noticed in this thread, for
> > > those
> > > >> >> >> whose customary environment does not include Windows.
> > > >> >> >>
> > > >> >> >> > If we implemented full "test-patch" support for Windows on
> > > trunk,
> > > >> >> >> > would
> > > >> >> >> > that
> > > >> >> >> > fulfill both your items #1 and #2?  Please note that:
> > > >> >> >> > a) Pushing the "Patch Available" button in Jira shall cause
> a
> > > >> >> >> > pre-commit
> > > >> >> >> > build to start within, I believe, 20 minutes.
> > > >> >> >> > b) That build keeps logs for both java build and unit tests
> for
> > > >> >> >> > several
> > > >> >> >> > days, that are accessible to all viewers.
> > > >> >> >>
> > > >> >> >> In item #1 I mostly asking for the nightly build, which is
> simpler
> > > >> >> >> than "test-patch". The latter would be ideal from the platform
> > > >> >> >> support
> > > >> >> >> viewpoint, but it is for the community to decide if we want
> to add
> > > >> >> >> extra +3 hours to the build.
> > > >> >> >> Nightly build in my understanding is triggered by the timer
> rather
> > > >> >> >> than by Jira's "submit patch" button.  On Jenkins build
> > > >> >> >> configuration
> > > >> >> >> you can specify it under "Build periodically".
> > > >> >> >>
> > > >> >> >> > So, does this provide sufficient on-demand support that we
> don't
> > > >> >> >> > have
> > > >> >> >> > to
> > > >> >> >> > implement a whole new on-demand VM support structure of some
> > > sort
> > > >> >> >> > for
> > > >> >> >> > #2
> > > >> >> >> > (which would be an extraordinary and impractical demand)?
> > > >> >> >>
> > > >> >> >> I did not mention VMs. Item #2 means a build, which runs
> > > >> >> >> "test-patch"
> > > >> >> >> target with the file specified by a user (instead of a jira
> > > >> >> >> attachment).
> > > >> >> >> When user clicks "Build Now" link a box is displayed where the
> > > user
> > > >> >> >> can enter the file path containing the patch. This can be
> > > specified
> > > >> >> >> in
> > > >> >> >> the Build Configuration under "This build is parameterized" by
> > > >> >> >> choosing AddParameter / FileParameter. The build can run on
> the
> > > same
> > > >> >> >> Windows machine as the nightly build.
> > > >> >> >> Such build will let people test their patches for Windows on
> > > Jenkins
> > > >> >> >> if they don't posses a license for the right version of
> Windows.
> > > >> >> >> I hope this will not turn into extraordinary or impractical
> > > effort.
> > > >> >> >>
> > > >> >> >> Thanks,
> > > >> >> >> --Konst
> > > >> >> >>
> > > >> >> >> > Thanks,
> > > >> >> >> > --Matt
> > > >> >> >> >
> > > >> >> >> >
> > > >> >> >> > On Fri, Mar 1, 2013 at 12:18 PM, Konstantin Shvachko
> > > >> >> >> > <sh...@gmail.com>
> > > >> >> >> > wrote:
> > > >> >> >> >>
> > > >> >> >> >> -1
> > > >> >> >> >> We should have a CI infrastructure in place before we can
> > > commit
> > > >> >> >> >> to
> > > >> >> >> >> supporting Windows platform.
> > > >> >> >> >>
> > > >> >> >> >> Eric is right Win/Cygwin was supported since day one.
> > > >> >> >> >> I had a Windows box under my desk running nightly builds
> back
> > > in
> > > >> >> >> >> 2006-07.
> > > >> >> >> >> People were irritated but I was filing windows bugs until
> 0.22
> > > >> >> >> >> release.
> > > >> >> >> >> Times changing and I am glad to see wider support for Win
> > > >> >> >> >> platform.
> > > >> >> >> >>
> > > >> >> >> >> But in order to make it work you guys need to put the CI
> > > process
> > > >> >> >> >> in
> > > >> >> >> >> place
> > > >> >> >> >>
> > > >> >> >> >> 1. windows jenkins build: could be nightly or PreCommit.
> > > >> >> >> >> - Nightly would mean that changes can be committed to trunk
> > > based
> > > >> >> >> >> on
> > > >> >> >> >> linux PreCommit build. And people will file bugs if the
> change
> > > >> >> >> >> broke
> > > >> >> >> >> Windows nightly build.
> > > >> >> >> >> - PreCommit-win build will mean automatic reporting failed
> > > tests
> > > >> >> >> >> to
> > > >> >> >> >> respective jira blocking commits the same way as it is now
> with
> > > >> >> >> >> linux
> > > >> >> >> >> PreCommit builds.
> > > >> >> >> >> We should discuss which way is more efficient for
> developers.
> > > >> >> >> >>
> > > >> >> >> >> 2. On-demand-windows Jenkins build.
> > > >> >> >> >> I see it as a build to which I can attach my patch and the
> > > build
> > > >> >> >> >> will
> > > >> >> >> >> run my changes on a dedicated windows box.
> > > >> >> >> >> That way people can test their changes without having
> personal
> > > >> >> >> >> windows
> > > >> >> >> >> nodes.
> > > >> >> >> >>
> > > >> >> >> >> I think this is the minimal set of requirement for us to be
> > > able
> > > >> >> >> >> to
> > > >> >> >> >> commit to the new platform.
> > > >> >> >> >> Right now I see only one windows related build
> > > >> >> >> >> https://builds.apache.org/view/Hadoop/job/Hadoop-1-win/
> > > >> >> >> >> Which was failing since Sept 8, 2012 and did not run in the
> > > last
> > > >> >> >> >> month.
> > > >> >> >> >>
> > > >> >> >> >> Thanks,
> > > >> >> >> >> --Konst
> > > >> >> >> >>
> > > >> >> >> >> On Thu, Feb 28, 2013 at 8:47 PM, Eric Baldeschwieler
> > > >> >> >> >> <er...@hortonworks.com> wrote:
> > > >> >> >> >> > +1 (non-binding)
> > > >> >> >> >> >
> > > >> >> >> >> > A few of observations:
> > > >> >> >> >> >
> > > >> >> >> >> > - Windows has actually been a supported platform for
> Hadoop
> > > >> >> >> >> > since
> > > >> >> >> >> > 0.1
> > > >> >> >> >> > .
> > > >> >> >> >> > Doug championed supporting windows then and we've
> continued
> > > to
> > > >> >> >> >> > do
> > > >> >> >> >> > it
> > > >> >> >> >> > with
> > > >> >> >> >> > varying vigor over time.  To my knowledge we've never
> made a
> > > >> >> >> >> > decision
> > > >> >> >> >> > to
> > > >> >> >> >> > drop windows support.  The change here is improving our
> > > support
> > > >> >> >> >> > and
> > > >> >> >> >> > dropping
> > > >> >> >> >> > the requirement of cigwin.  We had Nutch windows users
> on the
> > > >> >> >> >> > list
> > > >> >> >> >> > in
> > > >> >> >> >> > 2006
> > > >> >> >> >> > and we've been supporting windows FS requirements since
> > > >> >> >> >> > inception.
> > > >> >> >> >> >
> > > >> >> >> >> > - A little pragmatism will go a long way.  As a community
> > > we've
> > > >> >> >> >> > got
> > > >> >> >> >> > to
> > > >> >> >> >> > stay committed to keeping hadoop simple (so it does work
> on
> > > >> >> >> >> > many
> > > >> >> >> >> > platforms)
> > > >> >> >> >> > and extending it to take advantage of key emerging
> > > OS/hardware
> > > >> >> >> >> > features,
> > > >> >> >> >> > such as containers, new FSs, virtualization, flash ...
>  We
> > > >> >> >> >> > should
> > > >> >> >> >> > all
> > > >> >> >> >> > plan
> > > >> >> >> >> > to let new features & optimizations emerge that don't
> work
> > > >> >> >> >> > everywhere, if
> > > >> >> >> >> > they are compelling and central to hadoop's mission of
> being
> > > >> >> >> >> > THE
> > > >> >> >> >> > best
> > > >> >> >> >> > fabric
> > > >> >> >> >> > for storing and processing big data.
> > > >> >> >> >> >
> > > >> >> >> >> > - A UI project like KDE has to deal with the MANY
> differences
> > > >> >> >> >> > between
> > > >> >> >> >> > windows and linux UI APIs.  Hadoop faces no such complex
> > > >> >> >> >> > challenge
> > > >> >> >> >> > and hence
> > > >> >> >> >> > can be maintained from a single codeline IMO.  It is
> mostly
> > > >> >> >> >> > abstracted from
> > > >> >> >> >> > the OS APIs via Java and our design choices.  Where it
> is not
> > > >> >> >> >> > we
> > > >> >> >> >> > can
> > > >> >> >> >> > continue to add plugable abstractions.
> > > >> >> >> >> >
> > > >> >> >> >
> > > >> >> >> >
> > > >> >> >
> > > >> >> >
> > > >> >
> > > >> >
> > > >
> > > >
> > >
>



-- 
http://hortonworks.com/download/

Re: [Vote] Merge branch-trunk-win to trunk

[Konstantin Boudnik <co...@apache.org>]

It doesn't look like any progress has been done on the ticket below in the
last 3 weeks. And now branch-2 can't be compiled because of 

hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSShell.java:[895,15]
WINDOWS is not public in org.apache.hadoop.fs.Path; cannot be accessed from
outside package

That's exactly why I was -1'ing this...
  Cos

On Mon, Mar 04, 2013 at 05:41PM, Matt Foley wrote:
> Thanks, gentlemen.  I've opened and taken responsibility for
> https://issues.apache.org/jira/browse/HADOOP-9359.  Giri Kesavan has agreed
> to help with the parts that require Jenkins admin access.
> 
> Thanks,
> --Matt
> 
> 
> 
> On Mon, Mar 4, 2013 at 5:00 PM, Konstantin Shvachko <sh...@gmail.com>wrote:
> 
> > +1 on the merge.
> >
> > I am glad we agreed.
> > Having Jira to track the CI effort is a good idea.
> >
> > Thanks,
> > --Konstantin
> >
> > On Mon, Mar 4, 2013 at 3:29 PM, Matt Foley <mf...@hortonworks.com> wrote:
> > > Thanks.  I agree Windows -1's in test-patch should not block commits.
> > >
> > > --Matt
> > >
> > >
> > >
> > > On Mon, Mar 4, 2013 at 2:30 PM, Konstantin Shvachko <
> > shv.hadoop@gmail.com>
> > > wrote:
> > >>
> > >> On Mon, Mar 4, 2013 at 12:22 PM, Matt Foley <mf...@hortonworks.com>
> > >> wrote:
> > >> > Konstantine, you have voted -1, and stated some requirements before
> > >> > you'll
> > >> > withdraw that -1.  As I plan to do work to fulfill those
> > requirements, I
> > >> > want to make sure that what I'm proposing will, in fact, satisfy you.
> > >> > That's why I'm asking, if we implement full "test-patch" integration
> > for
> > >> > Windows, does it seem to you that that would provide adequate support?
> > >>
> > >> Yes.
> > >>
> > >> > I have learned not to presume that my interpretation is correct.  My
> > >> > interpretation of item #1 is that test-patch provides pre-commit
> > build,
> > >> > so
> > >> > it would satisfy item #1.  But rather than assuming that I am
> > >> > interpreting
> > >> > it correctly, I simply want your agreement that it would, or if not,
> > >> > clarification why it won't.
> > >>
> > >> I agree it will satisfy my item #1.
> > >> I did not agree in my previous email, but I changed my mind based on
> > >> the latest discussion. I have to explain why now.
> > >> I was proposing nightly build because I did not want pre-commit build
> > >> for Windows block commits to Linux. But if people are fine just ignoring
> > >> -1s for the Windows part of the build it should be good.
> > >>
> > >> > Regarding item #2, it is also my interpretation that test-patch
> > provides
> > >> > an
> > >> > on-demand (perhaps 20-minutes deferred) Jenkins build and unit test,
> > >> > with
> > >> > logs available to the developer, so it would satisfy item #2.  But
> > >> > rather
> > >> > than assuming that I am interpreting it correctly, I simply want your
> > >> > agreement that it would, or if not, clarification why it won't.
> > >>
> > >> It will satisfy my item #2 in the following way:
> > >> I can duplicate your pre-commit build for Windows and add an input
> > >> parameter, which would let people run the build on their patches
> > >> chosen from local machine rather than attaching them to Jiras.
> > >>
> > >> Thanks,
> > >> --Konstantin
> > >>
> > >> > In agile terms, you are the Owner of these requirements.  Please give
> > me
> > >> > owner feedback as to whether my proposed work sounds like it will
> > >> > satisfy
> > >> > the requirements.
> > >> >
> > >> > Thank you,
> > >> > --Matt
> > >> >
> > >> >
> > >> > On Sun, Mar 3, 2013 at 12:16 PM, Konstantin Shvachko
> > >> > <sh...@gmail.com>
> > >> > wrote:
> > >> >>
> > >> >> Didn't I explain in details what I am asking for?
> > >> >>
> > >> >> Thanks,
> > >> >> --Konst
> > >> >>
> > >> >> On Sun, Mar 3, 2013 at 11:08 AM, Matt Foley <mf...@hortonworks.com>
> > >> >> wrote:
> > >> >> > Hi Konstantin,
> > >> >> > I'd like to point out two things:
> > >> >> > First, I already committed in this thread (email of Thu, Feb 28,
> > 2013
> > >> >> > at
> > >> >> > 6:01 PM) to providing CI for Windows builds.  So please stop acting
> > >> >> > like
> > >> >> > I'm
> > >> >> > resisting this idea or something.
> > >> >> > Second, you didn't answer my question, you just kvetched about the
> > >> >> > phrasing.
> > >> >> > So I ask again:
> > >> >> >
> > >> >> > Will providing full "test-patch" integration (pre-commit build and
> > >> >> > unit
> > >> >> > test
> > >> >> > triggered by Jira "Patch Available" state) satisfy your request for
> > >> >> > functionality #1 and #2?  Yes or no, please.
> > >> >> >
> > >> >> > Thanks,
> > >> >> > --Matt
> > >> >> >
> > >> >> >
> > >> >> > On Sat, Mar 2, 2013 at 7:32 PM, Konstantin Shvachko
> > >> >> > <sh...@gmail.com>
> > >> >> > wrote:
> > >> >> >>
> > >> >> >> Hi Matt,
> > >> >> >>
> > >> >> >> On Sat, Mar 2, 2013 at 12:32 PM, Matt Foley <
> > mfoley@hortonworks.com>
> > >> >> >> wrote:
> > >> >> >> > Konstantin,
> > >> >> >> > I would like to explore what it would take to remove this
> > >> >> >> > perceived
> > >> >> >> > impediment --
> > >> >> >>
> > >> >> >> Glad you decided to explore. Thank you.
> > >> >> >>
> > >> >> >> > although I reserve the right to argue that this is not
> > >> >> >> > pre-requisite to merging the cross-platform support patch.
> > >> >> >>
> > >> >> >> It's your right indeed. So as mine to question what the platform
> > >> >> >> support means for you, which I believe remained unclear.
> > >> >> >> I do not impede the change as you should have noticed. My
> > >> >> >> requirement
> > >> >> >> comes from my perception of the support, which means to me exactly
> > >> >> >> two
> > >> >> >> things:
> > >> >> >> 1. The ability to recognise the code is broken for the platform
> > >> >> >> 2. The ability to test new patches on the platform
> > >> >> >> The latter is problematic, as many noticed in this thread, for
> > those
> > >> >> >> whose customary environment does not include Windows.
> > >> >> >>
> > >> >> >> > If we implemented full "test-patch" support for Windows on
> > trunk,
> > >> >> >> > would
> > >> >> >> > that
> > >> >> >> > fulfill both your items #1 and #2?  Please note that:
> > >> >> >> > a) Pushing the "Patch Available" button in Jira shall cause a
> > >> >> >> > pre-commit
> > >> >> >> > build to start within, I believe, 20 minutes.
> > >> >> >> > b) That build keeps logs for both java build and unit tests for
> > >> >> >> > several
> > >> >> >> > days, that are accessible to all viewers.
> > >> >> >>
> > >> >> >> In item #1 I mostly asking for the nightly build, which is simpler
> > >> >> >> than "test-patch". The latter would be ideal from the platform
> > >> >> >> support
> > >> >> >> viewpoint, but it is for the community to decide if we want to add
> > >> >> >> extra +3 hours to the build.
> > >> >> >> Nightly build in my understanding is triggered by the timer rather
> > >> >> >> than by Jira's "submit patch" button.  On Jenkins build
> > >> >> >> configuration
> > >> >> >> you can specify it under "Build periodically".
> > >> >> >>
> > >> >> >> > So, does this provide sufficient on-demand support that we don't
> > >> >> >> > have
> > >> >> >> > to
> > >> >> >> > implement a whole new on-demand VM support structure of some
> > sort
> > >> >> >> > for
> > >> >> >> > #2
> > >> >> >> > (which would be an extraordinary and impractical demand)?
> > >> >> >>
> > >> >> >> I did not mention VMs. Item #2 means a build, which runs
> > >> >> >> "test-patch"
> > >> >> >> target with the file specified by a user (instead of a jira
> > >> >> >> attachment).
> > >> >> >> When user clicks "Build Now" link a box is displayed where the
> > user
> > >> >> >> can enter the file path containing the patch. This can be
> > specified
> > >> >> >> in
> > >> >> >> the Build Configuration under "This build is parameterized" by
> > >> >> >> choosing AddParameter / FileParameter. The build can run on the
> > same
> > >> >> >> Windows machine as the nightly build.
> > >> >> >> Such build will let people test their patches for Windows on
> > Jenkins
> > >> >> >> if they don't posses a license for the right version of Windows.
> > >> >> >> I hope this will not turn into extraordinary or impractical
> > effort.
> > >> >> >>
> > >> >> >> Thanks,
> > >> >> >> --Konst
> > >> >> >>
> > >> >> >> > Thanks,
> > >> >> >> > --Matt
> > >> >> >> >
> > >> >> >> >
> > >> >> >> > On Fri, Mar 1, 2013 at 12:18 PM, Konstantin Shvachko
> > >> >> >> > <sh...@gmail.com>
> > >> >> >> > wrote:
> > >> >> >> >>
> > >> >> >> >> -1
> > >> >> >> >> We should have a CI infrastructure in place before we can
> > commit
> > >> >> >> >> to
> > >> >> >> >> supporting Windows platform.
> > >> >> >> >>
> > >> >> >> >> Eric is right Win/Cygwin was supported since day one.
> > >> >> >> >> I had a Windows box under my desk running nightly builds back
> > in
> > >> >> >> >> 2006-07.
> > >> >> >> >> People were irritated but I was filing windows bugs until 0.22
> > >> >> >> >> release.
> > >> >> >> >> Times changing and I am glad to see wider support for Win
> > >> >> >> >> platform.
> > >> >> >> >>
> > >> >> >> >> But in order to make it work you guys need to put the CI
> > process
> > >> >> >> >> in
> > >> >> >> >> place
> > >> >> >> >>
> > >> >> >> >> 1. windows jenkins build: could be nightly or PreCommit.
> > >> >> >> >> - Nightly would mean that changes can be committed to trunk
> > based
> > >> >> >> >> on
> > >> >> >> >> linux PreCommit build. And people will file bugs if the change
> > >> >> >> >> broke
> > >> >> >> >> Windows nightly build.
> > >> >> >> >> - PreCommit-win build will mean automatic reporting failed
> > tests
> > >> >> >> >> to
> > >> >> >> >> respective jira blocking commits the same way as it is now with
> > >> >> >> >> linux
> > >> >> >> >> PreCommit builds.
> > >> >> >> >> We should discuss which way is more efficient for developers.
> > >> >> >> >>
> > >> >> >> >> 2. On-demand-windows Jenkins build.
> > >> >> >> >> I see it as a build to which I can attach my patch and the
> > build
> > >> >> >> >> will
> > >> >> >> >> run my changes on a dedicated windows box.
> > >> >> >> >> That way people can test their changes without having personal
> > >> >> >> >> windows
> > >> >> >> >> nodes.
> > >> >> >> >>
> > >> >> >> >> I think this is the minimal set of requirement for us to be
> > able
> > >> >> >> >> to
> > >> >> >> >> commit to the new platform.
> > >> >> >> >> Right now I see only one windows related build
> > >> >> >> >> https://builds.apache.org/view/Hadoop/job/Hadoop-1-win/
> > >> >> >> >> Which was failing since Sept 8, 2012 and did not run in the
> > last
> > >> >> >> >> month.
> > >> >> >> >>
> > >> >> >> >> Thanks,
> > >> >> >> >> --Konst
> > >> >> >> >>
> > >> >> >> >> On Thu, Feb 28, 2013 at 8:47 PM, Eric Baldeschwieler
> > >> >> >> >> <er...@hortonworks.com> wrote:
> > >> >> >> >> > +1 (non-binding)
> > >> >> >> >> >
> > >> >> >> >> > A few of observations:
> > >> >> >> >> >
> > >> >> >> >> > - Windows has actually been a supported platform for Hadoop
> > >> >> >> >> > since
> > >> >> >> >> > 0.1
> > >> >> >> >> > .
> > >> >> >> >> > Doug championed supporting windows then and we've continued
> > to
> > >> >> >> >> > do
> > >> >> >> >> > it
> > >> >> >> >> > with
> > >> >> >> >> > varying vigor over time.  To my knowledge we've never made a
> > >> >> >> >> > decision
> > >> >> >> >> > to
> > >> >> >> >> > drop windows support.  The change here is improving our
> > support
> > >> >> >> >> > and
> > >> >> >> >> > dropping
> > >> >> >> >> > the requirement of cigwin.  We had Nutch windows users on the
> > >> >> >> >> > list
> > >> >> >> >> > in
> > >> >> >> >> > 2006
> > >> >> >> >> > and we've been supporting windows FS requirements since
> > >> >> >> >> > inception.
> > >> >> >> >> >
> > >> >> >> >> > - A little pragmatism will go a long way.  As a community
> > we've
> > >> >> >> >> > got
> > >> >> >> >> > to
> > >> >> >> >> > stay committed to keeping hadoop simple (so it does work on
> > >> >> >> >> > many
> > >> >> >> >> > platforms)
> > >> >> >> >> > and extending it to take advantage of key emerging
> > OS/hardware
> > >> >> >> >> > features,
> > >> >> >> >> > such as containers, new FSs, virtualization, flash ...  We
> > >> >> >> >> > should
> > >> >> >> >> > all
> > >> >> >> >> > plan
> > >> >> >> >> > to let new features & optimizations emerge that don't work
> > >> >> >> >> > everywhere, if
> > >> >> >> >> > they are compelling and central to hadoop's mission of being
> > >> >> >> >> > THE
> > >> >> >> >> > best
> > >> >> >> >> > fabric
> > >> >> >> >> > for storing and processing big data.
> > >> >> >> >> >
> > >> >> >> >> > - A UI project like KDE has to deal with the MANY differences
> > >> >> >> >> > between
> > >> >> >> >> > windows and linux UI APIs.  Hadoop faces no such complex
> > >> >> >> >> > challenge
> > >> >> >> >> > and hence
> > >> >> >> >> > can be maintained from a single codeline IMO.  It is mostly
> > >> >> >> >> > abstracted from
> > >> >> >> >> > the OS APIs via Java and our design choices.  Where it is not
> > >> >> >> >> > we
> > >> >> >> >> > can
> > >> >> >> >> > continue to add plugable abstractions.
> > >> >> >> >> >
> > >> >> >> >
> > >> >> >> >
> > >> >> >
> > >> >> >
> > >> >
> > >> >
> > >
> > >
> >

Re: [Vote] Merge branch-trunk-win to trunk

[Konstantin Boudnik <co...@apache.org>]

It doesn't look like any progress has been done on the ticket below in the
last 3 weeks. And now branch-2 can't be compiled because of 

hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSShell.java:[895,15]
WINDOWS is not public in org.apache.hadoop.fs.Path; cannot be accessed from
outside package

That's exactly why I was -1'ing this...
  Cos

On Mon, Mar 04, 2013 at 05:41PM, Matt Foley wrote:
> Thanks, gentlemen.  I've opened and taken responsibility for
> https://issues.apache.org/jira/browse/HADOOP-9359.  Giri Kesavan has agreed
> to help with the parts that require Jenkins admin access.
> 
> Thanks,
> --Matt
> 
> 
> 
> On Mon, Mar 4, 2013 at 5:00 PM, Konstantin Shvachko <sh...@gmail.com>wrote:
> 
> > +1 on the merge.
> >
> > I am glad we agreed.
> > Having Jira to track the CI effort is a good idea.
> >
> > Thanks,
> > --Konstantin
> >
> > On Mon, Mar 4, 2013 at 3:29 PM, Matt Foley <mf...@hortonworks.com> wrote:
> > > Thanks.  I agree Windows -1's in test-patch should not block commits.
> > >
> > > --Matt
> > >
> > >
> > >
> > > On Mon, Mar 4, 2013 at 2:30 PM, Konstantin Shvachko <
> > shv.hadoop@gmail.com>
> > > wrote:
> > >>
> > >> On Mon, Mar 4, 2013 at 12:22 PM, Matt Foley <mf...@hortonworks.com>
> > >> wrote:
> > >> > Konstantine, you have voted -1, and stated some requirements before
> > >> > you'll
> > >> > withdraw that -1.  As I plan to do work to fulfill those
> > requirements, I
> > >> > want to make sure that what I'm proposing will, in fact, satisfy you.
> > >> > That's why I'm asking, if we implement full "test-patch" integration
> > for
> > >> > Windows, does it seem to you that that would provide adequate support?
> > >>
> > >> Yes.
> > >>
> > >> > I have learned not to presume that my interpretation is correct.  My
> > >> > interpretation of item #1 is that test-patch provides pre-commit
> > build,
> > >> > so
> > >> > it would satisfy item #1.  But rather than assuming that I am
> > >> > interpreting
> > >> > it correctly, I simply want your agreement that it would, or if not,
> > >> > clarification why it won't.
> > >>
> > >> I agree it will satisfy my item #1.
> > >> I did not agree in my previous email, but I changed my mind based on
> > >> the latest discussion. I have to explain why now.
> > >> I was proposing nightly build because I did not want pre-commit build
> > >> for Windows block commits to Linux. But if people are fine just ignoring
> > >> -1s for the Windows part of the build it should be good.
> > >>
> > >> > Regarding item #2, it is also my interpretation that test-patch
> > provides
> > >> > an
> > >> > on-demand (perhaps 20-minutes deferred) Jenkins build and unit test,
> > >> > with
> > >> > logs available to the developer, so it would satisfy item #2.  But
> > >> > rather
> > >> > than assuming that I am interpreting it correctly, I simply want your
> > >> > agreement that it would, or if not, clarification why it won't.
> > >>
> > >> It will satisfy my item #2 in the following way:
> > >> I can duplicate your pre-commit build for Windows and add an input
> > >> parameter, which would let people run the build on their patches
> > >> chosen from local machine rather than attaching them to Jiras.
> > >>
> > >> Thanks,
> > >> --Konstantin
> > >>
> > >> > In agile terms, you are the Owner of these requirements.  Please give
> > me
> > >> > owner feedback as to whether my proposed work sounds like it will
> > >> > satisfy
> > >> > the requirements.
> > >> >
> > >> > Thank you,
> > >> > --Matt
> > >> >
> > >> >
> > >> > On Sun, Mar 3, 2013 at 12:16 PM, Konstantin Shvachko
> > >> > <sh...@gmail.com>
> > >> > wrote:
> > >> >>
> > >> >> Didn't I explain in details what I am asking for?
> > >> >>
> > >> >> Thanks,
> > >> >> --Konst
> > >> >>
> > >> >> On Sun, Mar 3, 2013 at 11:08 AM, Matt Foley <mf...@hortonworks.com>
> > >> >> wrote:
> > >> >> > Hi Konstantin,
> > >> >> > I'd like to point out two things:
> > >> >> > First, I already committed in this thread (email of Thu, Feb 28,
> > 2013
> > >> >> > at
> > >> >> > 6:01 PM) to providing CI for Windows builds.  So please stop acting
> > >> >> > like
> > >> >> > I'm
> > >> >> > resisting this idea or something.
> > >> >> > Second, you didn't answer my question, you just kvetched about the
> > >> >> > phrasing.
> > >> >> > So I ask again:
> > >> >> >
> > >> >> > Will providing full "test-patch" integration (pre-commit build and
> > >> >> > unit
> > >> >> > test
> > >> >> > triggered by Jira "Patch Available" state) satisfy your request for
> > >> >> > functionality #1 and #2?  Yes or no, please.
> > >> >> >
> > >> >> > Thanks,
> > >> >> > --Matt
> > >> >> >
> > >> >> >
> > >> >> > On Sat, Mar 2, 2013 at 7:32 PM, Konstantin Shvachko
> > >> >> > <sh...@gmail.com>
> > >> >> > wrote:
> > >> >> >>
> > >> >> >> Hi Matt,
> > >> >> >>
> > >> >> >> On Sat, Mar 2, 2013 at 12:32 PM, Matt Foley <
> > mfoley@hortonworks.com>
> > >> >> >> wrote:
> > >> >> >> > Konstantin,
> > >> >> >> > I would like to explore what it would take to remove this
> > >> >> >> > perceived
> > >> >> >> > impediment --
> > >> >> >>
> > >> >> >> Glad you decided to explore. Thank you.
> > >> >> >>
> > >> >> >> > although I reserve the right to argue that this is not
> > >> >> >> > pre-requisite to merging the cross-platform support patch.
> > >> >> >>
> > >> >> >> It's your right indeed. So as mine to question what the platform
> > >> >> >> support means for you, which I believe remained unclear.
> > >> >> >> I do not impede the change as you should have noticed. My
> > >> >> >> requirement
> > >> >> >> comes from my perception of the support, which means to me exactly
> > >> >> >> two
> > >> >> >> things:
> > >> >> >> 1. The ability to recognise the code is broken for the platform
> > >> >> >> 2. The ability to test new patches on the platform
> > >> >> >> The latter is problematic, as many noticed in this thread, for
> > those
> > >> >> >> whose customary environment does not include Windows.
> > >> >> >>
> > >> >> >> > If we implemented full "test-patch" support for Windows on
> > trunk,
> > >> >> >> > would
> > >> >> >> > that
> > >> >> >> > fulfill both your items #1 and #2?  Please note that:
> > >> >> >> > a) Pushing the "Patch Available" button in Jira shall cause a
> > >> >> >> > pre-commit
> > >> >> >> > build to start within, I believe, 20 minutes.
> > >> >> >> > b) That build keeps logs for both java build and unit tests for
> > >> >> >> > several
> > >> >> >> > days, that are accessible to all viewers.
> > >> >> >>
> > >> >> >> In item #1 I mostly asking for the nightly build, which is simpler
> > >> >> >> than "test-patch". The latter would be ideal from the platform
> > >> >> >> support
> > >> >> >> viewpoint, but it is for the community to decide if we want to add
> > >> >> >> extra +3 hours to the build.
> > >> >> >> Nightly build in my understanding is triggered by the timer rather
> > >> >> >> than by Jira's "submit patch" button.  On Jenkins build
> > >> >> >> configuration
> > >> >> >> you can specify it under "Build periodically".
> > >> >> >>
> > >> >> >> > So, does this provide sufficient on-demand support that we don't
> > >> >> >> > have
> > >> >> >> > to
> > >> >> >> > implement a whole new on-demand VM support structure of some
> > sort
> > >> >> >> > for
> > >> >> >> > #2
> > >> >> >> > (which would be an extraordinary and impractical demand)?
> > >> >> >>
> > >> >> >> I did not mention VMs. Item #2 means a build, which runs
> > >> >> >> "test-patch"
> > >> >> >> target with the file specified by a user (instead of a jira
> > >> >> >> attachment).
> > >> >> >> When user clicks "Build Now" link a box is displayed where the
> > user
> > >> >> >> can enter the file path containing the patch. This can be
> > specified
> > >> >> >> in
> > >> >> >> the Build Configuration under "This build is parameterized" by
> > >> >> >> choosing AddParameter / FileParameter. The build can run on the
> > same
> > >> >> >> Windows machine as the nightly build.
> > >> >> >> Such build will let people test their patches for Windows on
> > Jenkins
> > >> >> >> if they don't posses a license for the right version of Windows.
> > >> >> >> I hope this will not turn into extraordinary or impractical
> > effort.
> > >> >> >>
> > >> >> >> Thanks,
> > >> >> >> --Konst
> > >> >> >>
> > >> >> >> > Thanks,
> > >> >> >> > --Matt
> > >> >> >> >
> > >> >> >> >
> > >> >> >> > On Fri, Mar 1, 2013 at 12:18 PM, Konstantin Shvachko
> > >> >> >> > <sh...@gmail.com>
> > >> >> >> > wrote:
> > >> >> >> >>
> > >> >> >> >> -1
> > >> >> >> >> We should have a CI infrastructure in place before we can
> > commit
> > >> >> >> >> to
> > >> >> >> >> supporting Windows platform.
> > >> >> >> >>
> > >> >> >> >> Eric is right Win/Cygwin was supported since day one.
> > >> >> >> >> I had a Windows box under my desk running nightly builds back
> > in
> > >> >> >> >> 2006-07.
> > >> >> >> >> People were irritated but I was filing windows bugs until 0.22
> > >> >> >> >> release.
> > >> >> >> >> Times changing and I am glad to see wider support for Win
> > >> >> >> >> platform.
> > >> >> >> >>
> > >> >> >> >> But in order to make it work you guys need to put the CI
> > process
> > >> >> >> >> in
> > >> >> >> >> place
> > >> >> >> >>
> > >> >> >> >> 1. windows jenkins build: could be nightly or PreCommit.
> > >> >> >> >> - Nightly would mean that changes can be committed to trunk
> > based
> > >> >> >> >> on
> > >> >> >> >> linux PreCommit build. And people will file bugs if the change
> > >> >> >> >> broke
> > >> >> >> >> Windows nightly build.
> > >> >> >> >> - PreCommit-win build will mean automatic reporting failed
> > tests
> > >> >> >> >> to
> > >> >> >> >> respective jira blocking commits the same way as it is now with
> > >> >> >> >> linux
> > >> >> >> >> PreCommit builds.
> > >> >> >> >> We should discuss which way is more efficient for developers.
> > >> >> >> >>
> > >> >> >> >> 2. On-demand-windows Jenkins build.
> > >> >> >> >> I see it as a build to which I can attach my patch and the
> > build
> > >> >> >> >> will
> > >> >> >> >> run my changes on a dedicated windows box.
> > >> >> >> >> That way people can test their changes without having personal
> > >> >> >> >> windows
> > >> >> >> >> nodes.
> > >> >> >> >>
> > >> >> >> >> I think this is the minimal set of requirement for us to be
> > able
> > >> >> >> >> to
> > >> >> >> >> commit to the new platform.
> > >> >> >> >> Right now I see only one windows related build
> > >> >> >> >> https://builds.apache.org/view/Hadoop/job/Hadoop-1-win/
> > >> >> >> >> Which was failing since Sept 8, 2012 and did not run in the
> > last
> > >> >> >> >> month.
> > >> >> >> >>
> > >> >> >> >> Thanks,
> > >> >> >> >> --Konst
> > >> >> >> >>
> > >> >> >> >> On Thu, Feb 28, 2013 at 8:47 PM, Eric Baldeschwieler
> > >> >> >> >> <er...@hortonworks.com> wrote:
> > >> >> >> >> > +1 (non-binding)
> > >> >> >> >> >
> > >> >> >> >> > A few of observations:
> > >> >> >> >> >
> > >> >> >> >> > - Windows has actually been a supported platform for Hadoop
> > >> >> >> >> > since
> > >> >> >> >> > 0.1
> > >> >> >> >> > .
> > >> >> >> >> > Doug championed supporting windows then and we've continued
> > to
> > >> >> >> >> > do
> > >> >> >> >> > it
> > >> >> >> >> > with
> > >> >> >> >> > varying vigor over time.  To my knowledge we've never made a
> > >> >> >> >> > decision
> > >> >> >> >> > to
> > >> >> >> >> > drop windows support.  The change here is improving our
> > support
> > >> >> >> >> > and
> > >> >> >> >> > dropping
> > >> >> >> >> > the requirement of cigwin.  We had Nutch windows users on the
> > >> >> >> >> > list
> > >> >> >> >> > in
> > >> >> >> >> > 2006
> > >> >> >> >> > and we've been supporting windows FS requirements since
> > >> >> >> >> > inception.
> > >> >> >> >> >
> > >> >> >> >> > - A little pragmatism will go a long way.  As a community
> > we've
> > >> >> >> >> > got
> > >> >> >> >> > to
> > >> >> >> >> > stay committed to keeping hadoop simple (so it does work on
> > >> >> >> >> > many
> > >> >> >> >> > platforms)
> > >> >> >> >> > and extending it to take advantage of key emerging
> > OS/hardware
> > >> >> >> >> > features,
> > >> >> >> >> > such as containers, new FSs, virtualization, flash ...  We
> > >> >> >> >> > should
> > >> >> >> >> > all
> > >> >> >> >> > plan
> > >> >> >> >> > to let new features & optimizations emerge that don't work
> > >> >> >> >> > everywhere, if
> > >> >> >> >> > they are compelling and central to hadoop's mission of being
> > >> >> >> >> > THE
> > >> >> >> >> > best
> > >> >> >> >> > fabric
> > >> >> >> >> > for storing and processing big data.
> > >> >> >> >> >
> > >> >> >> >> > - A UI project like KDE has to deal with the MANY differences
> > >> >> >> >> > between
> > >> >> >> >> > windows and linux UI APIs.  Hadoop faces no such complex
> > >> >> >> >> > challenge
> > >> >> >> >> > and hence
> > >> >> >> >> > can be maintained from a single codeline IMO.  It is mostly
> > >> >> >> >> > abstracted from
> > >> >> >> >> > the OS APIs via Java and our design choices.  Where it is not
> > >> >> >> >> > we
> > >> >> >> >> > can
> > >> >> >> >> > continue to add plugable abstractions.
> > >> >> >> >> >
> > >> >> >> >
> > >> >> >> >
> > >> >> >
> > >> >> >
> > >> >
> > >> >
> > >
> > >
> >

Re: [Vote] Merge branch-trunk-win to trunk

[Konstantin Boudnik <co...@apache.org>]

It doesn't look like any progress has been done on the ticket below in the
last 3 weeks. And now branch-2 can't be compiled because of 

hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSShell.java:[895,15]
WINDOWS is not public in org.apache.hadoop.fs.Path; cannot be accessed from
outside package

That's exactly why I was -1'ing this...
  Cos

On Mon, Mar 04, 2013 at 05:41PM, Matt Foley wrote:
> Thanks, gentlemen.  I've opened and taken responsibility for
> https://issues.apache.org/jira/browse/HADOOP-9359.  Giri Kesavan has agreed
> to help with the parts that require Jenkins admin access.
> 
> Thanks,
> --Matt
> 
> 
> 
> On Mon, Mar 4, 2013 at 5:00 PM, Konstantin Shvachko <sh...@gmail.com>wrote:
> 
> > +1 on the merge.
> >
> > I am glad we agreed.
> > Having Jira to track the CI effort is a good idea.
> >
> > Thanks,
> > --Konstantin
> >
> > On Mon, Mar 4, 2013 at 3:29 PM, Matt Foley <mf...@hortonworks.com> wrote:
> > > Thanks.  I agree Windows -1's in test-patch should not block commits.
> > >
> > > --Matt
> > >
> > >
> > >
> > > On Mon, Mar 4, 2013 at 2:30 PM, Konstantin Shvachko <
> > shv.hadoop@gmail.com>
> > > wrote:
> > >>
> > >> On Mon, Mar 4, 2013 at 12:22 PM, Matt Foley <mf...@hortonworks.com>
> > >> wrote:
> > >> > Konstantine, you have voted -1, and stated some requirements before
> > >> > you'll
> > >> > withdraw that -1.  As I plan to do work to fulfill those
> > requirements, I
> > >> > want to make sure that what I'm proposing will, in fact, satisfy you.
> > >> > That's why I'm asking, if we implement full "test-patch" integration
> > for
> > >> > Windows, does it seem to you that that would provide adequate support?
> > >>
> > >> Yes.
> > >>
> > >> > I have learned not to presume that my interpretation is correct.  My
> > >> > interpretation of item #1 is that test-patch provides pre-commit
> > build,
> > >> > so
> > >> > it would satisfy item #1.  But rather than assuming that I am
> > >> > interpreting
> > >> > it correctly, I simply want your agreement that it would, or if not,
> > >> > clarification why it won't.
> > >>
> > >> I agree it will satisfy my item #1.
> > >> I did not agree in my previous email, but I changed my mind based on
> > >> the latest discussion. I have to explain why now.
> > >> I was proposing nightly build because I did not want pre-commit build
> > >> for Windows block commits to Linux. But if people are fine just ignoring
> > >> -1s for the Windows part of the build it should be good.
> > >>
> > >> > Regarding item #2, it is also my interpretation that test-patch
> > provides
> > >> > an
> > >> > on-demand (perhaps 20-minutes deferred) Jenkins build and unit test,
> > >> > with
> > >> > logs available to the developer, so it would satisfy item #2.  But
> > >> > rather
> > >> > than assuming that I am interpreting it correctly, I simply want your
> > >> > agreement that it would, or if not, clarification why it won't.
> > >>
> > >> It will satisfy my item #2 in the following way:
> > >> I can duplicate your pre-commit build for Windows and add an input
> > >> parameter, which would let people run the build on their patches
> > >> chosen from local machine rather than attaching them to Jiras.
> > >>
> > >> Thanks,
> > >> --Konstantin
> > >>
> > >> > In agile terms, you are the Owner of these requirements.  Please give
> > me
> > >> > owner feedback as to whether my proposed work sounds like it will
> > >> > satisfy
> > >> > the requirements.
> > >> >
> > >> > Thank you,
> > >> > --Matt
> > >> >
> > >> >
> > >> > On Sun, Mar 3, 2013 at 12:16 PM, Konstantin Shvachko
> > >> > <sh...@gmail.com>
> > >> > wrote:
> > >> >>
> > >> >> Didn't I explain in details what I am asking for?
> > >> >>
> > >> >> Thanks,
> > >> >> --Konst
> > >> >>
> > >> >> On Sun, Mar 3, 2013 at 11:08 AM, Matt Foley <mf...@hortonworks.com>
> > >> >> wrote:
> > >> >> > Hi Konstantin,
> > >> >> > I'd like to point out two things:
> > >> >> > First, I already committed in this thread (email of Thu, Feb 28,
> > 2013
> > >> >> > at
> > >> >> > 6:01 PM) to providing CI for Windows builds.  So please stop acting
> > >> >> > like
> > >> >> > I'm
> > >> >> > resisting this idea or something.
> > >> >> > Second, you didn't answer my question, you just kvetched about the
> > >> >> > phrasing.
> > >> >> > So I ask again:
> > >> >> >
> > >> >> > Will providing full "test-patch" integration (pre-commit build and
> > >> >> > unit
> > >> >> > test
> > >> >> > triggered by Jira "Patch Available" state) satisfy your request for
> > >> >> > functionality #1 and #2?  Yes or no, please.
> > >> >> >
> > >> >> > Thanks,
> > >> >> > --Matt
> > >> >> >
> > >> >> >
> > >> >> > On Sat, Mar 2, 2013 at 7:32 PM, Konstantin Shvachko
> > >> >> > <sh...@gmail.com>
> > >> >> > wrote:
> > >> >> >>
> > >> >> >> Hi Matt,
> > >> >> >>
> > >> >> >> On Sat, Mar 2, 2013 at 12:32 PM, Matt Foley <
> > mfoley@hortonworks.com>
> > >> >> >> wrote:
> > >> >> >> > Konstantin,
> > >> >> >> > I would like to explore what it would take to remove this
> > >> >> >> > perceived
> > >> >> >> > impediment --
> > >> >> >>
> > >> >> >> Glad you decided to explore. Thank you.
> > >> >> >>
> > >> >> >> > although I reserve the right to argue that this is not
> > >> >> >> > pre-requisite to merging the cross-platform support patch.
> > >> >> >>
> > >> >> >> It's your right indeed. So as mine to question what the platform
> > >> >> >> support means for you, which I believe remained unclear.
> > >> >> >> I do not impede the change as you should have noticed. My
> > >> >> >> requirement
> > >> >> >> comes from my perception of the support, which means to me exactly
> > >> >> >> two
> > >> >> >> things:
> > >> >> >> 1. The ability to recognise the code is broken for the platform
> > >> >> >> 2. The ability to test new patches on the platform
> > >> >> >> The latter is problematic, as many noticed in this thread, for
> > those
> > >> >> >> whose customary environment does not include Windows.
> > >> >> >>
> > >> >> >> > If we implemented full "test-patch" support for Windows on
> > trunk,
> > >> >> >> > would
> > >> >> >> > that
> > >> >> >> > fulfill both your items #1 and #2?  Please note that:
> > >> >> >> > a) Pushing the "Patch Available" button in Jira shall cause a
> > >> >> >> > pre-commit
> > >> >> >> > build to start within, I believe, 20 minutes.
> > >> >> >> > b) That build keeps logs for both java build and unit tests for
> > >> >> >> > several
> > >> >> >> > days, that are accessible to all viewers.
> > >> >> >>
> > >> >> >> In item #1 I mostly asking for the nightly build, which is simpler
> > >> >> >> than "test-patch". The latter would be ideal from the platform
> > >> >> >> support
> > >> >> >> viewpoint, but it is for the community to decide if we want to add
> > >> >> >> extra +3 hours to the build.
> > >> >> >> Nightly build in my understanding is triggered by the timer rather
> > >> >> >> than by Jira's "submit patch" button.  On Jenkins build
> > >> >> >> configuration
> > >> >> >> you can specify it under "Build periodically".
> > >> >> >>
> > >> >> >> > So, does this provide sufficient on-demand support that we don't
> > >> >> >> > have
> > >> >> >> > to
> > >> >> >> > implement a whole new on-demand VM support structure of some
> > sort
> > >> >> >> > for
> > >> >> >> > #2
> > >> >> >> > (which would be an extraordinary and impractical demand)?
> > >> >> >>
> > >> >> >> I did not mention VMs. Item #2 means a build, which runs
> > >> >> >> "test-patch"
> > >> >> >> target with the file specified by a user (instead of a jira
> > >> >> >> attachment).
> > >> >> >> When user clicks "Build Now" link a box is displayed where the
> > user
> > >> >> >> can enter the file path containing the patch. This can be
> > specified
> > >> >> >> in
> > >> >> >> the Build Configuration under "This build is parameterized" by
> > >> >> >> choosing AddParameter / FileParameter. The build can run on the
> > same
> > >> >> >> Windows machine as the nightly build.
> > >> >> >> Such build will let people test their patches for Windows on
> > Jenkins
> > >> >> >> if they don't posses a license for the right version of Windows.
> > >> >> >> I hope this will not turn into extraordinary or impractical
> > effort.
> > >> >> >>
> > >> >> >> Thanks,
> > >> >> >> --Konst
> > >> >> >>
> > >> >> >> > Thanks,
> > >> >> >> > --Matt
> > >> >> >> >
> > >> >> >> >
> > >> >> >> > On Fri, Mar 1, 2013 at 12:18 PM, Konstantin Shvachko
> > >> >> >> > <sh...@gmail.com>
> > >> >> >> > wrote:
> > >> >> >> >>
> > >> >> >> >> -1
> > >> >> >> >> We should have a CI infrastructure in place before we can
> > commit
> > >> >> >> >> to
> > >> >> >> >> supporting Windows platform.
> > >> >> >> >>
> > >> >> >> >> Eric is right Win/Cygwin was supported since day one.
> > >> >> >> >> I had a Windows box under my desk running nightly builds back
> > in
> > >> >> >> >> 2006-07.
> > >> >> >> >> People were irritated but I was filing windows bugs until 0.22
> > >> >> >> >> release.
> > >> >> >> >> Times changing and I am glad to see wider support for Win
> > >> >> >> >> platform.
> > >> >> >> >>
> > >> >> >> >> But in order to make it work you guys need to put the CI
> > process
> > >> >> >> >> in
> > >> >> >> >> place
> > >> >> >> >>
> > >> >> >> >> 1. windows jenkins build: could be nightly or PreCommit.
> > >> >> >> >> - Nightly would mean that changes can be committed to trunk
> > based
> > >> >> >> >> on
> > >> >> >> >> linux PreCommit build. And people will file bugs if the change
> > >> >> >> >> broke
> > >> >> >> >> Windows nightly build.
> > >> >> >> >> - PreCommit-win build will mean automatic reporting failed
> > tests
> > >> >> >> >> to
> > >> >> >> >> respective jira blocking commits the same way as it is now with
> > >> >> >> >> linux
> > >> >> >> >> PreCommit builds.
> > >> >> >> >> We should discuss which way is more efficient for developers.
> > >> >> >> >>
> > >> >> >> >> 2. On-demand-windows Jenkins build.
> > >> >> >> >> I see it as a build to which I can attach my patch and the
> > build
> > >> >> >> >> will
> > >> >> >> >> run my changes on a dedicated windows box.
> > >> >> >> >> That way people can test their changes without having personal
> > >> >> >> >> windows
> > >> >> >> >> nodes.
> > >> >> >> >>
> > >> >> >> >> I think this is the minimal set of requirement for us to be
> > able
> > >> >> >> >> to
> > >> >> >> >> commit to the new platform.
> > >> >> >> >> Right now I see only one windows related build
> > >> >> >> >> https://builds.apache.org/view/Hadoop/job/Hadoop-1-win/
> > >> >> >> >> Which was failing since Sept 8, 2012 and did not run in the
> > last
> > >> >> >> >> month.
> > >> >> >> >>
> > >> >> >> >> Thanks,
> > >> >> >> >> --Konst
> > >> >> >> >>
> > >> >> >> >> On Thu, Feb 28, 2013 at 8:47 PM, Eric Baldeschwieler
> > >> >> >> >> <er...@hortonworks.com> wrote:
> > >> >> >> >> > +1 (non-binding)
> > >> >> >> >> >
> > >> >> >> >> > A few of observations:
> > >> >> >> >> >
> > >> >> >> >> > - Windows has actually been a supported platform for Hadoop
> > >> >> >> >> > since
> > >> >> >> >> > 0.1
> > >> >> >> >> > .
> > >> >> >> >> > Doug championed supporting windows then and we've continued
> > to
> > >> >> >> >> > do
> > >> >> >> >> > it
> > >> >> >> >> > with
> > >> >> >> >> > varying vigor over time.  To my knowledge we've never made a
> > >> >> >> >> > decision
> > >> >> >> >> > to
> > >> >> >> >> > drop windows support.  The change here is improving our
> > support
> > >> >> >> >> > and
> > >> >> >> >> > dropping
> > >> >> >> >> > the requirement of cigwin.  We had Nutch windows users on the
> > >> >> >> >> > list
> > >> >> >> >> > in
> > >> >> >> >> > 2006
> > >> >> >> >> > and we've been supporting windows FS requirements since
> > >> >> >> >> > inception.
> > >> >> >> >> >
> > >> >> >> >> > - A little pragmatism will go a long way.  As a community
> > we've
> > >> >> >> >> > got
> > >> >> >> >> > to
> > >> >> >> >> > stay committed to keeping hadoop simple (so it does work on
> > >> >> >> >> > many
> > >> >> >> >> > platforms)
> > >> >> >> >> > and extending it to take advantage of key emerging
> > OS/hardware
> > >> >> >> >> > features,
> > >> >> >> >> > such as containers, new FSs, virtualization, flash ...  We
> > >> >> >> >> > should
> > >> >> >> >> > all
> > >> >> >> >> > plan
> > >> >> >> >> > to let new features & optimizations emerge that don't work
> > >> >> >> >> > everywhere, if
> > >> >> >> >> > they are compelling and central to hadoop's mission of being
> > >> >> >> >> > THE
> > >> >> >> >> > best
> > >> >> >> >> > fabric
> > >> >> >> >> > for storing and processing big data.
> > >> >> >> >> >
> > >> >> >> >> > - A UI project like KDE has to deal with the MANY differences
> > >> >> >> >> > between
> > >> >> >> >> > windows and linux UI APIs.  Hadoop faces no such complex
> > >> >> >> >> > challenge
> > >> >> >> >> > and hence
> > >> >> >> >> > can be maintained from a single codeline IMO.  It is mostly
> > >> >> >> >> > abstracted from
> > >> >> >> >> > the OS APIs via Java and our design choices.  Where it is not
> > >> >> >> >> > we
> > >> >> >> >> > can
> > >> >> >> >> > continue to add plugable abstractions.
> > >> >> >> >> >
> > >> >> >> >
> > >> >> >> >
> > >> >> >
> > >> >> >
> > >> >
> > >> >
> > >
> > >
> >

Re: [Vote] Merge branch-trunk-win to trunk

[Konstantin Boudnik <co...@apache.org>]

It doesn't look like any progress has been done on the ticket below in the
last 3 weeks. And now branch-2 can't be compiled because of 

hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSShell.java:[895,15]
WINDOWS is not public in org.apache.hadoop.fs.Path; cannot be accessed from
outside package

That's exactly why I was -1'ing this...
  Cos

On Mon, Mar 04, 2013 at 05:41PM, Matt Foley wrote:
> Thanks, gentlemen.  I've opened and taken responsibility for
> https://issues.apache.org/jira/browse/HADOOP-9359.  Giri Kesavan has agreed
> to help with the parts that require Jenkins admin access.
> 
> Thanks,
> --Matt
> 
> 
> 
> On Mon, Mar 4, 2013 at 5:00 PM, Konstantin Shvachko <sh...@gmail.com>wrote:
> 
> > +1 on the merge.
> >
> > I am glad we agreed.
> > Having Jira to track the CI effort is a good idea.
> >
> > Thanks,
> > --Konstantin
> >
> > On Mon, Mar 4, 2013 at 3:29 PM, Matt Foley <mf...@hortonworks.com> wrote:
> > > Thanks.  I agree Windows -1's in test-patch should not block commits.
> > >
> > > --Matt
> > >
> > >
> > >
> > > On Mon, Mar 4, 2013 at 2:30 PM, Konstantin Shvachko <
> > shv.hadoop@gmail.com>
> > > wrote:
> > >>
> > >> On Mon, Mar 4, 2013 at 12:22 PM, Matt Foley <mf...@hortonworks.com>
> > >> wrote:
> > >> > Konstantine, you have voted -1, and stated some requirements before
> > >> > you'll
> > >> > withdraw that -1.  As I plan to do work to fulfill those
> > requirements, I
> > >> > want to make sure that what I'm proposing will, in fact, satisfy you.
> > >> > That's why I'm asking, if we implement full "test-patch" integration
> > for
> > >> > Windows, does it seem to you that that would provide adequate support?
> > >>
> > >> Yes.
> > >>
> > >> > I have learned not to presume that my interpretation is correct.  My
> > >> > interpretation of item #1 is that test-patch provides pre-commit
> > build,
> > >> > so
> > >> > it would satisfy item #1.  But rather than assuming that I am
> > >> > interpreting
> > >> > it correctly, I simply want your agreement that it would, or if not,
> > >> > clarification why it won't.
> > >>
> > >> I agree it will satisfy my item #1.
> > >> I did not agree in my previous email, but I changed my mind based on
> > >> the latest discussion. I have to explain why now.
> > >> I was proposing nightly build because I did not want pre-commit build
> > >> for Windows block commits to Linux. But if people are fine just ignoring
> > >> -1s for the Windows part of the build it should be good.
> > >>
> > >> > Regarding item #2, it is also my interpretation that test-patch
> > provides
> > >> > an
> > >> > on-demand (perhaps 20-minutes deferred) Jenkins build and unit test,
> > >> > with
> > >> > logs available to the developer, so it would satisfy item #2.  But
> > >> > rather
> > >> > than assuming that I am interpreting it correctly, I simply want your
> > >> > agreement that it would, or if not, clarification why it won't.
> > >>
> > >> It will satisfy my item #2 in the following way:
> > >> I can duplicate your pre-commit build for Windows and add an input
> > >> parameter, which would let people run the build on their patches
> > >> chosen from local machine rather than attaching them to Jiras.
> > >>
> > >> Thanks,
> > >> --Konstantin
> > >>
> > >> > In agile terms, you are the Owner of these requirements.  Please give
> > me
> > >> > owner feedback as to whether my proposed work sounds like it will
> > >> > satisfy
> > >> > the requirements.
> > >> >
> > >> > Thank you,
> > >> > --Matt
> > >> >
> > >> >
> > >> > On Sun, Mar 3, 2013 at 12:16 PM, Konstantin Shvachko
> > >> > <sh...@gmail.com>
> > >> > wrote:
> > >> >>
> > >> >> Didn't I explain in details what I am asking for?
> > >> >>
> > >> >> Thanks,
> > >> >> --Konst
> > >> >>
> > >> >> On Sun, Mar 3, 2013 at 11:08 AM, Matt Foley <mf...@hortonworks.com>
> > >> >> wrote:
> > >> >> > Hi Konstantin,
> > >> >> > I'd like to point out two things:
> > >> >> > First, I already committed in this thread (email of Thu, Feb 28,
> > 2013
> > >> >> > at
> > >> >> > 6:01 PM) to providing CI for Windows builds.  So please stop acting
> > >> >> > like
> > >> >> > I'm
> > >> >> > resisting this idea or something.
> > >> >> > Second, you didn't answer my question, you just kvetched about the
> > >> >> > phrasing.
> > >> >> > So I ask again:
> > >> >> >
> > >> >> > Will providing full "test-patch" integration (pre-commit build and
> > >> >> > unit
> > >> >> > test
> > >> >> > triggered by Jira "Patch Available" state) satisfy your request for
> > >> >> > functionality #1 and #2?  Yes or no, please.
> > >> >> >
> > >> >> > Thanks,
> > >> >> > --Matt
> > >> >> >
> > >> >> >
> > >> >> > On Sat, Mar 2, 2013 at 7:32 PM, Konstantin Shvachko
> > >> >> > <sh...@gmail.com>
> > >> >> > wrote:
> > >> >> >>
> > >> >> >> Hi Matt,
> > >> >> >>
> > >> >> >> On Sat, Mar 2, 2013 at 12:32 PM, Matt Foley <
> > mfoley@hortonworks.com>
> > >> >> >> wrote:
> > >> >> >> > Konstantin,
> > >> >> >> > I would like to explore what it would take to remove this
> > >> >> >> > perceived
> > >> >> >> > impediment --
> > >> >> >>
> > >> >> >> Glad you decided to explore. Thank you.
> > >> >> >>
> > >> >> >> > although I reserve the right to argue that this is not
> > >> >> >> > pre-requisite to merging the cross-platform support patch.
> > >> >> >>
> > >> >> >> It's your right indeed. So as mine to question what the platform
> > >> >> >> support means for you, which I believe remained unclear.
> > >> >> >> I do not impede the change as you should have noticed. My
> > >> >> >> requirement
> > >> >> >> comes from my perception of the support, which means to me exactly
> > >> >> >> two
> > >> >> >> things:
> > >> >> >> 1. The ability to recognise the code is broken for the platform
> > >> >> >> 2. The ability to test new patches on the platform
> > >> >> >> The latter is problematic, as many noticed in this thread, for
> > those
> > >> >> >> whose customary environment does not include Windows.
> > >> >> >>
> > >> >> >> > If we implemented full "test-patch" support for Windows on
> > trunk,
> > >> >> >> > would
> > >> >> >> > that
> > >> >> >> > fulfill both your items #1 and #2?  Please note that:
> > >> >> >> > a) Pushing the "Patch Available" button in Jira shall cause a
> > >> >> >> > pre-commit
> > >> >> >> > build to start within, I believe, 20 minutes.
> > >> >> >> > b) That build keeps logs for both java build and unit tests for
> > >> >> >> > several
> > >> >> >> > days, that are accessible to all viewers.
> > >> >> >>
> > >> >> >> In item #1 I mostly asking for the nightly build, which is simpler
> > >> >> >> than "test-patch". The latter would be ideal from the platform
> > >> >> >> support
> > >> >> >> viewpoint, but it is for the community to decide if we want to add
> > >> >> >> extra +3 hours to the build.
> > >> >> >> Nightly build in my understanding is triggered by the timer rather
> > >> >> >> than by Jira's "submit patch" button.  On Jenkins build
> > >> >> >> configuration
> > >> >> >> you can specify it under "Build periodically".
> > >> >> >>
> > >> >> >> > So, does this provide sufficient on-demand support that we don't
> > >> >> >> > have
> > >> >> >> > to
> > >> >> >> > implement a whole new on-demand VM support structure of some
> > sort
> > >> >> >> > for
> > >> >> >> > #2
> > >> >> >> > (which would be an extraordinary and impractical demand)?
> > >> >> >>
> > >> >> >> I did not mention VMs. Item #2 means a build, which runs
> > >> >> >> "test-patch"
> > >> >> >> target with the file specified by a user (instead of a jira
> > >> >> >> attachment).
> > >> >> >> When user clicks "Build Now" link a box is displayed where the
> > user
> > >> >> >> can enter the file path containing the patch. This can be
> > specified
> > >> >> >> in
> > >> >> >> the Build Configuration under "This build is parameterized" by
> > >> >> >> choosing AddParameter / FileParameter. The build can run on the
> > same
> > >> >> >> Windows machine as the nightly build.
> > >> >> >> Such build will let people test their patches for Windows on
> > Jenkins
> > >> >> >> if they don't posses a license for the right version of Windows.
> > >> >> >> I hope this will not turn into extraordinary or impractical
> > effort.
> > >> >> >>
> > >> >> >> Thanks,
> > >> >> >> --Konst
> > >> >> >>
> > >> >> >> > Thanks,
> > >> >> >> > --Matt
> > >> >> >> >
> > >> >> >> >
> > >> >> >> > On Fri, Mar 1, 2013 at 12:18 PM, Konstantin Shvachko
> > >> >> >> > <sh...@gmail.com>
> > >> >> >> > wrote:
> > >> >> >> >>
> > >> >> >> >> -1
> > >> >> >> >> We should have a CI infrastructure in place before we can
> > commit
> > >> >> >> >> to
> > >> >> >> >> supporting Windows platform.
> > >> >> >> >>
> > >> >> >> >> Eric is right Win/Cygwin was supported since day one.
> > >> >> >> >> I had a Windows box under my desk running nightly builds back
> > in
> > >> >> >> >> 2006-07.
> > >> >> >> >> People were irritated but I was filing windows bugs until 0.22
> > >> >> >> >> release.
> > >> >> >> >> Times changing and I am glad to see wider support for Win
> > >> >> >> >> platform.
> > >> >> >> >>
> > >> >> >> >> But in order to make it work you guys need to put the CI
> > process
> > >> >> >> >> in
> > >> >> >> >> place
> > >> >> >> >>
> > >> >> >> >> 1. windows jenkins build: could be nightly or PreCommit.
> > >> >> >> >> - Nightly would mean that changes can be committed to trunk
> > based
> > >> >> >> >> on
> > >> >> >> >> linux PreCommit build. And people will file bugs if the change
> > >> >> >> >> broke
> > >> >> >> >> Windows nightly build.
> > >> >> >> >> - PreCommit-win build will mean automatic reporting failed
> > tests
> > >> >> >> >> to
> > >> >> >> >> respective jira blocking commits the same way as it is now with
> > >> >> >> >> linux
> > >> >> >> >> PreCommit builds.
> > >> >> >> >> We should discuss which way is more efficient for developers.
> > >> >> >> >>
> > >> >> >> >> 2. On-demand-windows Jenkins build.
> > >> >> >> >> I see it as a build to which I can attach my patch and the
> > build
> > >> >> >> >> will
> > >> >> >> >> run my changes on a dedicated windows box.
> > >> >> >> >> That way people can test their changes without having personal
> > >> >> >> >> windows
> > >> >> >> >> nodes.
> > >> >> >> >>
> > >> >> >> >> I think this is the minimal set of requirement for us to be
> > able
> > >> >> >> >> to
> > >> >> >> >> commit to the new platform.
> > >> >> >> >> Right now I see only one windows related build
> > >> >> >> >> https://builds.apache.org/view/Hadoop/job/Hadoop-1-win/
> > >> >> >> >> Which was failing since Sept 8, 2012 and did not run in the
> > last
> > >> >> >> >> month.
> > >> >> >> >>
> > >> >> >> >> Thanks,
> > >> >> >> >> --Konst
> > >> >> >> >>
> > >> >> >> >> On Thu, Feb 28, 2013 at 8:47 PM, Eric Baldeschwieler
> > >> >> >> >> <er...@hortonworks.com> wrote:
> > >> >> >> >> > +1 (non-binding)
> > >> >> >> >> >
> > >> >> >> >> > A few of observations:
> > >> >> >> >> >
> > >> >> >> >> > - Windows has actually been a supported platform for Hadoop
> > >> >> >> >> > since
> > >> >> >> >> > 0.1
> > >> >> >> >> > .
> > >> >> >> >> > Doug championed supporting windows then and we've continued
> > to
> > >> >> >> >> > do
> > >> >> >> >> > it
> > >> >> >> >> > with
> > >> >> >> >> > varying vigor over time.  To my knowledge we've never made a
> > >> >> >> >> > decision
> > >> >> >> >> > to
> > >> >> >> >> > drop windows support.  The change here is improving our
> > support
> > >> >> >> >> > and
> > >> >> >> >> > dropping
> > >> >> >> >> > the requirement of cigwin.  We had Nutch windows users on the
> > >> >> >> >> > list
> > >> >> >> >> > in
> > >> >> >> >> > 2006
> > >> >> >> >> > and we've been supporting windows FS requirements since
> > >> >> >> >> > inception.
> > >> >> >> >> >
> > >> >> >> >> > - A little pragmatism will go a long way.  As a community
> > we've
> > >> >> >> >> > got
> > >> >> >> >> > to
> > >> >> >> >> > stay committed to keeping hadoop simple (so it does work on
> > >> >> >> >> > many
> > >> >> >> >> > platforms)
> > >> >> >> >> > and extending it to take advantage of key emerging
> > OS/hardware
> > >> >> >> >> > features,
> > >> >> >> >> > such as containers, new FSs, virtualization, flash ...  We
> > >> >> >> >> > should
> > >> >> >> >> > all
> > >> >> >> >> > plan
> > >> >> >> >> > to let new features & optimizations emerge that don't work
> > >> >> >> >> > everywhere, if
> > >> >> >> >> > they are compelling and central to hadoop's mission of being
> > >> >> >> >> > THE
> > >> >> >> >> > best
> > >> >> >> >> > fabric
> > >> >> >> >> > for storing and processing big data.
> > >> >> >> >> >
> > >> >> >> >> > - A UI project like KDE has to deal with the MANY differences
> > >> >> >> >> > between
> > >> >> >> >> > windows and linux UI APIs.  Hadoop faces no such complex
> > >> >> >> >> > challenge
> > >> >> >> >> > and hence
> > >> >> >> >> > can be maintained from a single codeline IMO.  It is mostly
> > >> >> >> >> > abstracted from
> > >> >> >> >> > the OS APIs via Java and our design choices.  Where it is not
> > >> >> >> >> > we
> > >> >> >> >> > can
> > >> >> >> >> > continue to add plugable abstractions.
> > >> >> >> >> >
> > >> >> >> >
> > >> >> >> >
> > >> >> >
> > >> >> >
> > >> >
> > >> >
> > >
> > >
> >

Re: [Vote] Merge branch-trunk-win to trunk

[Suresh Srinivas <su...@hortonworks.com>]

Thank you all for voting and participating in the discussions.

With 11 +1s from committers (more than the required 3 +1s from
active committers per the Hadoop bylaws), 1 +0, 8 +1s from other
contributors, and no -1s the merge vote passes.

I have committed the consolidated patch from branch-trunk-win
to trunk. People who are interested in following up on the progress
related to Jenkins setup for Windows and other work that came up
during the discussion, please watch:
https://issues.apache.org/jira/browse/HADOOP-9359

Regards,
Suresh


On Mon, Mar 4, 2013 at 5:41 PM, Matt Foley <mf...@hortonworks.com> wrote:

> Thanks, gentlemen.  I've opened and taken responsibility for
> https://issues.apache.org/jira/browse/HADOOP-9359.  Giri Kesavan has
> agreed
> to help with the parts that require Jenkins admin access.
>
> Thanks,
> --Matt
>
>
>
> On Mon, Mar 4, 2013 at 5:00 PM, Konstantin Shvachko <shv.hadoop@gmail.com
> >wrote:
>
> > +1 on the merge.
> >
> > I am glad we agreed.
> > Having Jira to track the CI effort is a good idea.
> >
> > Thanks,
> > --Konstantin
> >
> > On Mon, Mar 4, 2013 at 3:29 PM, Matt Foley <mf...@hortonworks.com>
> wrote:
> > > Thanks.  I agree Windows -1's in test-patch should not block commits.
> > >
> > > --Matt
> > >
> > >
> > >
> > > On Mon, Mar 4, 2013 at 2:30 PM, Konstantin Shvachko <
> > shv.hadoop@gmail.com>
> > > wrote:
> > >>
> > >> On Mon, Mar 4, 2013 at 12:22 PM, Matt Foley <mf...@hortonworks.com>
> > >> wrote:
> > >> > Konstantine, you have voted -1, and stated some requirements before
> > >> > you'll
> > >> > withdraw that -1.  As I plan to do work to fulfill those
> > requirements, I
> > >> > want to make sure that what I'm proposing will, in fact, satisfy
> you.
> > >> > That's why I'm asking, if we implement full "test-patch" integration
> > for
> > >> > Windows, does it seem to you that that would provide adequate
> support?
> > >>
> > >> Yes.
> > >>
> > >> > I have learned not to presume that my interpretation is correct.  My
> > >> > interpretation of item #1 is that test-patch provides pre-commit
> > build,
> > >> > so
> > >> > it would satisfy item #1.  But rather than assuming that I am
> > >> > interpreting
> > >> > it correctly, I simply want your agreement that it would, or if not,
> > >> > clarification why it won't.
> > >>
> > >> I agree it will satisfy my item #1.
> > >> I did not agree in my previous email, but I changed my mind based on
> > >> the latest discussion. I have to explain why now.
> > >> I was proposing nightly build because I did not want pre-commit build
> > >> for Windows block commits to Linux. But if people are fine just
> ignoring
> > >> -1s for the Windows part of the build it should be good.
> > >>
> > >> > Regarding item #2, it is also my interpretation that test-patch
> > provides
> > >> > an
> > >> > on-demand (perhaps 20-minutes deferred) Jenkins build and unit test,
> > >> > with
> > >> > logs available to the developer, so it would satisfy item #2.  But
> > >> > rather
> > >> > than assuming that I am interpreting it correctly, I simply want
> your
> > >> > agreement that it would, or if not, clarification why it won't.
> > >>
> > >> It will satisfy my item #2 in the following way:
> > >> I can duplicate your pre-commit build for Windows and add an input
> > >> parameter, which would let people run the build on their patches
> > >> chosen from local machine rather than attaching them to Jiras.
> > >>
> > >> Thanks,
> > >> --Konstantin
> > >>
> > >> > In agile terms, you are the Owner of these requirements.  Please
> give
> > me
> > >> > owner feedback as to whether my proposed work sounds like it will
> > >> > satisfy
> > >> > the requirements.
> > >> >
> > >> > Thank you,
> > >> > --Matt
> > >> >
> > >> >
> > >> > On Sun, Mar 3, 2013 at 12:16 PM, Konstantin Shvachko
> > >> > <sh...@gmail.com>
> > >> > wrote:
> > >> >>
> > >> >> Didn't I explain in details what I am asking for?
> > >> >>
> > >> >> Thanks,
> > >> >> --Konst
> > >> >>
> > >> >> On Sun, Mar 3, 2013 at 11:08 AM, Matt Foley <
> mfoley@hortonworks.com>
> > >> >> wrote:
> > >> >> > Hi Konstantin,
> > >> >> > I'd like to point out two things:
> > >> >> > First, I already committed in this thread (email of Thu, Feb 28,
> > 2013
> > >> >> > at
> > >> >> > 6:01 PM) to providing CI for Windows builds.  So please stop
> acting
> > >> >> > like
> > >> >> > I'm
> > >> >> > resisting this idea or something.
> > >> >> > Second, you didn't answer my question, you just kvetched about
> the
> > >> >> > phrasing.
> > >> >> > So I ask again:
> > >> >> >
> > >> >> > Will providing full "test-patch" integration (pre-commit build
> and
> > >> >> > unit
> > >> >> > test
> > >> >> > triggered by Jira "Patch Available" state) satisfy your request
> for
> > >> >> > functionality #1 and #2?  Yes or no, please.
> > >> >> >
> > >> >> > Thanks,
> > >> >> > --Matt
> > >> >> >
> > >> >> >
> > >> >> > On Sat, Mar 2, 2013 at 7:32 PM, Konstantin Shvachko
> > >> >> > <sh...@gmail.com>
> > >> >> > wrote:
> > >> >> >>
> > >> >> >> Hi Matt,
> > >> >> >>
> > >> >> >> On Sat, Mar 2, 2013 at 12:32 PM, Matt Foley <
> > mfoley@hortonworks.com>
> > >> >> >> wrote:
> > >> >> >> > Konstantin,
> > >> >> >> > I would like to explore what it would take to remove this
> > >> >> >> > perceived
> > >> >> >> > impediment --
> > >> >> >>
> > >> >> >> Glad you decided to explore. Thank you.
> > >> >> >>
> > >> >> >> > although I reserve the right to argue that this is not
> > >> >> >> > pre-requisite to merging the cross-platform support patch.
> > >> >> >>
> > >> >> >> It's your right indeed. So as mine to question what the platform
> > >> >> >> support means for you, which I believe remained unclear.
> > >> >> >> I do not impede the change as you should have noticed. My
> > >> >> >> requirement
> > >> >> >> comes from my perception of the support, which means to me
> exactly
> > >> >> >> two
> > >> >> >> things:
> > >> >> >> 1. The ability to recognise the code is broken for the platform
> > >> >> >> 2. The ability to test new patches on the platform
> > >> >> >> The latter is problematic, as many noticed in this thread, for
> > those
> > >> >> >> whose customary environment does not include Windows.
> > >> >> >>
> > >> >> >> > If we implemented full "test-patch" support for Windows on
> > trunk,
> > >> >> >> > would
> > >> >> >> > that
> > >> >> >> > fulfill both your items #1 and #2?  Please note that:
> > >> >> >> > a) Pushing the "Patch Available" button in Jira shall cause a
> > >> >> >> > pre-commit
> > >> >> >> > build to start within, I believe, 20 minutes.
> > >> >> >> > b) That build keeps logs for both java build and unit tests
> for
> > >> >> >> > several
> > >> >> >> > days, that are accessible to all viewers.
> > >> >> >>
> > >> >> >> In item #1 I mostly asking for the nightly build, which is
> simpler
> > >> >> >> than "test-patch". The latter would be ideal from the platform
> > >> >> >> support
> > >> >> >> viewpoint, but it is for the community to decide if we want to
> add
> > >> >> >> extra +3 hours to the build.
> > >> >> >> Nightly build in my understanding is triggered by the timer
> rather
> > >> >> >> than by Jira's "submit patch" button.  On Jenkins build
> > >> >> >> configuration
> > >> >> >> you can specify it under "Build periodically".
> > >> >> >>
> > >> >> >> > So, does this provide sufficient on-demand support that we
> don't
> > >> >> >> > have
> > >> >> >> > to
> > >> >> >> > implement a whole new on-demand VM support structure of some
> > sort
> > >> >> >> > for
> > >> >> >> > #2
> > >> >> >> > (which would be an extraordinary and impractical demand)?
> > >> >> >>
> > >> >> >> I did not mention VMs. Item #2 means a build, which runs
> > >> >> >> "test-patch"
> > >> >> >> target with the file specified by a user (instead of a jira
> > >> >> >> attachment).
> > >> >> >> When user clicks "Build Now" link a box is displayed where the
> > user
> > >> >> >> can enter the file path containing the patch. This can be
> > specified
> > >> >> >> in
> > >> >> >> the Build Configuration under "This build is parameterized" by
> > >> >> >> choosing AddParameter / FileParameter. The build can run on the
> > same
> > >> >> >> Windows machine as the nightly build.
> > >> >> >> Such build will let people test their patches for Windows on
> > Jenkins
> > >> >> >> if they don't posses a license for the right version of Windows.
> > >> >> >> I hope this will not turn into extraordinary or impractical
> > effort.
> > >> >> >>
> > >> >> >> Thanks,
> > >> >> >> --Konst
> > >> >> >>
> > >> >> >> > Thanks,
> > >> >> >> > --Matt
> > >> >> >> >
> > >> >> >> >
> > >> >> >> > On Fri, Mar 1, 2013 at 12:18 PM, Konstantin Shvachko
> > >> >> >> > <sh...@gmail.com>
> > >> >> >> > wrote:
> > >> >> >> >>
> > >> >> >> >> -1
> > >> >> >> >> We should have a CI infrastructure in place before we can
> > commit
> > >> >> >> >> to
> > >> >> >> >> supporting Windows platform.
> > >> >> >> >>
> > >> >> >> >> Eric is right Win/Cygwin was supported since day one.
> > >> >> >> >> I had a Windows box under my desk running nightly builds back
> > in
> > >> >> >> >> 2006-07.
> > >> >> >> >> People were irritated but I was filing windows bugs until
> 0.22
> > >> >> >> >> release.
> > >> >> >> >> Times changing and I am glad to see wider support for Win
> > >> >> >> >> platform.
> > >> >> >> >>
> > >> >> >> >> But in order to make it work you guys need to put the CI
> > process
> > >> >> >> >> in
> > >> >> >> >> place
> > >> >> >> >>
> > >> >> >> >> 1. windows jenkins build: could be nightly or PreCommit.
> > >> >> >> >> - Nightly would mean that changes can be committed to trunk
> > based
> > >> >> >> >> on
> > >> >> >> >> linux PreCommit build. And people will file bugs if the
> change
> > >> >> >> >> broke
> > >> >> >> >> Windows nightly build.
> > >> >> >> >> - PreCommit-win build will mean automatic reporting failed
> > tests
> > >> >> >> >> to
> > >> >> >> >> respective jira blocking commits the same way as it is now
> with
> > >> >> >> >> linux
> > >> >> >> >> PreCommit builds.
> > >> >> >> >> We should discuss which way is more efficient for developers.
> > >> >> >> >>
> > >> >> >> >> 2. On-demand-windows Jenkins build.
> > >> >> >> >> I see it as a build to which I can attach my patch and the
> > build
> > >> >> >> >> will
> > >> >> >> >> run my changes on a dedicated windows box.
> > >> >> >> >> That way people can test their changes without having
> personal
> > >> >> >> >> windows
> > >> >> >> >> nodes.
> > >> >> >> >>
> > >> >> >> >> I think this is the minimal set of requirement for us to be
> > able
> > >> >> >> >> to
> > >> >> >> >> commit to the new platform.
> > >> >> >> >> Right now I see only one windows related build
> > >> >> >> >> https://builds.apache.org/view/Hadoop/job/Hadoop-1-win/
> > >> >> >> >> Which was failing since Sept 8, 2012 and did not run in the
> > last
> > >> >> >> >> month.
> > >> >> >> >>
> > >> >> >> >> Thanks,
> > >> >> >> >> --Konst
> > >> >> >> >>
> > >> >> >> >> On Thu, Feb 28, 2013 at 8:47 PM, Eric Baldeschwieler
> > >> >> >> >> <er...@hortonworks.com> wrote:
> > >> >> >> >> > +1 (non-binding)
> > >> >> >> >> >
> > >> >> >> >> > A few of observations:
> > >> >> >> >> >
> > >> >> >> >> > - Windows has actually been a supported platform for Hadoop
> > >> >> >> >> > since
> > >> >> >> >> > 0.1
> > >> >> >> >> > .
> > >> >> >> >> > Doug championed supporting windows then and we've continued
> > to
> > >> >> >> >> > do
> > >> >> >> >> > it
> > >> >> >> >> > with
> > >> >> >> >> > varying vigor over time.  To my knowledge we've never made
> a
> > >> >> >> >> > decision
> > >> >> >> >> > to
> > >> >> >> >> > drop windows support.  The change here is improving our
> > support
> > >> >> >> >> > and
> > >> >> >> >> > dropping
> > >> >> >> >> > the requirement of cigwin.  We had Nutch windows users on
> the
> > >> >> >> >> > list
> > >> >> >> >> > in
> > >> >> >> >> > 2006
> > >> >> >> >> > and we've been supporting windows FS requirements since
> > >> >> >> >> > inception.
> > >> >> >> >> >
> > >> >> >> >> > - A little pragmatism will go a long way.  As a community
> > we've
> > >> >> >> >> > got
> > >> >> >> >> > to
> > >> >> >> >> > stay committed to keeping hadoop simple (so it does work on
> > >> >> >> >> > many
> > >> >> >> >> > platforms)
> > >> >> >> >> > and extending it to take advantage of key emerging
> > OS/hardware
> > >> >> >> >> > features,
> > >> >> >> >> > such as containers, new FSs, virtualization, flash ...  We
> > >> >> >> >> > should
> > >> >> >> >> > all
> > >> >> >> >> > plan
> > >> >> >> >> > to let new features & optimizations emerge that don't work
> > >> >> >> >> > everywhere, if
> > >> >> >> >> > they are compelling and central to hadoop's mission of
> being
> > >> >> >> >> > THE
> > >> >> >> >> > best
> > >> >> >> >> > fabric
> > >> >> >> >> > for storing and processing big data.
> > >> >> >> >> >
> > >> >> >> >> > - A UI project like KDE has to deal with the MANY
> differences
> > >> >> >> >> > between
> > >> >> >> >> > windows and linux UI APIs.  Hadoop faces no such complex
> > >> >> >> >> > challenge
> > >> >> >> >> > and hence
> > >> >> >> >> > can be maintained from a single codeline IMO.  It is mostly
> > >> >> >> >> > abstracted from
> > >> >> >> >> > the OS APIs via Java and our design choices.  Where it is
> not
> > >> >> >> >> > we
> > >> >> >> >> > can
> > >> >> >> >> > continue to add plugable abstractions.
> > >> >> >> >> >
> > >> >> >> >
> > >> >> >> >
> > >> >> >
> > >> >> >
> > >> >
> > >> >
> > >
> > >
> >
>



-- 
http://hortonworks.com/download/

Re: [Vote] Merge branch-trunk-win to trunk

[Suresh Srinivas <su...@hortonworks.com>]

Thank you all for voting and participating in the discussions.

With 11 +1s from committers (more than the required 3 +1s from
active committers per the Hadoop bylaws), 1 +0, 8 +1s from other
contributors, and no -1s the merge vote passes.

I have committed the consolidated patch from branch-trunk-win
to trunk. People who are interested in following up on the progress
related to Jenkins setup for Windows and other work that came up
during the discussion, please watch:
https://issues.apache.org/jira/browse/HADOOP-9359

Regards,
Suresh


On Mon, Mar 4, 2013 at 5:41 PM, Matt Foley <mf...@hortonworks.com> wrote:

> Thanks, gentlemen.  I've opened and taken responsibility for
> https://issues.apache.org/jira/browse/HADOOP-9359.  Giri Kesavan has
> agreed
> to help with the parts that require Jenkins admin access.
>
> Thanks,
> --Matt
>
>
>
> On Mon, Mar 4, 2013 at 5:00 PM, Konstantin Shvachko <shv.hadoop@gmail.com
> >wrote:
>
> > +1 on the merge.
> >
> > I am glad we agreed.
> > Having Jira to track the CI effort is a good idea.
> >
> > Thanks,
> > --Konstantin
> >
> > On Mon, Mar 4, 2013 at 3:29 PM, Matt Foley <mf...@hortonworks.com>
> wrote:
> > > Thanks.  I agree Windows -1's in test-patch should not block commits.
> > >
> > > --Matt
> > >
> > >
> > >
> > > On Mon, Mar 4, 2013 at 2:30 PM, Konstantin Shvachko <
> > shv.hadoop@gmail.com>
> > > wrote:
> > >>
> > >> On Mon, Mar 4, 2013 at 12:22 PM, Matt Foley <mf...@hortonworks.com>
> > >> wrote:
> > >> > Konstantine, you have voted -1, and stated some requirements before
> > >> > you'll
> > >> > withdraw that -1.  As I plan to do work to fulfill those
> > requirements, I
> > >> > want to make sure that what I'm proposing will, in fact, satisfy
> you.
> > >> > That's why I'm asking, if we implement full "test-patch" integration
> > for
> > >> > Windows, does it seem to you that that would provide adequate
> support?
> > >>
> > >> Yes.
> > >>
> > >> > I have learned not to presume that my interpretation is correct.  My
> > >> > interpretation of item #1 is that test-patch provides pre-commit
> > build,
> > >> > so
> > >> > it would satisfy item #1.  But rather than assuming that I am
> > >> > interpreting
> > >> > it correctly, I simply want your agreement that it would, or if not,
> > >> > clarification why it won't.
> > >>
> > >> I agree it will satisfy my item #1.
> > >> I did not agree in my previous email, but I changed my mind based on
> > >> the latest discussion. I have to explain why now.
> > >> I was proposing nightly build because I did not want pre-commit build
> > >> for Windows block commits to Linux. But if people are fine just
> ignoring
> > >> -1s for the Windows part of the build it should be good.
> > >>
> > >> > Regarding item #2, it is also my interpretation that test-patch
> > provides
> > >> > an
> > >> > on-demand (perhaps 20-minutes deferred) Jenkins build and unit test,
> > >> > with
> > >> > logs available to the developer, so it would satisfy item #2.  But
> > >> > rather
> > >> > than assuming that I am interpreting it correctly, I simply want
> your
> > >> > agreement that it would, or if not, clarification why it won't.
> > >>
> > >> It will satisfy my item #2 in the following way:
> > >> I can duplicate your pre-commit build for Windows and add an input
> > >> parameter, which would let people run the build on their patches
> > >> chosen from local machine rather than attaching them to Jiras.
> > >>
> > >> Thanks,
> > >> --Konstantin
> > >>
> > >> > In agile terms, you are the Owner of these requirements.  Please
> give
> > me
> > >> > owner feedback as to whether my proposed work sounds like it will
> > >> > satisfy
> > >> > the requirements.
> > >> >
> > >> > Thank you,
> > >> > --Matt
> > >> >
> > >> >
> > >> > On Sun, Mar 3, 2013 at 12:16 PM, Konstantin Shvachko
> > >> > <sh...@gmail.com>
> > >> > wrote:
> > >> >>
> > >> >> Didn't I explain in details what I am asking for?
> > >> >>
> > >> >> Thanks,
> > >> >> --Konst
> > >> >>
> > >> >> On Sun, Mar 3, 2013 at 11:08 AM, Matt Foley <
> mfoley@hortonworks.com>
> > >> >> wrote:
> > >> >> > Hi Konstantin,
> > >> >> > I'd like to point out two things:
> > >> >> > First, I already committed in this thread (email of Thu, Feb 28,
> > 2013
> > >> >> > at
> > >> >> > 6:01 PM) to providing CI for Windows builds.  So please stop
> acting
> > >> >> > like
> > >> >> > I'm
> > >> >> > resisting this idea or something.
> > >> >> > Second, you didn't answer my question, you just kvetched about
> the
> > >> >> > phrasing.
> > >> >> > So I ask again:
> > >> >> >
> > >> >> > Will providing full "test-patch" integration (pre-commit build
> and
> > >> >> > unit
> > >> >> > test
> > >> >> > triggered by Jira "Patch Available" state) satisfy your request
> for
> > >> >> > functionality #1 and #2?  Yes or no, please.
> > >> >> >
> > >> >> > Thanks,
> > >> >> > --Matt
> > >> >> >
> > >> >> >
> > >> >> > On Sat, Mar 2, 2013 at 7:32 PM, Konstantin Shvachko
> > >> >> > <sh...@gmail.com>
> > >> >> > wrote:
> > >> >> >>
> > >> >> >> Hi Matt,
> > >> >> >>
> > >> >> >> On Sat, Mar 2, 2013 at 12:32 PM, Matt Foley <
> > mfoley@hortonworks.com>
> > >> >> >> wrote:
> > >> >> >> > Konstantin,
> > >> >> >> > I would like to explore what it would take to remove this
> > >> >> >> > perceived
> > >> >> >> > impediment --
> > >> >> >>
> > >> >> >> Glad you decided to explore. Thank you.
> > >> >> >>
> > >> >> >> > although I reserve the right to argue that this is not
> > >> >> >> > pre-requisite to merging the cross-platform support patch.
> > >> >> >>
> > >> >> >> It's your right indeed. So as mine to question what the platform
> > >> >> >> support means for you, which I believe remained unclear.
> > >> >> >> I do not impede the change as you should have noticed. My
> > >> >> >> requirement
> > >> >> >> comes from my perception of the support, which means to me
> exactly
> > >> >> >> two
> > >> >> >> things:
> > >> >> >> 1. The ability to recognise the code is broken for the platform
> > >> >> >> 2. The ability to test new patches on the platform
> > >> >> >> The latter is problematic, as many noticed in this thread, for
> > those
> > >> >> >> whose customary environment does not include Windows.
> > >> >> >>
> > >> >> >> > If we implemented full "test-patch" support for Windows on
> > trunk,
> > >> >> >> > would
> > >> >> >> > that
> > >> >> >> > fulfill both your items #1 and #2?  Please note that:
> > >> >> >> > a) Pushing the "Patch Available" button in Jira shall cause a
> > >> >> >> > pre-commit
> > >> >> >> > build to start within, I believe, 20 minutes.
> > >> >> >> > b) That build keeps logs for both java build and unit tests
> for
> > >> >> >> > several
> > >> >> >> > days, that are accessible to all viewers.
> > >> >> >>
> > >> >> >> In item #1 I mostly asking for the nightly build, which is
> simpler
> > >> >> >> than "test-patch". The latter would be ideal from the platform
> > >> >> >> support
> > >> >> >> viewpoint, but it is for the community to decide if we want to
> add
> > >> >> >> extra +3 hours to the build.
> > >> >> >> Nightly build in my understanding is triggered by the timer
> rather
> > >> >> >> than by Jira's "submit patch" button.  On Jenkins build
> > >> >> >> configuration
> > >> >> >> you can specify it under "Build periodically".
> > >> >> >>
> > >> >> >> > So, does this provide sufficient on-demand support that we
> don't
> > >> >> >> > have
> > >> >> >> > to
> > >> >> >> > implement a whole new on-demand VM support structure of some
> > sort
> > >> >> >> > for
> > >> >> >> > #2
> > >> >> >> > (which would be an extraordinary and impractical demand)?
> > >> >> >>
> > >> >> >> I did not mention VMs. Item #2 means a build, which runs
> > >> >> >> "test-patch"
> > >> >> >> target with the file specified by a user (instead of a jira
> > >> >> >> attachment).
> > >> >> >> When user clicks "Build Now" link a box is displayed where the
> > user
> > >> >> >> can enter the file path containing the patch. This can be
> > specified
> > >> >> >> in
> > >> >> >> the Build Configuration under "This build is parameterized" by
> > >> >> >> choosing AddParameter / FileParameter. The build can run on the
> > same
> > >> >> >> Windows machine as the nightly build.
> > >> >> >> Such build will let people test their patches for Windows on
> > Jenkins
> > >> >> >> if they don't posses a license for the right version of Windows.
> > >> >> >> I hope this will not turn into extraordinary or impractical
> > effort.
> > >> >> >>
> > >> >> >> Thanks,
> > >> >> >> --Konst
> > >> >> >>
> > >> >> >> > Thanks,
> > >> >> >> > --Matt
> > >> >> >> >
> > >> >> >> >
> > >> >> >> > On Fri, Mar 1, 2013 at 12:18 PM, Konstantin Shvachko
> > >> >> >> > <sh...@gmail.com>
> > >> >> >> > wrote:
> > >> >> >> >>
> > >> >> >> >> -1
> > >> >> >> >> We should have a CI infrastructure in place before we can
> > commit
> > >> >> >> >> to
> > >> >> >> >> supporting Windows platform.
> > >> >> >> >>
> > >> >> >> >> Eric is right Win/Cygwin was supported since day one.
> > >> >> >> >> I had a Windows box under my desk running nightly builds back
> > in
> > >> >> >> >> 2006-07.
> > >> >> >> >> People were irritated but I was filing windows bugs until
> 0.22
> > >> >> >> >> release.
> > >> >> >> >> Times changing and I am glad to see wider support for Win
> > >> >> >> >> platform.
> > >> >> >> >>
> > >> >> >> >> But in order to make it work you guys need to put the CI
> > process
> > >> >> >> >> in
> > >> >> >> >> place
> > >> >> >> >>
> > >> >> >> >> 1. windows jenkins build: could be nightly or PreCommit.
> > >> >> >> >> - Nightly would mean that changes can be committed to trunk
> > based
> > >> >> >> >> on
> > >> >> >> >> linux PreCommit build. And people will file bugs if the
> change
> > >> >> >> >> broke
> > >> >> >> >> Windows nightly build.
> > >> >> >> >> - PreCommit-win build will mean automatic reporting failed
> > tests
> > >> >> >> >> to
> > >> >> >> >> respective jira blocking commits the same way as it is now
> with
> > >> >> >> >> linux
> > >> >> >> >> PreCommit builds.
> > >> >> >> >> We should discuss which way is more efficient for developers.
> > >> >> >> >>
> > >> >> >> >> 2. On-demand-windows Jenkins build.
> > >> >> >> >> I see it as a build to which I can attach my patch and the
> > build
> > >> >> >> >> will
> > >> >> >> >> run my changes on a dedicated windows box.
> > >> >> >> >> That way people can test their changes without having
> personal
> > >> >> >> >> windows
> > >> >> >> >> nodes.
> > >> >> >> >>
> > >> >> >> >> I think this is the minimal set of requirement for us to be
> > able
> > >> >> >> >> to
> > >> >> >> >> commit to the new platform.
> > >> >> >> >> Right now I see only one windows related build
> > >> >> >> >> https://builds.apache.org/view/Hadoop/job/Hadoop-1-win/
> > >> >> >> >> Which was failing since Sept 8, 2012 and did not run in the
> > last
> > >> >> >> >> month.
> > >> >> >> >>
> > >> >> >> >> Thanks,
> > >> >> >> >> --Konst
> > >> >> >> >>
> > >> >> >> >> On Thu, Feb 28, 2013 at 8:47 PM, Eric Baldeschwieler
> > >> >> >> >> <er...@hortonworks.com> wrote:
> > >> >> >> >> > +1 (non-binding)
> > >> >> >> >> >
> > >> >> >> >> > A few of observations:
> > >> >> >> >> >
> > >> >> >> >> > - Windows has actually been a supported platform for Hadoop
> > >> >> >> >> > since
> > >> >> >> >> > 0.1
> > >> >> >> >> > .
> > >> >> >> >> > Doug championed supporting windows then and we've continued
> > to
> > >> >> >> >> > do
> > >> >> >> >> > it
> > >> >> >> >> > with
> > >> >> >> >> > varying vigor over time.  To my knowledge we've never made
> a
> > >> >> >> >> > decision
> > >> >> >> >> > to
> > >> >> >> >> > drop windows support.  The change here is improving our
> > support
> > >> >> >> >> > and
> > >> >> >> >> > dropping
> > >> >> >> >> > the requirement of cigwin.  We had Nutch windows users on
> the
> > >> >> >> >> > list
> > >> >> >> >> > in
> > >> >> >> >> > 2006
> > >> >> >> >> > and we've been supporting windows FS requirements since
> > >> >> >> >> > inception.
> > >> >> >> >> >
> > >> >> >> >> > - A little pragmatism will go a long way.  As a community
> > we've
> > >> >> >> >> > got
> > >> >> >> >> > to
> > >> >> >> >> > stay committed to keeping hadoop simple (so it does work on
> > >> >> >> >> > many
> > >> >> >> >> > platforms)
> > >> >> >> >> > and extending it to take advantage of key emerging
> > OS/hardware
> > >> >> >> >> > features,
> > >> >> >> >> > such as containers, new FSs, virtualization, flash ...  We
> > >> >> >> >> > should
> > >> >> >> >> > all
> > >> >> >> >> > plan
> > >> >> >> >> > to let new features & optimizations emerge that don't work
> > >> >> >> >> > everywhere, if
> > >> >> >> >> > they are compelling and central to hadoop's mission of
> being
> > >> >> >> >> > THE
> > >> >> >> >> > best
> > >> >> >> >> > fabric
> > >> >> >> >> > for storing and processing big data.
> > >> >> >> >> >
> > >> >> >> >> > - A UI project like KDE has to deal with the MANY
> differences
> > >> >> >> >> > between
> > >> >> >> >> > windows and linux UI APIs.  Hadoop faces no such complex
> > >> >> >> >> > challenge
> > >> >> >> >> > and hence
> > >> >> >> >> > can be maintained from a single codeline IMO.  It is mostly
> > >> >> >> >> > abstracted from
> > >> >> >> >> > the OS APIs via Java and our design choices.  Where it is
> not
> > >> >> >> >> > we
> > >> >> >> >> > can
> > >> >> >> >> > continue to add plugable abstractions.
> > >> >> >> >> >
> > >> >> >> >
> > >> >> >> >
> > >> >> >
> > >> >> >
> > >> >
> > >> >
> > >
> > >
> >
>



-- 
http://hortonworks.com/download/

Re: [Vote] Merge branch-trunk-win to trunk

[Suresh Srinivas <su...@hortonworks.com>]

Thank you all for voting and participating in the discussions.

With 11 +1s from committers (more than the required 3 +1s from
active committers per the Hadoop bylaws), 1 +0, 8 +1s from other
contributors, and no -1s the merge vote passes.

I have committed the consolidated patch from branch-trunk-win
to trunk. People who are interested in following up on the progress
related to Jenkins setup for Windows and other work that came up
during the discussion, please watch:
https://issues.apache.org/jira/browse/HADOOP-9359

Regards,
Suresh


On Mon, Mar 4, 2013 at 5:41 PM, Matt Foley <mf...@hortonworks.com> wrote:

> Thanks, gentlemen.  I've opened and taken responsibility for
> https://issues.apache.org/jira/browse/HADOOP-9359.  Giri Kesavan has
> agreed
> to help with the parts that require Jenkins admin access.
>
> Thanks,
> --Matt
>
>
>
> On Mon, Mar 4, 2013 at 5:00 PM, Konstantin Shvachko <shv.hadoop@gmail.com
> >wrote:
>
> > +1 on the merge.
> >
> > I am glad we agreed.
> > Having Jira to track the CI effort is a good idea.
> >
> > Thanks,
> > --Konstantin
> >
> > On Mon, Mar 4, 2013 at 3:29 PM, Matt Foley <mf...@hortonworks.com>
> wrote:
> > > Thanks.  I agree Windows -1's in test-patch should not block commits.
> > >
> > > --Matt
> > >
> > >
> > >
> > > On Mon, Mar 4, 2013 at 2:30 PM, Konstantin Shvachko <
> > shv.hadoop@gmail.com>
> > > wrote:
> > >>
> > >> On Mon, Mar 4, 2013 at 12:22 PM, Matt Foley <mf...@hortonworks.com>
> > >> wrote:
> > >> > Konstantine, you have voted -1, and stated some requirements before
> > >> > you'll
> > >> > withdraw that -1.  As I plan to do work to fulfill those
> > requirements, I
> > >> > want to make sure that what I'm proposing will, in fact, satisfy
> you.
> > >> > That's why I'm asking, if we implement full "test-patch" integration
> > for
> > >> > Windows, does it seem to you that that would provide adequate
> support?
> > >>
> > >> Yes.
> > >>
> > >> > I have learned not to presume that my interpretation is correct.  My
> > >> > interpretation of item #1 is that test-patch provides pre-commit
> > build,
> > >> > so
> > >> > it would satisfy item #1.  But rather than assuming that I am
> > >> > interpreting
> > >> > it correctly, I simply want your agreement that it would, or if not,
> > >> > clarification why it won't.
> > >>
> > >> I agree it will satisfy my item #1.
> > >> I did not agree in my previous email, but I changed my mind based on
> > >> the latest discussion. I have to explain why now.
> > >> I was proposing nightly build because I did not want pre-commit build
> > >> for Windows block commits to Linux. But if people are fine just
> ignoring
> > >> -1s for the Windows part of the build it should be good.
> > >>
> > >> > Regarding item #2, it is also my interpretation that test-patch
> > provides
> > >> > an
> > >> > on-demand (perhaps 20-minutes deferred) Jenkins build and unit test,
> > >> > with
> > >> > logs available to the developer, so it would satisfy item #2.  But
> > >> > rather
> > >> > than assuming that I am interpreting it correctly, I simply want
> your
> > >> > agreement that it would, or if not, clarification why it won't.
> > >>
> > >> It will satisfy my item #2 in the following way:
> > >> I can duplicate your pre-commit build for Windows and add an input
> > >> parameter, which would let people run the build on their patches
> > >> chosen from local machine rather than attaching them to Jiras.
> > >>
> > >> Thanks,
> > >> --Konstantin
> > >>
> > >> > In agile terms, you are the Owner of these requirements.  Please
> give
> > me
> > >> > owner feedback as to whether my proposed work sounds like it will
> > >> > satisfy
> > >> > the requirements.
> > >> >
> > >> > Thank you,
> > >> > --Matt
> > >> >
> > >> >
> > >> > On Sun, Mar 3, 2013 at 12:16 PM, Konstantin Shvachko
> > >> > <sh...@gmail.com>
> > >> > wrote:
> > >> >>
> > >> >> Didn't I explain in details what I am asking for?
> > >> >>
> > >> >> Thanks,
> > >> >> --Konst
> > >> >>
> > >> >> On Sun, Mar 3, 2013 at 11:08 AM, Matt Foley <
> mfoley@hortonworks.com>
> > >> >> wrote:
> > >> >> > Hi Konstantin,
> > >> >> > I'd like to point out two things:
> > >> >> > First, I already committed in this thread (email of Thu, Feb 28,
> > 2013
> > >> >> > at
> > >> >> > 6:01 PM) to providing CI for Windows builds.  So please stop
> acting
> > >> >> > like
> > >> >> > I'm
> > >> >> > resisting this idea or something.
> > >> >> > Second, you didn't answer my question, you just kvetched about
> the
> > >> >> > phrasing.
> > >> >> > So I ask again:
> > >> >> >
> > >> >> > Will providing full "test-patch" integration (pre-commit build
> and
> > >> >> > unit
> > >> >> > test
> > >> >> > triggered by Jira "Patch Available" state) satisfy your request
> for
> > >> >> > functionality #1 and #2?  Yes or no, please.
> > >> >> >
> > >> >> > Thanks,
> > >> >> > --Matt
> > >> >> >
> > >> >> >
> > >> >> > On Sat, Mar 2, 2013 at 7:32 PM, Konstantin Shvachko
> > >> >> > <sh...@gmail.com>
> > >> >> > wrote:
> > >> >> >>
> > >> >> >> Hi Matt,
> > >> >> >>
> > >> >> >> On Sat, Mar 2, 2013 at 12:32 PM, Matt Foley <
> > mfoley@hortonworks.com>
> > >> >> >> wrote:
> > >> >> >> > Konstantin,
> > >> >> >> > I would like to explore what it would take to remove this
> > >> >> >> > perceived
> > >> >> >> > impediment --
> > >> >> >>
> > >> >> >> Glad you decided to explore. Thank you.
> > >> >> >>
> > >> >> >> > although I reserve the right to argue that this is not
> > >> >> >> > pre-requisite to merging the cross-platform support patch.
> > >> >> >>
> > >> >> >> It's your right indeed. So as mine to question what the platform
> > >> >> >> support means for you, which I believe remained unclear.
> > >> >> >> I do not impede the change as you should have noticed. My
> > >> >> >> requirement
> > >> >> >> comes from my perception of the support, which means to me
> exactly
> > >> >> >> two
> > >> >> >> things:
> > >> >> >> 1. The ability to recognise the code is broken for the platform
> > >> >> >> 2. The ability to test new patches on the platform
> > >> >> >> The latter is problematic, as many noticed in this thread, for
> > those
> > >> >> >> whose customary environment does not include Windows.
> > >> >> >>
> > >> >> >> > If we implemented full "test-patch" support for Windows on
> > trunk,
> > >> >> >> > would
> > >> >> >> > that
> > >> >> >> > fulfill both your items #1 and #2?  Please note that:
> > >> >> >> > a) Pushing the "Patch Available" button in Jira shall cause a
> > >> >> >> > pre-commit
> > >> >> >> > build to start within, I believe, 20 minutes.
> > >> >> >> > b) That build keeps logs for both java build and unit tests
> for
> > >> >> >> > several
> > >> >> >> > days, that are accessible to all viewers.
> > >> >> >>
> > >> >> >> In item #1 I mostly asking for the nightly build, which is
> simpler
> > >> >> >> than "test-patch". The latter would be ideal from the platform
> > >> >> >> support
> > >> >> >> viewpoint, but it is for the community to decide if we want to
> add
> > >> >> >> extra +3 hours to the build.
> > >> >> >> Nightly build in my understanding is triggered by the timer
> rather
> > >> >> >> than by Jira's "submit patch" button.  On Jenkins build
> > >> >> >> configuration
> > >> >> >> you can specify it under "Build periodically".
> > >> >> >>
> > >> >> >> > So, does this provide sufficient on-demand support that we
> don't
> > >> >> >> > have
> > >> >> >> > to
> > >> >> >> > implement a whole new on-demand VM support structure of some
> > sort
> > >> >> >> > for
> > >> >> >> > #2
> > >> >> >> > (which would be an extraordinary and impractical demand)?
> > >> >> >>
> > >> >> >> I did not mention VMs. Item #2 means a build, which runs
> > >> >> >> "test-patch"
> > >> >> >> target with the file specified by a user (instead of a jira
> > >> >> >> attachment).
> > >> >> >> When user clicks "Build Now" link a box is displayed where the
> > user
> > >> >> >> can enter the file path containing the patch. This can be
> > specified
> > >> >> >> in
> > >> >> >> the Build Configuration under "This build is parameterized" by
> > >> >> >> choosing AddParameter / FileParameter. The build can run on the
> > same
> > >> >> >> Windows machine as the nightly build.
> > >> >> >> Such build will let people test their patches for Windows on
> > Jenkins
> > >> >> >> if they don't posses a license for the right version of Windows.
> > >> >> >> I hope this will not turn into extraordinary or impractical
> > effort.
> > >> >> >>
> > >> >> >> Thanks,
> > >> >> >> --Konst
> > >> >> >>
> > >> >> >> > Thanks,
> > >> >> >> > --Matt
> > >> >> >> >
> > >> >> >> >
> > >> >> >> > On Fri, Mar 1, 2013 at 12:18 PM, Konstantin Shvachko
> > >> >> >> > <sh...@gmail.com>
> > >> >> >> > wrote:
> > >> >> >> >>
> > >> >> >> >> -1
> > >> >> >> >> We should have a CI infrastructure in place before we can
> > commit
> > >> >> >> >> to
> > >> >> >> >> supporting Windows platform.
> > >> >> >> >>
> > >> >> >> >> Eric is right Win/Cygwin was supported since day one.
> > >> >> >> >> I had a Windows box under my desk running nightly builds back
> > in
> > >> >> >> >> 2006-07.
> > >> >> >> >> People were irritated but I was filing windows bugs until
> 0.22
> > >> >> >> >> release.
> > >> >> >> >> Times changing and I am glad to see wider support for Win
> > >> >> >> >> platform.
> > >> >> >> >>
> > >> >> >> >> But in order to make it work you guys need to put the CI
> > process
> > >> >> >> >> in
> > >> >> >> >> place
> > >> >> >> >>
> > >> >> >> >> 1. windows jenkins build: could be nightly or PreCommit.
> > >> >> >> >> - Nightly would mean that changes can be committed to trunk
> > based
> > >> >> >> >> on
> > >> >> >> >> linux PreCommit build. And people will file bugs if the
> change
> > >> >> >> >> broke
> > >> >> >> >> Windows nightly build.
> > >> >> >> >> - PreCommit-win build will mean automatic reporting failed
> > tests
> > >> >> >> >> to
> > >> >> >> >> respective jira blocking commits the same way as it is now
> with
> > >> >> >> >> linux
> > >> >> >> >> PreCommit builds.
> > >> >> >> >> We should discuss which way is more efficient for developers.
> > >> >> >> >>
> > >> >> >> >> 2. On-demand-windows Jenkins build.
> > >> >> >> >> I see it as a build to which I can attach my patch and the
> > build
> > >> >> >> >> will
> > >> >> >> >> run my changes on a dedicated windows box.
> > >> >> >> >> That way people can test their changes without having
> personal
> > >> >> >> >> windows
> > >> >> >> >> nodes.
> > >> >> >> >>
> > >> >> >> >> I think this is the minimal set of requirement for us to be
> > able
> > >> >> >> >> to
> > >> >> >> >> commit to the new platform.
> > >> >> >> >> Right now I see only one windows related build
> > >> >> >> >> https://builds.apache.org/view/Hadoop/job/Hadoop-1-win/
> > >> >> >> >> Which was failing since Sept 8, 2012 and did not run in the
> > last
> > >> >> >> >> month.
> > >> >> >> >>
> > >> >> >> >> Thanks,
> > >> >> >> >> --Konst
> > >> >> >> >>
> > >> >> >> >> On Thu, Feb 28, 2013 at 8:47 PM, Eric Baldeschwieler
> > >> >> >> >> <er...@hortonworks.com> wrote:
> > >> >> >> >> > +1 (non-binding)
> > >> >> >> >> >
> > >> >> >> >> > A few of observations:
> > >> >> >> >> >
> > >> >> >> >> > - Windows has actually been a supported platform for Hadoop
> > >> >> >> >> > since
> > >> >> >> >> > 0.1
> > >> >> >> >> > .
> > >> >> >> >> > Doug championed supporting windows then and we've continued
> > to
> > >> >> >> >> > do
> > >> >> >> >> > it
> > >> >> >> >> > with
> > >> >> >> >> > varying vigor over time.  To my knowledge we've never made
> a
> > >> >> >> >> > decision
> > >> >> >> >> > to
> > >> >> >> >> > drop windows support.  The change here is improving our
> > support
> > >> >> >> >> > and
> > >> >> >> >> > dropping
> > >> >> >> >> > the requirement of cigwin.  We had Nutch windows users on
> the
> > >> >> >> >> > list
> > >> >> >> >> > in
> > >> >> >> >> > 2006
> > >> >> >> >> > and we've been supporting windows FS requirements since
> > >> >> >> >> > inception.
> > >> >> >> >> >
> > >> >> >> >> > - A little pragmatism will go a long way.  As a community
> > we've
> > >> >> >> >> > got
> > >> >> >> >> > to
> > >> >> >> >> > stay committed to keeping hadoop simple (so it does work on
> > >> >> >> >> > many
> > >> >> >> >> > platforms)
> > >> >> >> >> > and extending it to take advantage of key emerging
> > OS/hardware
> > >> >> >> >> > features,
> > >> >> >> >> > such as containers, new FSs, virtualization, flash ...  We
> > >> >> >> >> > should
> > >> >> >> >> > all
> > >> >> >> >> > plan
> > >> >> >> >> > to let new features & optimizations emerge that don't work
> > >> >> >> >> > everywhere, if
> > >> >> >> >> > they are compelling and central to hadoop's mission of
> being
> > >> >> >> >> > THE
> > >> >> >> >> > best
> > >> >> >> >> > fabric
> > >> >> >> >> > for storing and processing big data.
> > >> >> >> >> >
> > >> >> >> >> > - A UI project like KDE has to deal with the MANY
> differences
> > >> >> >> >> > between
> > >> >> >> >> > windows and linux UI APIs.  Hadoop faces no such complex
> > >> >> >> >> > challenge
> > >> >> >> >> > and hence
> > >> >> >> >> > can be maintained from a single codeline IMO.  It is mostly
> > >> >> >> >> > abstracted from
> > >> >> >> >> > the OS APIs via Java and our design choices.  Where it is
> not
> > >> >> >> >> > we
> > >> >> >> >> > can
> > >> >> >> >> > continue to add plugable abstractions.
> > >> >> >> >> >
> > >> >> >> >
> > >> >> >> >
> > >> >> >
> > >> >> >
> > >> >
> > >> >
> > >
> > >
> >
>



-- 
http://hortonworks.com/download/

Re: [Vote] Merge branch-trunk-win to trunk

[Suresh Srinivas <su...@hortonworks.com>]

Thank you all for voting and participating in the discussions.

With 11 +1s from committers (more than the required 3 +1s from
active committers per the Hadoop bylaws), 1 +0, 8 +1s from other
contributors, and no -1s the merge vote passes.

I have committed the consolidated patch from branch-trunk-win
to trunk. People who are interested in following up on the progress
related to Jenkins setup for Windows and other work that came up
during the discussion, please watch:
https://issues.apache.org/jira/browse/HADOOP-9359

Regards,
Suresh


On Mon, Mar 4, 2013 at 5:41 PM, Matt Foley <mf...@hortonworks.com> wrote:

> Thanks, gentlemen.  I've opened and taken responsibility for
> https://issues.apache.org/jira/browse/HADOOP-9359.  Giri Kesavan has
> agreed
> to help with the parts that require Jenkins admin access.
>
> Thanks,
> --Matt
>
>
>
> On Mon, Mar 4, 2013 at 5:00 PM, Konstantin Shvachko <shv.hadoop@gmail.com
> >wrote:
>
> > +1 on the merge.
> >
> > I am glad we agreed.
> > Having Jira to track the CI effort is a good idea.
> >
> > Thanks,
> > --Konstantin
> >
> > On Mon, Mar 4, 2013 at 3:29 PM, Matt Foley <mf...@hortonworks.com>
> wrote:
> > > Thanks.  I agree Windows -1's in test-patch should not block commits.
> > >
> > > --Matt
> > >
> > >
> > >
> > > On Mon, Mar 4, 2013 at 2:30 PM, Konstantin Shvachko <
> > shv.hadoop@gmail.com>
> > > wrote:
> > >>
> > >> On Mon, Mar 4, 2013 at 12:22 PM, Matt Foley <mf...@hortonworks.com>
> > >> wrote:
> > >> > Konstantine, you have voted -1, and stated some requirements before
> > >> > you'll
> > >> > withdraw that -1.  As I plan to do work to fulfill those
> > requirements, I
> > >> > want to make sure that what I'm proposing will, in fact, satisfy
> you.
> > >> > That's why I'm asking, if we implement full "test-patch" integration
> > for
> > >> > Windows, does it seem to you that that would provide adequate
> support?
> > >>
> > >> Yes.
> > >>
> > >> > I have learned not to presume that my interpretation is correct.  My
> > >> > interpretation of item #1 is that test-patch provides pre-commit
> > build,
> > >> > so
> > >> > it would satisfy item #1.  But rather than assuming that I am
> > >> > interpreting
> > >> > it correctly, I simply want your agreement that it would, or if not,
> > >> > clarification why it won't.
> > >>
> > >> I agree it will satisfy my item #1.
> > >> I did not agree in my previous email, but I changed my mind based on
> > >> the latest discussion. I have to explain why now.
> > >> I was proposing nightly build because I did not want pre-commit build
> > >> for Windows block commits to Linux. But if people are fine just
> ignoring
> > >> -1s for the Windows part of the build it should be good.
> > >>
> > >> > Regarding item #2, it is also my interpretation that test-patch
> > provides
> > >> > an
> > >> > on-demand (perhaps 20-minutes deferred) Jenkins build and unit test,
> > >> > with
> > >> > logs available to the developer, so it would satisfy item #2.  But
> > >> > rather
> > >> > than assuming that I am interpreting it correctly, I simply want
> your
> > >> > agreement that it would, or if not, clarification why it won't.
> > >>
> > >> It will satisfy my item #2 in the following way:
> > >> I can duplicate your pre-commit build for Windows and add an input
> > >> parameter, which would let people run the build on their patches
> > >> chosen from local machine rather than attaching them to Jiras.
> > >>
> > >> Thanks,
> > >> --Konstantin
> > >>
> > >> > In agile terms, you are the Owner of these requirements.  Please
> give
> > me
> > >> > owner feedback as to whether my proposed work sounds like it will
> > >> > satisfy
> > >> > the requirements.
> > >> >
> > >> > Thank you,
> > >> > --Matt
> > >> >
> > >> >
> > >> > On Sun, Mar 3, 2013 at 12:16 PM, Konstantin Shvachko
> > >> > <sh...@gmail.com>
> > >> > wrote:
> > >> >>
> > >> >> Didn't I explain in details what I am asking for?
> > >> >>
> > >> >> Thanks,
> > >> >> --Konst
> > >> >>
> > >> >> On Sun, Mar 3, 2013 at 11:08 AM, Matt Foley <
> mfoley@hortonworks.com>
> > >> >> wrote:
> > >> >> > Hi Konstantin,
> > >> >> > I'd like to point out two things:
> > >> >> > First, I already committed in this thread (email of Thu, Feb 28,
> > 2013
> > >> >> > at
> > >> >> > 6:01 PM) to providing CI for Windows builds.  So please stop
> acting
> > >> >> > like
> > >> >> > I'm
> > >> >> > resisting this idea or something.
> > >> >> > Second, you didn't answer my question, you just kvetched about
> the
> > >> >> > phrasing.
> > >> >> > So I ask again:
> > >> >> >
> > >> >> > Will providing full "test-patch" integration (pre-commit build
> and
> > >> >> > unit
> > >> >> > test
> > >> >> > triggered by Jira "Patch Available" state) satisfy your request
> for
> > >> >> > functionality #1 and #2?  Yes or no, please.
> > >> >> >
> > >> >> > Thanks,
> > >> >> > --Matt
> > >> >> >
> > >> >> >
> > >> >> > On Sat, Mar 2, 2013 at 7:32 PM, Konstantin Shvachko
> > >> >> > <sh...@gmail.com>
> > >> >> > wrote:
> > >> >> >>
> > >> >> >> Hi Matt,
> > >> >> >>
> > >> >> >> On Sat, Mar 2, 2013 at 12:32 PM, Matt Foley <
> > mfoley@hortonworks.com>
> > >> >> >> wrote:
> > >> >> >> > Konstantin,
> > >> >> >> > I would like to explore what it would take to remove this
> > >> >> >> > perceived
> > >> >> >> > impediment --
> > >> >> >>
> > >> >> >> Glad you decided to explore. Thank you.
> > >> >> >>
> > >> >> >> > although I reserve the right to argue that this is not
> > >> >> >> > pre-requisite to merging the cross-platform support patch.
> > >> >> >>
> > >> >> >> It's your right indeed. So as mine to question what the platform
> > >> >> >> support means for you, which I believe remained unclear.
> > >> >> >> I do not impede the change as you should have noticed. My
> > >> >> >> requirement
> > >> >> >> comes from my perception of the support, which means to me
> exactly
> > >> >> >> two
> > >> >> >> things:
> > >> >> >> 1. The ability to recognise the code is broken for the platform
> > >> >> >> 2. The ability to test new patches on the platform
> > >> >> >> The latter is problematic, as many noticed in this thread, for
> > those
> > >> >> >> whose customary environment does not include Windows.
> > >> >> >>
> > >> >> >> > If we implemented full "test-patch" support for Windows on
> > trunk,
> > >> >> >> > would
> > >> >> >> > that
> > >> >> >> > fulfill both your items #1 and #2?  Please note that:
> > >> >> >> > a) Pushing the "Patch Available" button in Jira shall cause a
> > >> >> >> > pre-commit
> > >> >> >> > build to start within, I believe, 20 minutes.
> > >> >> >> > b) That build keeps logs for both java build and unit tests
> for
> > >> >> >> > several
> > >> >> >> > days, that are accessible to all viewers.
> > >> >> >>
> > >> >> >> In item #1 I mostly asking for the nightly build, which is
> simpler
> > >> >> >> than "test-patch". The latter would be ideal from the platform
> > >> >> >> support
> > >> >> >> viewpoint, but it is for the community to decide if we want to
> add
> > >> >> >> extra +3 hours to the build.
> > >> >> >> Nightly build in my understanding is triggered by the timer
> rather
> > >> >> >> than by Jira's "submit patch" button.  On Jenkins build
> > >> >> >> configuration
> > >> >> >> you can specify it under "Build periodically".
> > >> >> >>
> > >> >> >> > So, does this provide sufficient on-demand support that we
> don't
> > >> >> >> > have
> > >> >> >> > to
> > >> >> >> > implement a whole new on-demand VM support structure of some
> > sort
> > >> >> >> > for
> > >> >> >> > #2
> > >> >> >> > (which would be an extraordinary and impractical demand)?
> > >> >> >>
> > >> >> >> I did not mention VMs. Item #2 means a build, which runs
> > >> >> >> "test-patch"
> > >> >> >> target with the file specified by a user (instead of a jira
> > >> >> >> attachment).
> > >> >> >> When user clicks "Build Now" link a box is displayed where the
> > user
> > >> >> >> can enter the file path containing the patch. This can be
> > specified
> > >> >> >> in
> > >> >> >> the Build Configuration under "This build is parameterized" by
> > >> >> >> choosing AddParameter / FileParameter. The build can run on the
> > same
> > >> >> >> Windows machine as the nightly build.
> > >> >> >> Such build will let people test their patches for Windows on
> > Jenkins
> > >> >> >> if they don't posses a license for the right version of Windows.
> > >> >> >> I hope this will not turn into extraordinary or impractical
> > effort.
> > >> >> >>
> > >> >> >> Thanks,
> > >> >> >> --Konst
> > >> >> >>
> > >> >> >> > Thanks,
> > >> >> >> > --Matt
> > >> >> >> >
> > >> >> >> >
> > >> >> >> > On Fri, Mar 1, 2013 at 12:18 PM, Konstantin Shvachko
> > >> >> >> > <sh...@gmail.com>
> > >> >> >> > wrote:
> > >> >> >> >>
> > >> >> >> >> -1
> > >> >> >> >> We should have a CI infrastructure in place before we can
> > commit
> > >> >> >> >> to
> > >> >> >> >> supporting Windows platform.
> > >> >> >> >>
> > >> >> >> >> Eric is right Win/Cygwin was supported since day one.
> > >> >> >> >> I had a Windows box under my desk running nightly builds back
> > in
> > >> >> >> >> 2006-07.
> > >> >> >> >> People were irritated but I was filing windows bugs until
> 0.22
> > >> >> >> >> release.
> > >> >> >> >> Times changing and I am glad to see wider support for Win
> > >> >> >> >> platform.
> > >> >> >> >>
> > >> >> >> >> But in order to make it work you guys need to put the CI
> > process
> > >> >> >> >> in
> > >> >> >> >> place
> > >> >> >> >>
> > >> >> >> >> 1. windows jenkins build: could be nightly or PreCommit.
> > >> >> >> >> - Nightly would mean that changes can be committed to trunk
> > based
> > >> >> >> >> on
> > >> >> >> >> linux PreCommit build. And people will file bugs if the
> change
> > >> >> >> >> broke
> > >> >> >> >> Windows nightly build.
> > >> >> >> >> - PreCommit-win build will mean automatic reporting failed
> > tests
> > >> >> >> >> to
> > >> >> >> >> respective jira blocking commits the same way as it is now
> with
> > >> >> >> >> linux
> > >> >> >> >> PreCommit builds.
> > >> >> >> >> We should discuss which way is more efficient for developers.
> > >> >> >> >>
> > >> >> >> >> 2. On-demand-windows Jenkins build.
> > >> >> >> >> I see it as a build to which I can attach my patch and the
> > build
> > >> >> >> >> will
> > >> >> >> >> run my changes on a dedicated windows box.
> > >> >> >> >> That way people can test their changes without having
> personal
> > >> >> >> >> windows
> > >> >> >> >> nodes.
> > >> >> >> >>
> > >> >> >> >> I think this is the minimal set of requirement for us to be
> > able
> > >> >> >> >> to
> > >> >> >> >> commit to the new platform.
> > >> >> >> >> Right now I see only one windows related build
> > >> >> >> >> https://builds.apache.org/view/Hadoop/job/Hadoop-1-win/
> > >> >> >> >> Which was failing since Sept 8, 2012 and did not run in the
> > last
> > >> >> >> >> month.
> > >> >> >> >>
> > >> >> >> >> Thanks,
> > >> >> >> >> --Konst
> > >> >> >> >>
> > >> >> >> >> On Thu, Feb 28, 2013 at 8:47 PM, Eric Baldeschwieler
> > >> >> >> >> <er...@hortonworks.com> wrote:
> > >> >> >> >> > +1 (non-binding)
> > >> >> >> >> >
> > >> >> >> >> > A few of observations:
> > >> >> >> >> >
> > >> >> >> >> > - Windows has actually been a supported platform for Hadoop
> > >> >> >> >> > since
> > >> >> >> >> > 0.1
> > >> >> >> >> > .
> > >> >> >> >> > Doug championed supporting windows then and we've continued
> > to
> > >> >> >> >> > do
> > >> >> >> >> > it
> > >> >> >> >> > with
> > >> >> >> >> > varying vigor over time.  To my knowledge we've never made
> a
> > >> >> >> >> > decision
> > >> >> >> >> > to
> > >> >> >> >> > drop windows support.  The change here is improving our
> > support
> > >> >> >> >> > and
> > >> >> >> >> > dropping
> > >> >> >> >> > the requirement of cigwin.  We had Nutch windows users on
> the
> > >> >> >> >> > list
> > >> >> >> >> > in
> > >> >> >> >> > 2006
> > >> >> >> >> > and we've been supporting windows FS requirements since
> > >> >> >> >> > inception.
> > >> >> >> >> >
> > >> >> >> >> > - A little pragmatism will go a long way.  As a community
> > we've
> > >> >> >> >> > got
> > >> >> >> >> > to
> > >> >> >> >> > stay committed to keeping hadoop simple (so it does work on
> > >> >> >> >> > many
> > >> >> >> >> > platforms)
> > >> >> >> >> > and extending it to take advantage of key emerging
> > OS/hardware
> > >> >> >> >> > features,
> > >> >> >> >> > such as containers, new FSs, virtualization, flash ...  We
> > >> >> >> >> > should
> > >> >> >> >> > all
> > >> >> >> >> > plan
> > >> >> >> >> > to let new features & optimizations emerge that don't work
> > >> >> >> >> > everywhere, if
> > >> >> >> >> > they are compelling and central to hadoop's mission of
> being
> > >> >> >> >> > THE
> > >> >> >> >> > best
> > >> >> >> >> > fabric
> > >> >> >> >> > for storing and processing big data.
> > >> >> >> >> >
> > >> >> >> >> > - A UI project like KDE has to deal with the MANY
> differences
> > >> >> >> >> > between
> > >> >> >> >> > windows and linux UI APIs.  Hadoop faces no such complex
> > >> >> >> >> > challenge
> > >> >> >> >> > and hence
> > >> >> >> >> > can be maintained from a single codeline IMO.  It is mostly
> > >> >> >> >> > abstracted from
> > >> >> >> >> > the OS APIs via Java and our design choices.  Where it is
> not
> > >> >> >> >> > we
> > >> >> >> >> > can
> > >> >> >> >> > continue to add plugable abstractions.
> > >> >> >> >> >
> > >> >> >> >
> > >> >> >> >
> > >> >> >
> > >> >> >
> > >> >
> > >> >
> > >
> > >
> >
>



-- 
http://hortonworks.com/download/

Re: [Vote] Merge branch-trunk-win to trunk

[Matt Foley <mf...@hortonworks.com>]

Thanks, gentlemen.  I've opened and taken responsibility for
https://issues.apache.org/jira/browse/HADOOP-9359.  Giri Kesavan has agreed
to help with the parts that require Jenkins admin access.

Thanks,
--Matt



On Mon, Mar 4, 2013 at 5:00 PM, Konstantin Shvachko <sh...@gmail.com>wrote:

> +1 on the merge.
>
> I am glad we agreed.
> Having Jira to track the CI effort is a good idea.
>
> Thanks,
> --Konstantin
>
> On Mon, Mar 4, 2013 at 3:29 PM, Matt Foley <mf...@hortonworks.com> wrote:
> > Thanks.  I agree Windows -1's in test-patch should not block commits.
> >
> > --Matt
> >
> >
> >
> > On Mon, Mar 4, 2013 at 2:30 PM, Konstantin Shvachko <
> shv.hadoop@gmail.com>
> > wrote:
> >>
> >> On Mon, Mar 4, 2013 at 12:22 PM, Matt Foley <mf...@hortonworks.com>
> >> wrote:
> >> > Konstantine, you have voted -1, and stated some requirements before
> >> > you'll
> >> > withdraw that -1.  As I plan to do work to fulfill those
> requirements, I
> >> > want to make sure that what I'm proposing will, in fact, satisfy you.
> >> > That's why I'm asking, if we implement full "test-patch" integration
> for
> >> > Windows, does it seem to you that that would provide adequate support?
> >>
> >> Yes.
> >>
> >> > I have learned not to presume that my interpretation is correct.  My
> >> > interpretation of item #1 is that test-patch provides pre-commit
> build,
> >> > so
> >> > it would satisfy item #1.  But rather than assuming that I am
> >> > interpreting
> >> > it correctly, I simply want your agreement that it would, or if not,
> >> > clarification why it won't.
> >>
> >> I agree it will satisfy my item #1.
> >> I did not agree in my previous email, but I changed my mind based on
> >> the latest discussion. I have to explain why now.
> >> I was proposing nightly build because I did not want pre-commit build
> >> for Windows block commits to Linux. But if people are fine just ignoring
> >> -1s for the Windows part of the build it should be good.
> >>
> >> > Regarding item #2, it is also my interpretation that test-patch
> provides
> >> > an
> >> > on-demand (perhaps 20-minutes deferred) Jenkins build and unit test,
> >> > with
> >> > logs available to the developer, so it would satisfy item #2.  But
> >> > rather
> >> > than assuming that I am interpreting it correctly, I simply want your
> >> > agreement that it would, or if not, clarification why it won't.
> >>
> >> It will satisfy my item #2 in the following way:
> >> I can duplicate your pre-commit build for Windows and add an input
> >> parameter, which would let people run the build on their patches
> >> chosen from local machine rather than attaching them to Jiras.
> >>
> >> Thanks,
> >> --Konstantin
> >>
> >> > In agile terms, you are the Owner of these requirements.  Please give
> me
> >> > owner feedback as to whether my proposed work sounds like it will
> >> > satisfy
> >> > the requirements.
> >> >
> >> > Thank you,
> >> > --Matt
> >> >
> >> >
> >> > On Sun, Mar 3, 2013 at 12:16 PM, Konstantin Shvachko
> >> > <sh...@gmail.com>
> >> > wrote:
> >> >>
> >> >> Didn't I explain in details what I am asking for?
> >> >>
> >> >> Thanks,
> >> >> --Konst
> >> >>
> >> >> On Sun, Mar 3, 2013 at 11:08 AM, Matt Foley <mf...@hortonworks.com>
> >> >> wrote:
> >> >> > Hi Konstantin,
> >> >> > I'd like to point out two things:
> >> >> > First, I already committed in this thread (email of Thu, Feb 28,
> 2013
> >> >> > at
> >> >> > 6:01 PM) to providing CI for Windows builds.  So please stop acting
> >> >> > like
> >> >> > I'm
> >> >> > resisting this idea or something.
> >> >> > Second, you didn't answer my question, you just kvetched about the
> >> >> > phrasing.
> >> >> > So I ask again:
> >> >> >
> >> >> > Will providing full "test-patch" integration (pre-commit build and
> >> >> > unit
> >> >> > test
> >> >> > triggered by Jira "Patch Available" state) satisfy your request for
> >> >> > functionality #1 and #2?  Yes or no, please.
> >> >> >
> >> >> > Thanks,
> >> >> > --Matt
> >> >> >
> >> >> >
> >> >> > On Sat, Mar 2, 2013 at 7:32 PM, Konstantin Shvachko
> >> >> > <sh...@gmail.com>
> >> >> > wrote:
> >> >> >>
> >> >> >> Hi Matt,
> >> >> >>
> >> >> >> On Sat, Mar 2, 2013 at 12:32 PM, Matt Foley <
> mfoley@hortonworks.com>
> >> >> >> wrote:
> >> >> >> > Konstantin,
> >> >> >> > I would like to explore what it would take to remove this
> >> >> >> > perceived
> >> >> >> > impediment --
> >> >> >>
> >> >> >> Glad you decided to explore. Thank you.
> >> >> >>
> >> >> >> > although I reserve the right to argue that this is not
> >> >> >> > pre-requisite to merging the cross-platform support patch.
> >> >> >>
> >> >> >> It's your right indeed. So as mine to question what the platform
> >> >> >> support means for you, which I believe remained unclear.
> >> >> >> I do not impede the change as you should have noticed. My
> >> >> >> requirement
> >> >> >> comes from my perception of the support, which means to me exactly
> >> >> >> two
> >> >> >> things:
> >> >> >> 1. The ability to recognise the code is broken for the platform
> >> >> >> 2. The ability to test new patches on the platform
> >> >> >> The latter is problematic, as many noticed in this thread, for
> those
> >> >> >> whose customary environment does not include Windows.
> >> >> >>
> >> >> >> > If we implemented full "test-patch" support for Windows on
> trunk,
> >> >> >> > would
> >> >> >> > that
> >> >> >> > fulfill both your items #1 and #2?  Please note that:
> >> >> >> > a) Pushing the "Patch Available" button in Jira shall cause a
> >> >> >> > pre-commit
> >> >> >> > build to start within, I believe, 20 minutes.
> >> >> >> > b) That build keeps logs for both java build and unit tests for
> >> >> >> > several
> >> >> >> > days, that are accessible to all viewers.
> >> >> >>
> >> >> >> In item #1 I mostly asking for the nightly build, which is simpler
> >> >> >> than "test-patch". The latter would be ideal from the platform
> >> >> >> support
> >> >> >> viewpoint, but it is for the community to decide if we want to add
> >> >> >> extra +3 hours to the build.
> >> >> >> Nightly build in my understanding is triggered by the timer rather
> >> >> >> than by Jira's "submit patch" button.  On Jenkins build
> >> >> >> configuration
> >> >> >> you can specify it under "Build periodically".
> >> >> >>
> >> >> >> > So, does this provide sufficient on-demand support that we don't
> >> >> >> > have
> >> >> >> > to
> >> >> >> > implement a whole new on-demand VM support structure of some
> sort
> >> >> >> > for
> >> >> >> > #2
> >> >> >> > (which would be an extraordinary and impractical demand)?
> >> >> >>
> >> >> >> I did not mention VMs. Item #2 means a build, which runs
> >> >> >> "test-patch"
> >> >> >> target with the file specified by a user (instead of a jira
> >> >> >> attachment).
> >> >> >> When user clicks "Build Now" link a box is displayed where the
> user
> >> >> >> can enter the file path containing the patch. This can be
> specified
> >> >> >> in
> >> >> >> the Build Configuration under "This build is parameterized" by
> >> >> >> choosing AddParameter / FileParameter. The build can run on the
> same
> >> >> >> Windows machine as the nightly build.
> >> >> >> Such build will let people test their patches for Windows on
> Jenkins
> >> >> >> if they don't posses a license for the right version of Windows.
> >> >> >> I hope this will not turn into extraordinary or impractical
> effort.
> >> >> >>
> >> >> >> Thanks,
> >> >> >> --Konst
> >> >> >>
> >> >> >> > Thanks,
> >> >> >> > --Matt
> >> >> >> >
> >> >> >> >
> >> >> >> > On Fri, Mar 1, 2013 at 12:18 PM, Konstantin Shvachko
> >> >> >> > <sh...@gmail.com>
> >> >> >> > wrote:
> >> >> >> >>
> >> >> >> >> -1
> >> >> >> >> We should have a CI infrastructure in place before we can
> commit
> >> >> >> >> to
> >> >> >> >> supporting Windows platform.
> >> >> >> >>
> >> >> >> >> Eric is right Win/Cygwin was supported since day one.
> >> >> >> >> I had a Windows box under my desk running nightly builds back
> in
> >> >> >> >> 2006-07.
> >> >> >> >> People were irritated but I was filing windows bugs until 0.22
> >> >> >> >> release.
> >> >> >> >> Times changing and I am glad to see wider support for Win
> >> >> >> >> platform.
> >> >> >> >>
> >> >> >> >> But in order to make it work you guys need to put the CI
> process
> >> >> >> >> in
> >> >> >> >> place
> >> >> >> >>
> >> >> >> >> 1. windows jenkins build: could be nightly or PreCommit.
> >> >> >> >> - Nightly would mean that changes can be committed to trunk
> based
> >> >> >> >> on
> >> >> >> >> linux PreCommit build. And people will file bugs if the change
> >> >> >> >> broke
> >> >> >> >> Windows nightly build.
> >> >> >> >> - PreCommit-win build will mean automatic reporting failed
> tests
> >> >> >> >> to
> >> >> >> >> respective jira blocking commits the same way as it is now with
> >> >> >> >> linux
> >> >> >> >> PreCommit builds.
> >> >> >> >> We should discuss which way is more efficient for developers.
> >> >> >> >>
> >> >> >> >> 2. On-demand-windows Jenkins build.
> >> >> >> >> I see it as a build to which I can attach my patch and the
> build
> >> >> >> >> will
> >> >> >> >> run my changes on a dedicated windows box.
> >> >> >> >> That way people can test their changes without having personal
> >> >> >> >> windows
> >> >> >> >> nodes.
> >> >> >> >>
> >> >> >> >> I think this is the minimal set of requirement for us to be
> able
> >> >> >> >> to
> >> >> >> >> commit to the new platform.
> >> >> >> >> Right now I see only one windows related build
> >> >> >> >> https://builds.apache.org/view/Hadoop/job/Hadoop-1-win/
> >> >> >> >> Which was failing since Sept 8, 2012 and did not run in the
> last
> >> >> >> >> month.
> >> >> >> >>
> >> >> >> >> Thanks,
> >> >> >> >> --Konst
> >> >> >> >>
> >> >> >> >> On Thu, Feb 28, 2013 at 8:47 PM, Eric Baldeschwieler
> >> >> >> >> <er...@hortonworks.com> wrote:
> >> >> >> >> > +1 (non-binding)
> >> >> >> >> >
> >> >> >> >> > A few of observations:
> >> >> >> >> >
> >> >> >> >> > - Windows has actually been a supported platform for Hadoop
> >> >> >> >> > since
> >> >> >> >> > 0.1
> >> >> >> >> > .
> >> >> >> >> > Doug championed supporting windows then and we've continued
> to
> >> >> >> >> > do
> >> >> >> >> > it
> >> >> >> >> > with
> >> >> >> >> > varying vigor over time.  To my knowledge we've never made a
> >> >> >> >> > decision
> >> >> >> >> > to
> >> >> >> >> > drop windows support.  The change here is improving our
> support
> >> >> >> >> > and
> >> >> >> >> > dropping
> >> >> >> >> > the requirement of cigwin.  We had Nutch windows users on the
> >> >> >> >> > list
> >> >> >> >> > in
> >> >> >> >> > 2006
> >> >> >> >> > and we've been supporting windows FS requirements since
> >> >> >> >> > inception.
> >> >> >> >> >
> >> >> >> >> > - A little pragmatism will go a long way.  As a community
> we've
> >> >> >> >> > got
> >> >> >> >> > to
> >> >> >> >> > stay committed to keeping hadoop simple (so it does work on
> >> >> >> >> > many
> >> >> >> >> > platforms)
> >> >> >> >> > and extending it to take advantage of key emerging
> OS/hardware
> >> >> >> >> > features,
> >> >> >> >> > such as containers, new FSs, virtualization, flash ...  We
> >> >> >> >> > should
> >> >> >> >> > all
> >> >> >> >> > plan
> >> >> >> >> > to let new features & optimizations emerge that don't work
> >> >> >> >> > everywhere, if
> >> >> >> >> > they are compelling and central to hadoop's mission of being
> >> >> >> >> > THE
> >> >> >> >> > best
> >> >> >> >> > fabric
> >> >> >> >> > for storing and processing big data.
> >> >> >> >> >
> >> >> >> >> > - A UI project like KDE has to deal with the MANY differences
> >> >> >> >> > between
> >> >> >> >> > windows and linux UI APIs.  Hadoop faces no such complex
> >> >> >> >> > challenge
> >> >> >> >> > and hence
> >> >> >> >> > can be maintained from a single codeline IMO.  It is mostly
> >> >> >> >> > abstracted from
> >> >> >> >> > the OS APIs via Java and our design choices.  Where it is not
> >> >> >> >> > we
> >> >> >> >> > can
> >> >> >> >> > continue to add plugable abstractions.
> >> >> >> >> >
> >> >> >> >
> >> >> >> >
> >> >> >
> >> >> >
> >> >
> >> >
> >
> >
>

Re: [Vote] Merge branch-trunk-win to trunk

[Matt Foley <mf...@hortonworks.com>]

Thanks, gentlemen.  I've opened and taken responsibility for
https://issues.apache.org/jira/browse/HADOOP-9359.  Giri Kesavan has agreed
to help with the parts that require Jenkins admin access.

Thanks,
--Matt



On Mon, Mar 4, 2013 at 5:00 PM, Konstantin Shvachko <sh...@gmail.com>wrote:

> +1 on the merge.
>
> I am glad we agreed.
> Having Jira to track the CI effort is a good idea.
>
> Thanks,
> --Konstantin
>
> On Mon, Mar 4, 2013 at 3:29 PM, Matt Foley <mf...@hortonworks.com> wrote:
> > Thanks.  I agree Windows -1's in test-patch should not block commits.
> >
> > --Matt
> >
> >
> >
> > On Mon, Mar 4, 2013 at 2:30 PM, Konstantin Shvachko <
> shv.hadoop@gmail.com>
> > wrote:
> >>
> >> On Mon, Mar 4, 2013 at 12:22 PM, Matt Foley <mf...@hortonworks.com>
> >> wrote:
> >> > Konstantine, you have voted -1, and stated some requirements before
> >> > you'll
> >> > withdraw that -1.  As I plan to do work to fulfill those
> requirements, I
> >> > want to make sure that what I'm proposing will, in fact, satisfy you.
> >> > That's why I'm asking, if we implement full "test-patch" integration
> for
> >> > Windows, does it seem to you that that would provide adequate support?
> >>
> >> Yes.
> >>
> >> > I have learned not to presume that my interpretation is correct.  My
> >> > interpretation of item #1 is that test-patch provides pre-commit
> build,
> >> > so
> >> > it would satisfy item #1.  But rather than assuming that I am
> >> > interpreting
> >> > it correctly, I simply want your agreement that it would, or if not,
> >> > clarification why it won't.
> >>
> >> I agree it will satisfy my item #1.
> >> I did not agree in my previous email, but I changed my mind based on
> >> the latest discussion. I have to explain why now.
> >> I was proposing nightly build because I did not want pre-commit build
> >> for Windows block commits to Linux. But if people are fine just ignoring
> >> -1s for the Windows part of the build it should be good.
> >>
> >> > Regarding item #2, it is also my interpretation that test-patch
> provides
> >> > an
> >> > on-demand (perhaps 20-minutes deferred) Jenkins build and unit test,
> >> > with
> >> > logs available to the developer, so it would satisfy item #2.  But
> >> > rather
> >> > than assuming that I am interpreting it correctly, I simply want your
> >> > agreement that it would, or if not, clarification why it won't.
> >>
> >> It will satisfy my item #2 in the following way:
> >> I can duplicate your pre-commit build for Windows and add an input
> >> parameter, which would let people run the build on their patches
> >> chosen from local machine rather than attaching them to Jiras.
> >>
> >> Thanks,
> >> --Konstantin
> >>
> >> > In agile terms, you are the Owner of these requirements.  Please give
> me
> >> > owner feedback as to whether my proposed work sounds like it will
> >> > satisfy
> >> > the requirements.
> >> >
> >> > Thank you,
> >> > --Matt
> >> >
> >> >
> >> > On Sun, Mar 3, 2013 at 12:16 PM, Konstantin Shvachko
> >> > <sh...@gmail.com>
> >> > wrote:
> >> >>
> >> >> Didn't I explain in details what I am asking for?
> >> >>
> >> >> Thanks,
> >> >> --Konst
> >> >>
> >> >> On Sun, Mar 3, 2013 at 11:08 AM, Matt Foley <mf...@hortonworks.com>
> >> >> wrote:
> >> >> > Hi Konstantin,
> >> >> > I'd like to point out two things:
> >> >> > First, I already committed in this thread (email of Thu, Feb 28,
> 2013
> >> >> > at
> >> >> > 6:01 PM) to providing CI for Windows builds.  So please stop acting
> >> >> > like
> >> >> > I'm
> >> >> > resisting this idea or something.
> >> >> > Second, you didn't answer my question, you just kvetched about the
> >> >> > phrasing.
> >> >> > So I ask again:
> >> >> >
> >> >> > Will providing full "test-patch" integration (pre-commit build and
> >> >> > unit
> >> >> > test
> >> >> > triggered by Jira "Patch Available" state) satisfy your request for
> >> >> > functionality #1 and #2?  Yes or no, please.
> >> >> >
> >> >> > Thanks,
> >> >> > --Matt
> >> >> >
> >> >> >
> >> >> > On Sat, Mar 2, 2013 at 7:32 PM, Konstantin Shvachko
> >> >> > <sh...@gmail.com>
> >> >> > wrote:
> >> >> >>
> >> >> >> Hi Matt,
> >> >> >>
> >> >> >> On Sat, Mar 2, 2013 at 12:32 PM, Matt Foley <
> mfoley@hortonworks.com>
> >> >> >> wrote:
> >> >> >> > Konstantin,
> >> >> >> > I would like to explore what it would take to remove this
> >> >> >> > perceived
> >> >> >> > impediment --
> >> >> >>
> >> >> >> Glad you decided to explore. Thank you.
> >> >> >>
> >> >> >> > although I reserve the right to argue that this is not
> >> >> >> > pre-requisite to merging the cross-platform support patch.
> >> >> >>
> >> >> >> It's your right indeed. So as mine to question what the platform
> >> >> >> support means for you, which I believe remained unclear.
> >> >> >> I do not impede the change as you should have noticed. My
> >> >> >> requirement
> >> >> >> comes from my perception of the support, which means to me exactly
> >> >> >> two
> >> >> >> things:
> >> >> >> 1. The ability to recognise the code is broken for the platform
> >> >> >> 2. The ability to test new patches on the platform
> >> >> >> The latter is problematic, as many noticed in this thread, for
> those
> >> >> >> whose customary environment does not include Windows.
> >> >> >>
> >> >> >> > If we implemented full "test-patch" support for Windows on
> trunk,
> >> >> >> > would
> >> >> >> > that
> >> >> >> > fulfill both your items #1 and #2?  Please note that:
> >> >> >> > a) Pushing the "Patch Available" button in Jira shall cause a
> >> >> >> > pre-commit
> >> >> >> > build to start within, I believe, 20 minutes.
> >> >> >> > b) That build keeps logs for both java build and unit tests for
> >> >> >> > several
> >> >> >> > days, that are accessible to all viewers.
> >> >> >>
> >> >> >> In item #1 I mostly asking for the nightly build, which is simpler
> >> >> >> than "test-patch". The latter would be ideal from the platform
> >> >> >> support
> >> >> >> viewpoint, but it is for the community to decide if we want to add
> >> >> >> extra +3 hours to the build.
> >> >> >> Nightly build in my understanding is triggered by the timer rather
> >> >> >> than by Jira's "submit patch" button.  On Jenkins build
> >> >> >> configuration
> >> >> >> you can specify it under "Build periodically".
> >> >> >>
> >> >> >> > So, does this provide sufficient on-demand support that we don't
> >> >> >> > have
> >> >> >> > to
> >> >> >> > implement a whole new on-demand VM support structure of some
> sort
> >> >> >> > for
> >> >> >> > #2
> >> >> >> > (which would be an extraordinary and impractical demand)?
> >> >> >>
> >> >> >> I did not mention VMs. Item #2 means a build, which runs
> >> >> >> "test-patch"
> >> >> >> target with the file specified by a user (instead of a jira
> >> >> >> attachment).
> >> >> >> When user clicks "Build Now" link a box is displayed where the
> user
> >> >> >> can enter the file path containing the patch. This can be
> specified
> >> >> >> in
> >> >> >> the Build Configuration under "This build is parameterized" by
> >> >> >> choosing AddParameter / FileParameter. The build can run on the
> same
> >> >> >> Windows machine as the nightly build.
> >> >> >> Such build will let people test their patches for Windows on
> Jenkins
> >> >> >> if they don't posses a license for the right version of Windows.
> >> >> >> I hope this will not turn into extraordinary or impractical
> effort.
> >> >> >>
> >> >> >> Thanks,
> >> >> >> --Konst
> >> >> >>
> >> >> >> > Thanks,
> >> >> >> > --Matt
> >> >> >> >
> >> >> >> >
> >> >> >> > On Fri, Mar 1, 2013 at 12:18 PM, Konstantin Shvachko
> >> >> >> > <sh...@gmail.com>
> >> >> >> > wrote:
> >> >> >> >>
> >> >> >> >> -1
> >> >> >> >> We should have a CI infrastructure in place before we can
> commit
> >> >> >> >> to
> >> >> >> >> supporting Windows platform.
> >> >> >> >>
> >> >> >> >> Eric is right Win/Cygwin was supported since day one.
> >> >> >> >> I had a Windows box under my desk running nightly builds back
> in
> >> >> >> >> 2006-07.
> >> >> >> >> People were irritated but I was filing windows bugs until 0.22
> >> >> >> >> release.
> >> >> >> >> Times changing and I am glad to see wider support for Win
> >> >> >> >> platform.
> >> >> >> >>
> >> >> >> >> But in order to make it work you guys need to put the CI
> process
> >> >> >> >> in
> >> >> >> >> place
> >> >> >> >>
> >> >> >> >> 1. windows jenkins build: could be nightly or PreCommit.
> >> >> >> >> - Nightly would mean that changes can be committed to trunk
> based
> >> >> >> >> on
> >> >> >> >> linux PreCommit build. And people will file bugs if the change
> >> >> >> >> broke
> >> >> >> >> Windows nightly build.
> >> >> >> >> - PreCommit-win build will mean automatic reporting failed
> tests
> >> >> >> >> to
> >> >> >> >> respective jira blocking commits the same way as it is now with
> >> >> >> >> linux
> >> >> >> >> PreCommit builds.
> >> >> >> >> We should discuss which way is more efficient for developers.
> >> >> >> >>
> >> >> >> >> 2. On-demand-windows Jenkins build.
> >> >> >> >> I see it as a build to which I can attach my patch and the
> build
> >> >> >> >> will
> >> >> >> >> run my changes on a dedicated windows box.
> >> >> >> >> That way people can test their changes without having personal
> >> >> >> >> windows
> >> >> >> >> nodes.
> >> >> >> >>
> >> >> >> >> I think this is the minimal set of requirement for us to be
> able
> >> >> >> >> to
> >> >> >> >> commit to the new platform.
> >> >> >> >> Right now I see only one windows related build
> >> >> >> >> https://builds.apache.org/view/Hadoop/job/Hadoop-1-win/
> >> >> >> >> Which was failing since Sept 8, 2012 and did not run in the
> last
> >> >> >> >> month.
> >> >> >> >>
> >> >> >> >> Thanks,
> >> >> >> >> --Konst
> >> >> >> >>
> >> >> >> >> On Thu, Feb 28, 2013 at 8:47 PM, Eric Baldeschwieler
> >> >> >> >> <er...@hortonworks.com> wrote:
> >> >> >> >> > +1 (non-binding)
> >> >> >> >> >
> >> >> >> >> > A few of observations:
> >> >> >> >> >
> >> >> >> >> > - Windows has actually been a supported platform for Hadoop
> >> >> >> >> > since
> >> >> >> >> > 0.1
> >> >> >> >> > .
> >> >> >> >> > Doug championed supporting windows then and we've continued
> to
> >> >> >> >> > do
> >> >> >> >> > it
> >> >> >> >> > with
> >> >> >> >> > varying vigor over time.  To my knowledge we've never made a
> >> >> >> >> > decision
> >> >> >> >> > to
> >> >> >> >> > drop windows support.  The change here is improving our
> support
> >> >> >> >> > and
> >> >> >> >> > dropping
> >> >> >> >> > the requirement of cigwin.  We had Nutch windows users on the
> >> >> >> >> > list
> >> >> >> >> > in
> >> >> >> >> > 2006
> >> >> >> >> > and we've been supporting windows FS requirements since
> >> >> >> >> > inception.
> >> >> >> >> >
> >> >> >> >> > - A little pragmatism will go a long way.  As a community
> we've
> >> >> >> >> > got
> >> >> >> >> > to
> >> >> >> >> > stay committed to keeping hadoop simple (so it does work on
> >> >> >> >> > many
> >> >> >> >> > platforms)
> >> >> >> >> > and extending it to take advantage of key emerging
> OS/hardware
> >> >> >> >> > features,
> >> >> >> >> > such as containers, new FSs, virtualization, flash ...  We
> >> >> >> >> > should
> >> >> >> >> > all
> >> >> >> >> > plan
> >> >> >> >> > to let new features & optimizations emerge that don't work
> >> >> >> >> > everywhere, if
> >> >> >> >> > they are compelling and central to hadoop's mission of being
> >> >> >> >> > THE
> >> >> >> >> > best
> >> >> >> >> > fabric
> >> >> >> >> > for storing and processing big data.
> >> >> >> >> >
> >> >> >> >> > - A UI project like KDE has to deal with the MANY differences
> >> >> >> >> > between
> >> >> >> >> > windows and linux UI APIs.  Hadoop faces no such complex
> >> >> >> >> > challenge
> >> >> >> >> > and hence
> >> >> >> >> > can be maintained from a single codeline IMO.  It is mostly
> >> >> >> >> > abstracted from
> >> >> >> >> > the OS APIs via Java and our design choices.  Where it is not
> >> >> >> >> > we
> >> >> >> >> > can
> >> >> >> >> > continue to add plugable abstractions.
> >> >> >> >> >
> >> >> >> >
> >> >> >> >
> >> >> >
> >> >> >
> >> >
> >> >
> >
> >
>

Re: [Vote] Merge branch-trunk-win to trunk

[Matt Foley <mf...@hortonworks.com>]

Thanks, gentlemen.  I've opened and taken responsibility for
https://issues.apache.org/jira/browse/HADOOP-9359.  Giri Kesavan has agreed
to help with the parts that require Jenkins admin access.

Thanks,
--Matt



On Mon, Mar 4, 2013 at 5:00 PM, Konstantin Shvachko <sh...@gmail.com>wrote:

> +1 on the merge.
>
> I am glad we agreed.
> Having Jira to track the CI effort is a good idea.
>
> Thanks,
> --Konstantin
>
> On Mon, Mar 4, 2013 at 3:29 PM, Matt Foley <mf...@hortonworks.com> wrote:
> > Thanks.  I agree Windows -1's in test-patch should not block commits.
> >
> > --Matt
> >
> >
> >
> > On Mon, Mar 4, 2013 at 2:30 PM, Konstantin Shvachko <
> shv.hadoop@gmail.com>
> > wrote:
> >>
> >> On Mon, Mar 4, 2013 at 12:22 PM, Matt Foley <mf...@hortonworks.com>
> >> wrote:
> >> > Konstantine, you have voted -1, and stated some requirements before
> >> > you'll
> >> > withdraw that -1.  As I plan to do work to fulfill those
> requirements, I
> >> > want to make sure that what I'm proposing will, in fact, satisfy you.
> >> > That's why I'm asking, if we implement full "test-patch" integration
> for
> >> > Windows, does it seem to you that that would provide adequate support?
> >>
> >> Yes.
> >>
> >> > I have learned not to presume that my interpretation is correct.  My
> >> > interpretation of item #1 is that test-patch provides pre-commit
> build,
> >> > so
> >> > it would satisfy item #1.  But rather than assuming that I am
> >> > interpreting
> >> > it correctly, I simply want your agreement that it would, or if not,
> >> > clarification why it won't.
> >>
> >> I agree it will satisfy my item #1.
> >> I did not agree in my previous email, but I changed my mind based on
> >> the latest discussion. I have to explain why now.
> >> I was proposing nightly build because I did not want pre-commit build
> >> for Windows block commits to Linux. But if people are fine just ignoring
> >> -1s for the Windows part of the build it should be good.
> >>
> >> > Regarding item #2, it is also my interpretation that test-patch
> provides
> >> > an
> >> > on-demand (perhaps 20-minutes deferred) Jenkins build and unit test,
> >> > with
> >> > logs available to the developer, so it would satisfy item #2.  But
> >> > rather
> >> > than assuming that I am interpreting it correctly, I simply want your
> >> > agreement that it would, or if not, clarification why it won't.
> >>
> >> It will satisfy my item #2 in the following way:
> >> I can duplicate your pre-commit build for Windows and add an input
> >> parameter, which would let people run the build on their patches
> >> chosen from local machine rather than attaching them to Jiras.
> >>
> >> Thanks,
> >> --Konstantin
> >>
> >> > In agile terms, you are the Owner of these requirements.  Please give
> me
> >> > owner feedback as to whether my proposed work sounds like it will
> >> > satisfy
> >> > the requirements.
> >> >
> >> > Thank you,
> >> > --Matt
> >> >
> >> >
> >> > On Sun, Mar 3, 2013 at 12:16 PM, Konstantin Shvachko
> >> > <sh...@gmail.com>
> >> > wrote:
> >> >>
> >> >> Didn't I explain in details what I am asking for?
> >> >>
> >> >> Thanks,
> >> >> --Konst
> >> >>
> >> >> On Sun, Mar 3, 2013 at 11:08 AM, Matt Foley <mf...@hortonworks.com>
> >> >> wrote:
> >> >> > Hi Konstantin,
> >> >> > I'd like to point out two things:
> >> >> > First, I already committed in this thread (email of Thu, Feb 28,
> 2013
> >> >> > at
> >> >> > 6:01 PM) to providing CI for Windows builds.  So please stop acting
> >> >> > like
> >> >> > I'm
> >> >> > resisting this idea or something.
> >> >> > Second, you didn't answer my question, you just kvetched about the
> >> >> > phrasing.
> >> >> > So I ask again:
> >> >> >
> >> >> > Will providing full "test-patch" integration (pre-commit build and
> >> >> > unit
> >> >> > test
> >> >> > triggered by Jira "Patch Available" state) satisfy your request for
> >> >> > functionality #1 and #2?  Yes or no, please.
> >> >> >
> >> >> > Thanks,
> >> >> > --Matt
> >> >> >
> >> >> >
> >> >> > On Sat, Mar 2, 2013 at 7:32 PM, Konstantin Shvachko
> >> >> > <sh...@gmail.com>
> >> >> > wrote:
> >> >> >>
> >> >> >> Hi Matt,
> >> >> >>
> >> >> >> On Sat, Mar 2, 2013 at 12:32 PM, Matt Foley <
> mfoley@hortonworks.com>
> >> >> >> wrote:
> >> >> >> > Konstantin,
> >> >> >> > I would like to explore what it would take to remove this
> >> >> >> > perceived
> >> >> >> > impediment --
> >> >> >>
> >> >> >> Glad you decided to explore. Thank you.
> >> >> >>
> >> >> >> > although I reserve the right to argue that this is not
> >> >> >> > pre-requisite to merging the cross-platform support patch.
> >> >> >>
> >> >> >> It's your right indeed. So as mine to question what the platform
> >> >> >> support means for you, which I believe remained unclear.
> >> >> >> I do not impede the change as you should have noticed. My
> >> >> >> requirement
> >> >> >> comes from my perception of the support, which means to me exactly
> >> >> >> two
> >> >> >> things:
> >> >> >> 1. The ability to recognise the code is broken for the platform
> >> >> >> 2. The ability to test new patches on the platform
> >> >> >> The latter is problematic, as many noticed in this thread, for
> those
> >> >> >> whose customary environment does not include Windows.
> >> >> >>
> >> >> >> > If we implemented full "test-patch" support for Windows on
> trunk,
> >> >> >> > would
> >> >> >> > that
> >> >> >> > fulfill both your items #1 and #2?  Please note that:
> >> >> >> > a) Pushing the "Patch Available" button in Jira shall cause a
> >> >> >> > pre-commit
> >> >> >> > build to start within, I believe, 20 minutes.
> >> >> >> > b) That build keeps logs for both java build and unit tests for
> >> >> >> > several
> >> >> >> > days, that are accessible to all viewers.
> >> >> >>
> >> >> >> In item #1 I mostly asking for the nightly build, which is simpler
> >> >> >> than "test-patch". The latter would be ideal from the platform
> >> >> >> support
> >> >> >> viewpoint, but it is for the community to decide if we want to add
> >> >> >> extra +3 hours to the build.
> >> >> >> Nightly build in my understanding is triggered by the timer rather
> >> >> >> than by Jira's "submit patch" button.  On Jenkins build
> >> >> >> configuration
> >> >> >> you can specify it under "Build periodically".
> >> >> >>
> >> >> >> > So, does this provide sufficient on-demand support that we don't
> >> >> >> > have
> >> >> >> > to
> >> >> >> > implement a whole new on-demand VM support structure of some
> sort
> >> >> >> > for
> >> >> >> > #2
> >> >> >> > (which would be an extraordinary and impractical demand)?
> >> >> >>
> >> >> >> I did not mention VMs. Item #2 means a build, which runs
> >> >> >> "test-patch"
> >> >> >> target with the file specified by a user (instead of a jira
> >> >> >> attachment).
> >> >> >> When user clicks "Build Now" link a box is displayed where the
> user
> >> >> >> can enter the file path containing the patch. This can be
> specified
> >> >> >> in
> >> >> >> the Build Configuration under "This build is parameterized" by
> >> >> >> choosing AddParameter / FileParameter. The build can run on the
> same
> >> >> >> Windows machine as the nightly build.
> >> >> >> Such build will let people test their patches for Windows on
> Jenkins
> >> >> >> if they don't posses a license for the right version of Windows.
> >> >> >> I hope this will not turn into extraordinary or impractical
> effort.
> >> >> >>
> >> >> >> Thanks,
> >> >> >> --Konst
> >> >> >>
> >> >> >> > Thanks,
> >> >> >> > --Matt
> >> >> >> >
> >> >> >> >
> >> >> >> > On Fri, Mar 1, 2013 at 12:18 PM, Konstantin Shvachko
> >> >> >> > <sh...@gmail.com>
> >> >> >> > wrote:
> >> >> >> >>
> >> >> >> >> -1
> >> >> >> >> We should have a CI infrastructure in place before we can
> commit
> >> >> >> >> to
> >> >> >> >> supporting Windows platform.
> >> >> >> >>
> >> >> >> >> Eric is right Win/Cygwin was supported since day one.
> >> >> >> >> I had a Windows box under my desk running nightly builds back
> in
> >> >> >> >> 2006-07.
> >> >> >> >> People were irritated but I was filing windows bugs until 0.22
> >> >> >> >> release.
> >> >> >> >> Times changing and I am glad to see wider support for Win
> >> >> >> >> platform.
> >> >> >> >>
> >> >> >> >> But in order to make it work you guys need to put the CI
> process
> >> >> >> >> in
> >> >> >> >> place
> >> >> >> >>
> >> >> >> >> 1. windows jenkins build: could be nightly or PreCommit.
> >> >> >> >> - Nightly would mean that changes can be committed to trunk
> based
> >> >> >> >> on
> >> >> >> >> linux PreCommit build. And people will file bugs if the change
> >> >> >> >> broke
> >> >> >> >> Windows nightly build.
> >> >> >> >> - PreCommit-win build will mean automatic reporting failed
> tests
> >> >> >> >> to
> >> >> >> >> respective jira blocking commits the same way as it is now with
> >> >> >> >> linux
> >> >> >> >> PreCommit builds.
> >> >> >> >> We should discuss which way is more efficient for developers.
> >> >> >> >>
> >> >> >> >> 2. On-demand-windows Jenkins build.
> >> >> >> >> I see it as a build to which I can attach my patch and the
> build
> >> >> >> >> will
> >> >> >> >> run my changes on a dedicated windows box.
> >> >> >> >> That way people can test their changes without having personal
> >> >> >> >> windows
> >> >> >> >> nodes.
> >> >> >> >>
> >> >> >> >> I think this is the minimal set of requirement for us to be
> able
> >> >> >> >> to
> >> >> >> >> commit to the new platform.
> >> >> >> >> Right now I see only one windows related build
> >> >> >> >> https://builds.apache.org/view/Hadoop/job/Hadoop-1-win/
> >> >> >> >> Which was failing since Sept 8, 2012 and did not run in the
> last
> >> >> >> >> month.
> >> >> >> >>
> >> >> >> >> Thanks,
> >> >> >> >> --Konst
> >> >> >> >>
> >> >> >> >> On Thu, Feb 28, 2013 at 8:47 PM, Eric Baldeschwieler
> >> >> >> >> <er...@hortonworks.com> wrote:
> >> >> >> >> > +1 (non-binding)
> >> >> >> >> >
> >> >> >> >> > A few of observations:
> >> >> >> >> >
> >> >> >> >> > - Windows has actually been a supported platform for Hadoop
> >> >> >> >> > since
> >> >> >> >> > 0.1
> >> >> >> >> > .
> >> >> >> >> > Doug championed supporting windows then and we've continued
> to
> >> >> >> >> > do
> >> >> >> >> > it
> >> >> >> >> > with
> >> >> >> >> > varying vigor over time.  To my knowledge we've never made a
> >> >> >> >> > decision
> >> >> >> >> > to
> >> >> >> >> > drop windows support.  The change here is improving our
> support
> >> >> >> >> > and
> >> >> >> >> > dropping
> >> >> >> >> > the requirement of cigwin.  We had Nutch windows users on the
> >> >> >> >> > list
> >> >> >> >> > in
> >> >> >> >> > 2006
> >> >> >> >> > and we've been supporting windows FS requirements since
> >> >> >> >> > inception.
> >> >> >> >> >
> >> >> >> >> > - A little pragmatism will go a long way.  As a community
> we've
> >> >> >> >> > got
> >> >> >> >> > to
> >> >> >> >> > stay committed to keeping hadoop simple (so it does work on
> >> >> >> >> > many
> >> >> >> >> > platforms)
> >> >> >> >> > and extending it to take advantage of key emerging
> OS/hardware
> >> >> >> >> > features,
> >> >> >> >> > such as containers, new FSs, virtualization, flash ...  We
> >> >> >> >> > should
> >> >> >> >> > all
> >> >> >> >> > plan
> >> >> >> >> > to let new features & optimizations emerge that don't work
> >> >> >> >> > everywhere, if
> >> >> >> >> > they are compelling and central to hadoop's mission of being
> >> >> >> >> > THE
> >> >> >> >> > best
> >> >> >> >> > fabric
> >> >> >> >> > for storing and processing big data.
> >> >> >> >> >
> >> >> >> >> > - A UI project like KDE has to deal with the MANY differences
> >> >> >> >> > between
> >> >> >> >> > windows and linux UI APIs.  Hadoop faces no such complex
> >> >> >> >> > challenge
> >> >> >> >> > and hence
> >> >> >> >> > can be maintained from a single codeline IMO.  It is mostly
> >> >> >> >> > abstracted from
> >> >> >> >> > the OS APIs via Java and our design choices.  Where it is not
> >> >> >> >> > we
> >> >> >> >> > can
> >> >> >> >> > continue to add plugable abstractions.
> >> >> >> >> >
> >> >> >> >
> >> >> >> >
> >> >> >
> >> >> >
> >> >
> >> >
> >
> >
>

Re: [Vote] Merge branch-trunk-win to trunk

[Konstantin Shvachko <sh...@gmail.com>]

+1 on the merge.

I am glad we agreed.
Having Jira to track the CI effort is a good idea.

Thanks,
--Konstantin

On Mon, Mar 4, 2013 at 3:29 PM, Matt Foley <mf...@hortonworks.com> wrote:
> Thanks.  I agree Windows -1's in test-patch should not block commits.
>
> --Matt
>
>
>
> On Mon, Mar 4, 2013 at 2:30 PM, Konstantin Shvachko <sh...@gmail.com>
> wrote:
>>
>> On Mon, Mar 4, 2013 at 12:22 PM, Matt Foley <mf...@hortonworks.com>
>> wrote:
>> > Konstantine, you have voted -1, and stated some requirements before
>> > you'll
>> > withdraw that -1.  As I plan to do work to fulfill those requirements, I
>> > want to make sure that what I'm proposing will, in fact, satisfy you.
>> > That's why I'm asking, if we implement full "test-patch" integration for
>> > Windows, does it seem to you that that would provide adequate support?
>>
>> Yes.
>>
>> > I have learned not to presume that my interpretation is correct.  My
>> > interpretation of item #1 is that test-patch provides pre-commit build,
>> > so
>> > it would satisfy item #1.  But rather than assuming that I am
>> > interpreting
>> > it correctly, I simply want your agreement that it would, or if not,
>> > clarification why it won't.
>>
>> I agree it will satisfy my item #1.
>> I did not agree in my previous email, but I changed my mind based on
>> the latest discussion. I have to explain why now.
>> I was proposing nightly build because I did not want pre-commit build
>> for Windows block commits to Linux. But if people are fine just ignoring
>> -1s for the Windows part of the build it should be good.
>>
>> > Regarding item #2, it is also my interpretation that test-patch provides
>> > an
>> > on-demand (perhaps 20-minutes deferred) Jenkins build and unit test,
>> > with
>> > logs available to the developer, so it would satisfy item #2.  But
>> > rather
>> > than assuming that I am interpreting it correctly, I simply want your
>> > agreement that it would, or if not, clarification why it won't.
>>
>> It will satisfy my item #2 in the following way:
>> I can duplicate your pre-commit build for Windows and add an input
>> parameter, which would let people run the build on their patches
>> chosen from local machine rather than attaching them to Jiras.
>>
>> Thanks,
>> --Konstantin
>>
>> > In agile terms, you are the Owner of these requirements.  Please give me
>> > owner feedback as to whether my proposed work sounds like it will
>> > satisfy
>> > the requirements.
>> >
>> > Thank you,
>> > --Matt
>> >
>> >
>> > On Sun, Mar 3, 2013 at 12:16 PM, Konstantin Shvachko
>> > <sh...@gmail.com>
>> > wrote:
>> >>
>> >> Didn't I explain in details what I am asking for?
>> >>
>> >> Thanks,
>> >> --Konst
>> >>
>> >> On Sun, Mar 3, 2013 at 11:08 AM, Matt Foley <mf...@hortonworks.com>
>> >> wrote:
>> >> > Hi Konstantin,
>> >> > I'd like to point out two things:
>> >> > First, I already committed in this thread (email of Thu, Feb 28, 2013
>> >> > at
>> >> > 6:01 PM) to providing CI for Windows builds.  So please stop acting
>> >> > like
>> >> > I'm
>> >> > resisting this idea or something.
>> >> > Second, you didn't answer my question, you just kvetched about the
>> >> > phrasing.
>> >> > So I ask again:
>> >> >
>> >> > Will providing full "test-patch" integration (pre-commit build and
>> >> > unit
>> >> > test
>> >> > triggered by Jira "Patch Available" state) satisfy your request for
>> >> > functionality #1 and #2?  Yes or no, please.
>> >> >
>> >> > Thanks,
>> >> > --Matt
>> >> >
>> >> >
>> >> > On Sat, Mar 2, 2013 at 7:32 PM, Konstantin Shvachko
>> >> > <sh...@gmail.com>
>> >> > wrote:
>> >> >>
>> >> >> Hi Matt,
>> >> >>
>> >> >> On Sat, Mar 2, 2013 at 12:32 PM, Matt Foley <mf...@hortonworks.com>
>> >> >> wrote:
>> >> >> > Konstantin,
>> >> >> > I would like to explore what it would take to remove this
>> >> >> > perceived
>> >> >> > impediment --
>> >> >>
>> >> >> Glad you decided to explore. Thank you.
>> >> >>
>> >> >> > although I reserve the right to argue that this is not
>> >> >> > pre-requisite to merging the cross-platform support patch.
>> >> >>
>> >> >> It's your right indeed. So as mine to question what the platform
>> >> >> support means for you, which I believe remained unclear.
>> >> >> I do not impede the change as you should have noticed. My
>> >> >> requirement
>> >> >> comes from my perception of the support, which means to me exactly
>> >> >> two
>> >> >> things:
>> >> >> 1. The ability to recognise the code is broken for the platform
>> >> >> 2. The ability to test new patches on the platform
>> >> >> The latter is problematic, as many noticed in this thread, for those
>> >> >> whose customary environment does not include Windows.
>> >> >>
>> >> >> > If we implemented full "test-patch" support for Windows on trunk,
>> >> >> > would
>> >> >> > that
>> >> >> > fulfill both your items #1 and #2?  Please note that:
>> >> >> > a) Pushing the "Patch Available" button in Jira shall cause a
>> >> >> > pre-commit
>> >> >> > build to start within, I believe, 20 minutes.
>> >> >> > b) That build keeps logs for both java build and unit tests for
>> >> >> > several
>> >> >> > days, that are accessible to all viewers.
>> >> >>
>> >> >> In item #1 I mostly asking for the nightly build, which is simpler
>> >> >> than "test-patch". The latter would be ideal from the platform
>> >> >> support
>> >> >> viewpoint, but it is for the community to decide if we want to add
>> >> >> extra +3 hours to the build.
>> >> >> Nightly build in my understanding is triggered by the timer rather
>> >> >> than by Jira's "submit patch" button.  On Jenkins build
>> >> >> configuration
>> >> >> you can specify it under "Build periodically".
>> >> >>
>> >> >> > So, does this provide sufficient on-demand support that we don't
>> >> >> > have
>> >> >> > to
>> >> >> > implement a whole new on-demand VM support structure of some sort
>> >> >> > for
>> >> >> > #2
>> >> >> > (which would be an extraordinary and impractical demand)?
>> >> >>
>> >> >> I did not mention VMs. Item #2 means a build, which runs
>> >> >> "test-patch"
>> >> >> target with the file specified by a user (instead of a jira
>> >> >> attachment).
>> >> >> When user clicks "Build Now" link a box is displayed where the user
>> >> >> can enter the file path containing the patch. This can be specified
>> >> >> in
>> >> >> the Build Configuration under "This build is parameterized" by
>> >> >> choosing AddParameter / FileParameter. The build can run on the same
>> >> >> Windows machine as the nightly build.
>> >> >> Such build will let people test their patches for Windows on Jenkins
>> >> >> if they don't posses a license for the right version of Windows.
>> >> >> I hope this will not turn into extraordinary or impractical effort.
>> >> >>
>> >> >> Thanks,
>> >> >> --Konst
>> >> >>
>> >> >> > Thanks,
>> >> >> > --Matt
>> >> >> >
>> >> >> >
>> >> >> > On Fri, Mar 1, 2013 at 12:18 PM, Konstantin Shvachko
>> >> >> > <sh...@gmail.com>
>> >> >> > wrote:
>> >> >> >>
>> >> >> >> -1
>> >> >> >> We should have a CI infrastructure in place before we can commit
>> >> >> >> to
>> >> >> >> supporting Windows platform.
>> >> >> >>
>> >> >> >> Eric is right Win/Cygwin was supported since day one.
>> >> >> >> I had a Windows box under my desk running nightly builds back in
>> >> >> >> 2006-07.
>> >> >> >> People were irritated but I was filing windows bugs until 0.22
>> >> >> >> release.
>> >> >> >> Times changing and I am glad to see wider support for Win
>> >> >> >> platform.
>> >> >> >>
>> >> >> >> But in order to make it work you guys need to put the CI process
>> >> >> >> in
>> >> >> >> place
>> >> >> >>
>> >> >> >> 1. windows jenkins build: could be nightly or PreCommit.
>> >> >> >> - Nightly would mean that changes can be committed to trunk based
>> >> >> >> on
>> >> >> >> linux PreCommit build. And people will file bugs if the change
>> >> >> >> broke
>> >> >> >> Windows nightly build.
>> >> >> >> - PreCommit-win build will mean automatic reporting failed tests
>> >> >> >> to
>> >> >> >> respective jira blocking commits the same way as it is now with
>> >> >> >> linux
>> >> >> >> PreCommit builds.
>> >> >> >> We should discuss which way is more efficient for developers.
>> >> >> >>
>> >> >> >> 2. On-demand-windows Jenkins build.
>> >> >> >> I see it as a build to which I can attach my patch and the build
>> >> >> >> will
>> >> >> >> run my changes on a dedicated windows box.
>> >> >> >> That way people can test their changes without having personal
>> >> >> >> windows
>> >> >> >> nodes.
>> >> >> >>
>> >> >> >> I think this is the minimal set of requirement for us to be able
>> >> >> >> to
>> >> >> >> commit to the new platform.
>> >> >> >> Right now I see only one windows related build
>> >> >> >> https://builds.apache.org/view/Hadoop/job/Hadoop-1-win/
>> >> >> >> Which was failing since Sept 8, 2012 and did not run in the last
>> >> >> >> month.
>> >> >> >>
>> >> >> >> Thanks,
>> >> >> >> --Konst
>> >> >> >>
>> >> >> >> On Thu, Feb 28, 2013 at 8:47 PM, Eric Baldeschwieler
>> >> >> >> <er...@hortonworks.com> wrote:
>> >> >> >> > +1 (non-binding)
>> >> >> >> >
>> >> >> >> > A few of observations:
>> >> >> >> >
>> >> >> >> > - Windows has actually been a supported platform for Hadoop
>> >> >> >> > since
>> >> >> >> > 0.1
>> >> >> >> > .
>> >> >> >> > Doug championed supporting windows then and we've continued to
>> >> >> >> > do
>> >> >> >> > it
>> >> >> >> > with
>> >> >> >> > varying vigor over time.  To my knowledge we've never made a
>> >> >> >> > decision
>> >> >> >> > to
>> >> >> >> > drop windows support.  The change here is improving our support
>> >> >> >> > and
>> >> >> >> > dropping
>> >> >> >> > the requirement of cigwin.  We had Nutch windows users on the
>> >> >> >> > list
>> >> >> >> > in
>> >> >> >> > 2006
>> >> >> >> > and we've been supporting windows FS requirements since
>> >> >> >> > inception.
>> >> >> >> >
>> >> >> >> > - A little pragmatism will go a long way.  As a community we've
>> >> >> >> > got
>> >> >> >> > to
>> >> >> >> > stay committed to keeping hadoop simple (so it does work on
>> >> >> >> > many
>> >> >> >> > platforms)
>> >> >> >> > and extending it to take advantage of key emerging OS/hardware
>> >> >> >> > features,
>> >> >> >> > such as containers, new FSs, virtualization, flash ...  We
>> >> >> >> > should
>> >> >> >> > all
>> >> >> >> > plan
>> >> >> >> > to let new features & optimizations emerge that don't work
>> >> >> >> > everywhere, if
>> >> >> >> > they are compelling and central to hadoop's mission of being
>> >> >> >> > THE
>> >> >> >> > best
>> >> >> >> > fabric
>> >> >> >> > for storing and processing big data.
>> >> >> >> >
>> >> >> >> > - A UI project like KDE has to deal with the MANY differences
>> >> >> >> > between
>> >> >> >> > windows and linux UI APIs.  Hadoop faces no such complex
>> >> >> >> > challenge
>> >> >> >> > and hence
>> >> >> >> > can be maintained from a single codeline IMO.  It is mostly
>> >> >> >> > abstracted from
>> >> >> >> > the OS APIs via Java and our design choices.  Where it is not
>> >> >> >> > we
>> >> >> >> > can
>> >> >> >> > continue to add plugable abstractions.
>> >> >> >> >
>> >> >> >
>> >> >> >
>> >> >
>> >> >
>> >
>> >
>
>

Re: [Vote] Merge branch-trunk-win to trunk

[Konstantin Shvachko <sh...@gmail.com>]

+1 on the merge.

I am glad we agreed.
Having Jira to track the CI effort is a good idea.

Thanks,
--Konstantin

On Mon, Mar 4, 2013 at 3:29 PM, Matt Foley <mf...@hortonworks.com> wrote:
> Thanks.  I agree Windows -1's in test-patch should not block commits.
>
> --Matt
>
>
>
> On Mon, Mar 4, 2013 at 2:30 PM, Konstantin Shvachko <sh...@gmail.com>
> wrote:
>>
>> On Mon, Mar 4, 2013 at 12:22 PM, Matt Foley <mf...@hortonworks.com>
>> wrote:
>> > Konstantine, you have voted -1, and stated some requirements before
>> > you'll
>> > withdraw that -1.  As I plan to do work to fulfill those requirements, I
>> > want to make sure that what I'm proposing will, in fact, satisfy you.
>> > That's why I'm asking, if we implement full "test-patch" integration for
>> > Windows, does it seem to you that that would provide adequate support?
>>
>> Yes.
>>
>> > I have learned not to presume that my interpretation is correct.  My
>> > interpretation of item #1 is that test-patch provides pre-commit build,
>> > so
>> > it would satisfy item #1.  But rather than assuming that I am
>> > interpreting
>> > it correctly, I simply want your agreement that it would, or if not,
>> > clarification why it won't.
>>
>> I agree it will satisfy my item #1.
>> I did not agree in my previous email, but I changed my mind based on
>> the latest discussion. I have to explain why now.
>> I was proposing nightly build because I did not want pre-commit build
>> for Windows block commits to Linux. But if people are fine just ignoring
>> -1s for the Windows part of the build it should be good.
>>
>> > Regarding item #2, it is also my interpretation that test-patch provides
>> > an
>> > on-demand (perhaps 20-minutes deferred) Jenkins build and unit test,
>> > with
>> > logs available to the developer, so it would satisfy item #2.  But
>> > rather
>> > than assuming that I am interpreting it correctly, I simply want your
>> > agreement that it would, or if not, clarification why it won't.
>>
>> It will satisfy my item #2 in the following way:
>> I can duplicate your pre-commit build for Windows and add an input
>> parameter, which would let people run the build on their patches
>> chosen from local machine rather than attaching them to Jiras.
>>
>> Thanks,
>> --Konstantin
>>
>> > In agile terms, you are the Owner of these requirements.  Please give me
>> > owner feedback as to whether my proposed work sounds like it will
>> > satisfy
>> > the requirements.
>> >
>> > Thank you,
>> > --Matt
>> >
>> >
>> > On Sun, Mar 3, 2013 at 12:16 PM, Konstantin Shvachko
>> > <sh...@gmail.com>
>> > wrote:
>> >>
>> >> Didn't I explain in details what I am asking for?
>> >>
>> >> Thanks,
>> >> --Konst
>> >>
>> >> On Sun, Mar 3, 2013 at 11:08 AM, Matt Foley <mf...@hortonworks.com>
>> >> wrote:
>> >> > Hi Konstantin,
>> >> > I'd like to point out two things:
>> >> > First, I already committed in this thread (email of Thu, Feb 28, 2013
>> >> > at
>> >> > 6:01 PM) to providing CI for Windows builds.  So please stop acting
>> >> > like
>> >> > I'm
>> >> > resisting this idea or something.
>> >> > Second, you didn't answer my question, you just kvetched about the
>> >> > phrasing.
>> >> > So I ask again:
>> >> >
>> >> > Will providing full "test-patch" integration (pre-commit build and
>> >> > unit
>> >> > test
>> >> > triggered by Jira "Patch Available" state) satisfy your request for
>> >> > functionality #1 and #2?  Yes or no, please.
>> >> >
>> >> > Thanks,
>> >> > --Matt
>> >> >
>> >> >
>> >> > On Sat, Mar 2, 2013 at 7:32 PM, Konstantin Shvachko
>> >> > <sh...@gmail.com>
>> >> > wrote:
>> >> >>
>> >> >> Hi Matt,
>> >> >>
>> >> >> On Sat, Mar 2, 2013 at 12:32 PM, Matt Foley <mf...@hortonworks.com>
>> >> >> wrote:
>> >> >> > Konstantin,
>> >> >> > I would like to explore what it would take to remove this
>> >> >> > perceived
>> >> >> > impediment --
>> >> >>
>> >> >> Glad you decided to explore. Thank you.
>> >> >>
>> >> >> > although I reserve the right to argue that this is not
>> >> >> > pre-requisite to merging the cross-platform support patch.
>> >> >>
>> >> >> It's your right indeed. So as mine to question what the platform
>> >> >> support means for you, which I believe remained unclear.
>> >> >> I do not impede the change as you should have noticed. My
>> >> >> requirement
>> >> >> comes from my perception of the support, which means to me exactly
>> >> >> two
>> >> >> things:
>> >> >> 1. The ability to recognise the code is broken for the platform
>> >> >> 2. The ability to test new patches on the platform
>> >> >> The latter is problematic, as many noticed in this thread, for those
>> >> >> whose customary environment does not include Windows.
>> >> >>
>> >> >> > If we implemented full "test-patch" support for Windows on trunk,
>> >> >> > would
>> >> >> > that
>> >> >> > fulfill both your items #1 and #2?  Please note that:
>> >> >> > a) Pushing the "Patch Available" button in Jira shall cause a
>> >> >> > pre-commit
>> >> >> > build to start within, I believe, 20 minutes.
>> >> >> > b) That build keeps logs for both java build and unit tests for
>> >> >> > several
>> >> >> > days, that are accessible to all viewers.
>> >> >>
>> >> >> In item #1 I mostly asking for the nightly build, which is simpler
>> >> >> than "test-patch". The latter would be ideal from the platform
>> >> >> support
>> >> >> viewpoint, but it is for the community to decide if we want to add
>> >> >> extra +3 hours to the build.
>> >> >> Nightly build in my understanding is triggered by the timer rather
>> >> >> than by Jira's "submit patch" button.  On Jenkins build
>> >> >> configuration
>> >> >> you can specify it under "Build periodically".
>> >> >>
>> >> >> > So, does this provide sufficient on-demand support that we don't
>> >> >> > have
>> >> >> > to
>> >> >> > implement a whole new on-demand VM support structure of some sort
>> >> >> > for
>> >> >> > #2
>> >> >> > (which would be an extraordinary and impractical demand)?
>> >> >>
>> >> >> I did not mention VMs. Item #2 means a build, which runs
>> >> >> "test-patch"
>> >> >> target with the file specified by a user (instead of a jira
>> >> >> attachment).
>> >> >> When user clicks "Build Now" link a box is displayed where the user
>> >> >> can enter the file path containing the patch. This can be specified
>> >> >> in
>> >> >> the Build Configuration under "This build is parameterized" by
>> >> >> choosing AddParameter / FileParameter. The build can run on the same
>> >> >> Windows machine as the nightly build.
>> >> >> Such build will let people test their patches for Windows on Jenkins
>> >> >> if they don't posses a license for the right version of Windows.
>> >> >> I hope this will not turn into extraordinary or impractical effort.
>> >> >>
>> >> >> Thanks,
>> >> >> --Konst
>> >> >>
>> >> >> > Thanks,
>> >> >> > --Matt
>> >> >> >
>> >> >> >
>> >> >> > On Fri, Mar 1, 2013 at 12:18 PM, Konstantin Shvachko
>> >> >> > <sh...@gmail.com>
>> >> >> > wrote:
>> >> >> >>
>> >> >> >> -1
>> >> >> >> We should have a CI infrastructure in place before we can commit
>> >> >> >> to
>> >> >> >> supporting Windows platform.
>> >> >> >>
>> >> >> >> Eric is right Win/Cygwin was supported since day one.
>> >> >> >> I had a Windows box under my desk running nightly builds back in
>> >> >> >> 2006-07.
>> >> >> >> People were irritated but I was filing windows bugs until 0.22
>> >> >> >> release.
>> >> >> >> Times changing and I am glad to see wider support for Win
>> >> >> >> platform.
>> >> >> >>
>> >> >> >> But in order to make it work you guys need to put the CI process
>> >> >> >> in
>> >> >> >> place
>> >> >> >>
>> >> >> >> 1. windows jenkins build: could be nightly or PreCommit.
>> >> >> >> - Nightly would mean that changes can be committed to trunk based
>> >> >> >> on
>> >> >> >> linux PreCommit build. And people will file bugs if the change
>> >> >> >> broke
>> >> >> >> Windows nightly build.
>> >> >> >> - PreCommit-win build will mean automatic reporting failed tests
>> >> >> >> to
>> >> >> >> respective jira blocking commits the same way as it is now with
>> >> >> >> linux
>> >> >> >> PreCommit builds.
>> >> >> >> We should discuss which way is more efficient for developers.
>> >> >> >>
>> >> >> >> 2. On-demand-windows Jenkins build.
>> >> >> >> I see it as a build to which I can attach my patch and the build
>> >> >> >> will
>> >> >> >> run my changes on a dedicated windows box.
>> >> >> >> That way people can test their changes without having personal
>> >> >> >> windows
>> >> >> >> nodes.
>> >> >> >>
>> >> >> >> I think this is the minimal set of requirement for us to be able
>> >> >> >> to
>> >> >> >> commit to the new platform.
>> >> >> >> Right now I see only one windows related build
>> >> >> >> https://builds.apache.org/view/Hadoop/job/Hadoop-1-win/
>> >> >> >> Which was failing since Sept 8, 2012 and did not run in the last
>> >> >> >> month.
>> >> >> >>
>> >> >> >> Thanks,
>> >> >> >> --Konst
>> >> >> >>
>> >> >> >> On Thu, Feb 28, 2013 at 8:47 PM, Eric Baldeschwieler
>> >> >> >> <er...@hortonworks.com> wrote:
>> >> >> >> > +1 (non-binding)
>> >> >> >> >
>> >> >> >> > A few of observations:
>> >> >> >> >
>> >> >> >> > - Windows has actually been a supported platform for Hadoop
>> >> >> >> > since
>> >> >> >> > 0.1
>> >> >> >> > .
>> >> >> >> > Doug championed supporting windows then and we've continued to
>> >> >> >> > do
>> >> >> >> > it
>> >> >> >> > with
>> >> >> >> > varying vigor over time.  To my knowledge we've never made a
>> >> >> >> > decision
>> >> >> >> > to
>> >> >> >> > drop windows support.  The change here is improving our support
>> >> >> >> > and
>> >> >> >> > dropping
>> >> >> >> > the requirement of cigwin.  We had Nutch windows users on the
>> >> >> >> > list
>> >> >> >> > in
>> >> >> >> > 2006
>> >> >> >> > and we've been supporting windows FS requirements since
>> >> >> >> > inception.
>> >> >> >> >
>> >> >> >> > - A little pragmatism will go a long way.  As a community we've
>> >> >> >> > got
>> >> >> >> > to
>> >> >> >> > stay committed to keeping hadoop simple (so it does work on
>> >> >> >> > many
>> >> >> >> > platforms)
>> >> >> >> > and extending it to take advantage of key emerging OS/hardware
>> >> >> >> > features,
>> >> >> >> > such as containers, new FSs, virtualization, flash ...  We
>> >> >> >> > should
>> >> >> >> > all
>> >> >> >> > plan
>> >> >> >> > to let new features & optimizations emerge that don't work
>> >> >> >> > everywhere, if
>> >> >> >> > they are compelling and central to hadoop's mission of being
>> >> >> >> > THE
>> >> >> >> > best
>> >> >> >> > fabric
>> >> >> >> > for storing and processing big data.
>> >> >> >> >
>> >> >> >> > - A UI project like KDE has to deal with the MANY differences
>> >> >> >> > between
>> >> >> >> > windows and linux UI APIs.  Hadoop faces no such complex
>> >> >> >> > challenge
>> >> >> >> > and hence
>> >> >> >> > can be maintained from a single codeline IMO.  It is mostly
>> >> >> >> > abstracted from
>> >> >> >> > the OS APIs via Java and our design choices.  Where it is not
>> >> >> >> > we
>> >> >> >> > can
>> >> >> >> > continue to add plugable abstractions.
>> >> >> >> >
>> >> >> >
>> >> >> >
>> >> >
>> >> >
>> >
>> >
>
>

Re: [Vote] Merge branch-trunk-win to trunk

[Konstantin Shvachko <sh...@gmail.com>]

+1 on the merge.

I am glad we agreed.
Having Jira to track the CI effort is a good idea.

Thanks,
--Konstantin

On Mon, Mar 4, 2013 at 3:29 PM, Matt Foley <mf...@hortonworks.com> wrote:
> Thanks.  I agree Windows -1's in test-patch should not block commits.
>
> --Matt
>
>
>
> On Mon, Mar 4, 2013 at 2:30 PM, Konstantin Shvachko <sh...@gmail.com>
> wrote:
>>
>> On Mon, Mar 4, 2013 at 12:22 PM, Matt Foley <mf...@hortonworks.com>
>> wrote:
>> > Konstantine, you have voted -1, and stated some requirements before
>> > you'll
>> > withdraw that -1.  As I plan to do work to fulfill those requirements, I
>> > want to make sure that what I'm proposing will, in fact, satisfy you.
>> > That's why I'm asking, if we implement full "test-patch" integration for
>> > Windows, does it seem to you that that would provide adequate support?
>>
>> Yes.
>>
>> > I have learned not to presume that my interpretation is correct.  My
>> > interpretation of item #1 is that test-patch provides pre-commit build,
>> > so
>> > it would satisfy item #1.  But rather than assuming that I am
>> > interpreting
>> > it correctly, I simply want your agreement that it would, or if not,
>> > clarification why it won't.
>>
>> I agree it will satisfy my item #1.
>> I did not agree in my previous email, but I changed my mind based on
>> the latest discussion. I have to explain why now.
>> I was proposing nightly build because I did not want pre-commit build
>> for Windows block commits to Linux. But if people are fine just ignoring
>> -1s for the Windows part of the build it should be good.
>>
>> > Regarding item #2, it is also my interpretation that test-patch provides
>> > an
>> > on-demand (perhaps 20-minutes deferred) Jenkins build and unit test,
>> > with
>> > logs available to the developer, so it would satisfy item #2.  But
>> > rather
>> > than assuming that I am interpreting it correctly, I simply want your
>> > agreement that it would, or if not, clarification why it won't.
>>
>> It will satisfy my item #2 in the following way:
>> I can duplicate your pre-commit build for Windows and add an input
>> parameter, which would let people run the build on their patches
>> chosen from local machine rather than attaching them to Jiras.
>>
>> Thanks,
>> --Konstantin
>>
>> > In agile terms, you are the Owner of these requirements.  Please give me
>> > owner feedback as to whether my proposed work sounds like it will
>> > satisfy
>> > the requirements.
>> >
>> > Thank you,
>> > --Matt
>> >
>> >
>> > On Sun, Mar 3, 2013 at 12:16 PM, Konstantin Shvachko
>> > <sh...@gmail.com>
>> > wrote:
>> >>
>> >> Didn't I explain in details what I am asking for?
>> >>
>> >> Thanks,
>> >> --Konst
>> >>
>> >> On Sun, Mar 3, 2013 at 11:08 AM, Matt Foley <mf...@hortonworks.com>
>> >> wrote:
>> >> > Hi Konstantin,
>> >> > I'd like to point out two things:
>> >> > First, I already committed in this thread (email of Thu, Feb 28, 2013
>> >> > at
>> >> > 6:01 PM) to providing CI for Windows builds.  So please stop acting
>> >> > like
>> >> > I'm
>> >> > resisting this idea or something.
>> >> > Second, you didn't answer my question, you just kvetched about the
>> >> > phrasing.
>> >> > So I ask again:
>> >> >
>> >> > Will providing full "test-patch" integration (pre-commit build and
>> >> > unit
>> >> > test
>> >> > triggered by Jira "Patch Available" state) satisfy your request for
>> >> > functionality #1 and #2?  Yes or no, please.
>> >> >
>> >> > Thanks,
>> >> > --Matt
>> >> >
>> >> >
>> >> > On Sat, Mar 2, 2013 at 7:32 PM, Konstantin Shvachko
>> >> > <sh...@gmail.com>
>> >> > wrote:
>> >> >>
>> >> >> Hi Matt,
>> >> >>
>> >> >> On Sat, Mar 2, 2013 at 12:32 PM, Matt Foley <mf...@hortonworks.com>
>> >> >> wrote:
>> >> >> > Konstantin,
>> >> >> > I would like to explore what it would take to remove this
>> >> >> > perceived
>> >> >> > impediment --
>> >> >>
>> >> >> Glad you decided to explore. Thank you.
>> >> >>
>> >> >> > although I reserve the right to argue that this is not
>> >> >> > pre-requisite to merging the cross-platform support patch.
>> >> >>
>> >> >> It's your right indeed. So as mine to question what the platform
>> >> >> support means for you, which I believe remained unclear.
>> >> >> I do not impede the change as you should have noticed. My
>> >> >> requirement
>> >> >> comes from my perception of the support, which means to me exactly
>> >> >> two
>> >> >> things:
>> >> >> 1. The ability to recognise the code is broken for the platform
>> >> >> 2. The ability to test new patches on the platform
>> >> >> The latter is problematic, as many noticed in this thread, for those
>> >> >> whose customary environment does not include Windows.
>> >> >>
>> >> >> > If we implemented full "test-patch" support for Windows on trunk,
>> >> >> > would
>> >> >> > that
>> >> >> > fulfill both your items #1 and #2?  Please note that:
>> >> >> > a) Pushing the "Patch Available" button in Jira shall cause a
>> >> >> > pre-commit
>> >> >> > build to start within, I believe, 20 minutes.
>> >> >> > b) That build keeps logs for both java build and unit tests for
>> >> >> > several
>> >> >> > days, that are accessible to all viewers.
>> >> >>
>> >> >> In item #1 I mostly asking for the nightly build, which is simpler
>> >> >> than "test-patch". The latter would be ideal from the platform
>> >> >> support
>> >> >> viewpoint, but it is for the community to decide if we want to add
>> >> >> extra +3 hours to the build.
>> >> >> Nightly build in my understanding is triggered by the timer rather
>> >> >> than by Jira's "submit patch" button.  On Jenkins build
>> >> >> configuration
>> >> >> you can specify it under "Build periodically".
>> >> >>
>> >> >> > So, does this provide sufficient on-demand support that we don't
>> >> >> > have
>> >> >> > to
>> >> >> > implement a whole new on-demand VM support structure of some sort
>> >> >> > for
>> >> >> > #2
>> >> >> > (which would be an extraordinary and impractical demand)?
>> >> >>
>> >> >> I did not mention VMs. Item #2 means a build, which runs
>> >> >> "test-patch"
>> >> >> target with the file specified by a user (instead of a jira
>> >> >> attachment).
>> >> >> When user clicks "Build Now" link a box is displayed where the user
>> >> >> can enter the file path containing the patch. This can be specified
>> >> >> in
>> >> >> the Build Configuration under "This build is parameterized" by
>> >> >> choosing AddParameter / FileParameter. The build can run on the same
>> >> >> Windows machine as the nightly build.
>> >> >> Such build will let people test their patches for Windows on Jenkins
>> >> >> if they don't posses a license for the right version of Windows.
>> >> >> I hope this will not turn into extraordinary or impractical effort.
>> >> >>
>> >> >> Thanks,
>> >> >> --Konst
>> >> >>
>> >> >> > Thanks,
>> >> >> > --Matt
>> >> >> >
>> >> >> >
>> >> >> > On Fri, Mar 1, 2013 at 12:18 PM, Konstantin Shvachko
>> >> >> > <sh...@gmail.com>
>> >> >> > wrote:
>> >> >> >>
>> >> >> >> -1
>> >> >> >> We should have a CI infrastructure in place before we can commit
>> >> >> >> to
>> >> >> >> supporting Windows platform.
>> >> >> >>
>> >> >> >> Eric is right Win/Cygwin was supported since day one.
>> >> >> >> I had a Windows box under my desk running nightly builds back in
>> >> >> >> 2006-07.
>> >> >> >> People were irritated but I was filing windows bugs until 0.22
>> >> >> >> release.
>> >> >> >> Times changing and I am glad to see wider support for Win
>> >> >> >> platform.
>> >> >> >>
>> >> >> >> But in order to make it work you guys need to put the CI process
>> >> >> >> in
>> >> >> >> place
>> >> >> >>
>> >> >> >> 1. windows jenkins build: could be nightly or PreCommit.
>> >> >> >> - Nightly would mean that changes can be committed to trunk based
>> >> >> >> on
>> >> >> >> linux PreCommit build. And people will file bugs if the change
>> >> >> >> broke
>> >> >> >> Windows nightly build.
>> >> >> >> - PreCommit-win build will mean automatic reporting failed tests
>> >> >> >> to
>> >> >> >> respective jira blocking commits the same way as it is now with
>> >> >> >> linux
>> >> >> >> PreCommit builds.
>> >> >> >> We should discuss which way is more efficient for developers.
>> >> >> >>
>> >> >> >> 2. On-demand-windows Jenkins build.
>> >> >> >> I see it as a build to which I can attach my patch and the build
>> >> >> >> will
>> >> >> >> run my changes on a dedicated windows box.
>> >> >> >> That way people can test their changes without having personal
>> >> >> >> windows
>> >> >> >> nodes.
>> >> >> >>
>> >> >> >> I think this is the minimal set of requirement for us to be able
>> >> >> >> to
>> >> >> >> commit to the new platform.
>> >> >> >> Right now I see only one windows related build
>> >> >> >> https://builds.apache.org/view/Hadoop/job/Hadoop-1-win/
>> >> >> >> Which was failing since Sept 8, 2012 and did not run in the last
>> >> >> >> month.
>> >> >> >>
>> >> >> >> Thanks,
>> >> >> >> --Konst
>> >> >> >>
>> >> >> >> On Thu, Feb 28, 2013 at 8:47 PM, Eric Baldeschwieler
>> >> >> >> <er...@hortonworks.com> wrote:
>> >> >> >> > +1 (non-binding)
>> >> >> >> >
>> >> >> >> > A few of observations:
>> >> >> >> >
>> >> >> >> > - Windows has actually been a supported platform for Hadoop
>> >> >> >> > since
>> >> >> >> > 0.1
>> >> >> >> > .
>> >> >> >> > Doug championed supporting windows then and we've continued to
>> >> >> >> > do
>> >> >> >> > it
>> >> >> >> > with
>> >> >> >> > varying vigor over time.  To my knowledge we've never made a
>> >> >> >> > decision
>> >> >> >> > to
>> >> >> >> > drop windows support.  The change here is improving our support
>> >> >> >> > and
>> >> >> >> > dropping
>> >> >> >> > the requirement of cigwin.  We had Nutch windows users on the
>> >> >> >> > list
>> >> >> >> > in
>> >> >> >> > 2006
>> >> >> >> > and we've been supporting windows FS requirements since
>> >> >> >> > inception.
>> >> >> >> >
>> >> >> >> > - A little pragmatism will go a long way.  As a community we've
>> >> >> >> > got
>> >> >> >> > to
>> >> >> >> > stay committed to keeping hadoop simple (so it does work on
>> >> >> >> > many
>> >> >> >> > platforms)
>> >> >> >> > and extending it to take advantage of key emerging OS/hardware
>> >> >> >> > features,
>> >> >> >> > such as containers, new FSs, virtualization, flash ...  We
>> >> >> >> > should
>> >> >> >> > all
>> >> >> >> > plan
>> >> >> >> > to let new features & optimizations emerge that don't work
>> >> >> >> > everywhere, if
>> >> >> >> > they are compelling and central to hadoop's mission of being
>> >> >> >> > THE
>> >> >> >> > best
>> >> >> >> > fabric
>> >> >> >> > for storing and processing big data.
>> >> >> >> >
>> >> >> >> > - A UI project like KDE has to deal with the MANY differences
>> >> >> >> > between
>> >> >> >> > windows and linux UI APIs.  Hadoop faces no such complex
>> >> >> >> > challenge
>> >> >> >> > and hence
>> >> >> >> > can be maintained from a single codeline IMO.  It is mostly
>> >> >> >> > abstracted from
>> >> >> >> > the OS APIs via Java and our design choices.  Where it is not
>> >> >> >> > we
>> >> >> >> > can
>> >> >> >> > continue to add plugable abstractions.
>> >> >> >> >
>> >> >> >
>> >> >> >
>> >> >
>> >> >
>> >
>> >
>
>

Re: [Vote] Merge branch-trunk-win to trunk

[Konstantin Shvachko <sh...@gmail.com>]

+1 on the merge.

I am glad we agreed.
Having Jira to track the CI effort is a good idea.

Thanks,
--Konstantin

On Mon, Mar 4, 2013 at 3:29 PM, Matt Foley <mf...@hortonworks.com> wrote:
> Thanks.  I agree Windows -1's in test-patch should not block commits.
>
> --Matt
>
>
>
> On Mon, Mar 4, 2013 at 2:30 PM, Konstantin Shvachko <sh...@gmail.com>
> wrote:
>>
>> On Mon, Mar 4, 2013 at 12:22 PM, Matt Foley <mf...@hortonworks.com>
>> wrote:
>> > Konstantine, you have voted -1, and stated some requirements before
>> > you'll
>> > withdraw that -1.  As I plan to do work to fulfill those requirements, I
>> > want to make sure that what I'm proposing will, in fact, satisfy you.
>> > That's why I'm asking, if we implement full "test-patch" integration for
>> > Windows, does it seem to you that that would provide adequate support?
>>
>> Yes.
>>
>> > I have learned not to presume that my interpretation is correct.  My
>> > interpretation of item #1 is that test-patch provides pre-commit build,
>> > so
>> > it would satisfy item #1.  But rather than assuming that I am
>> > interpreting
>> > it correctly, I simply want your agreement that it would, or if not,
>> > clarification why it won't.
>>
>> I agree it will satisfy my item #1.
>> I did not agree in my previous email, but I changed my mind based on
>> the latest discussion. I have to explain why now.
>> I was proposing nightly build because I did not want pre-commit build
>> for Windows block commits to Linux. But if people are fine just ignoring
>> -1s for the Windows part of the build it should be good.
>>
>> > Regarding item #2, it is also my interpretation that test-patch provides
>> > an
>> > on-demand (perhaps 20-minutes deferred) Jenkins build and unit test,
>> > with
>> > logs available to the developer, so it would satisfy item #2.  But
>> > rather
>> > than assuming that I am interpreting it correctly, I simply want your
>> > agreement that it would, or if not, clarification why it won't.
>>
>> It will satisfy my item #2 in the following way:
>> I can duplicate your pre-commit build for Windows and add an input
>> parameter, which would let people run the build on their patches
>> chosen from local machine rather than attaching them to Jiras.
>>
>> Thanks,
>> --Konstantin
>>
>> > In agile terms, you are the Owner of these requirements.  Please give me
>> > owner feedback as to whether my proposed work sounds like it will
>> > satisfy
>> > the requirements.
>> >
>> > Thank you,
>> > --Matt
>> >
>> >
>> > On Sun, Mar 3, 2013 at 12:16 PM, Konstantin Shvachko
>> > <sh...@gmail.com>
>> > wrote:
>> >>
>> >> Didn't I explain in details what I am asking for?
>> >>
>> >> Thanks,
>> >> --Konst
>> >>
>> >> On Sun, Mar 3, 2013 at 11:08 AM, Matt Foley <mf...@hortonworks.com>
>> >> wrote:
>> >> > Hi Konstantin,
>> >> > I'd like to point out two things:
>> >> > First, I already committed in this thread (email of Thu, Feb 28, 2013
>> >> > at
>> >> > 6:01 PM) to providing CI for Windows builds.  So please stop acting
>> >> > like
>> >> > I'm
>> >> > resisting this idea or something.
>> >> > Second, you didn't answer my question, you just kvetched about the
>> >> > phrasing.
>> >> > So I ask again:
>> >> >
>> >> > Will providing full "test-patch" integration (pre-commit build and
>> >> > unit
>> >> > test
>> >> > triggered by Jira "Patch Available" state) satisfy your request for
>> >> > functionality #1 and #2?  Yes or no, please.
>> >> >
>> >> > Thanks,
>> >> > --Matt
>> >> >
>> >> >
>> >> > On Sat, Mar 2, 2013 at 7:32 PM, Konstantin Shvachko
>> >> > <sh...@gmail.com>
>> >> > wrote:
>> >> >>
>> >> >> Hi Matt,
>> >> >>
>> >> >> On Sat, Mar 2, 2013 at 12:32 PM, Matt Foley <mf...@hortonworks.com>
>> >> >> wrote:
>> >> >> > Konstantin,
>> >> >> > I would like to explore what it would take to remove this
>> >> >> > perceived
>> >> >> > impediment --
>> >> >>
>> >> >> Glad you decided to explore. Thank you.
>> >> >>
>> >> >> > although I reserve the right to argue that this is not
>> >> >> > pre-requisite to merging the cross-platform support patch.
>> >> >>
>> >> >> It's your right indeed. So as mine to question what the platform
>> >> >> support means for you, which I believe remained unclear.
>> >> >> I do not impede the change as you should have noticed. My
>> >> >> requirement
>> >> >> comes from my perception of the support, which means to me exactly
>> >> >> two
>> >> >> things:
>> >> >> 1. The ability to recognise the code is broken for the platform
>> >> >> 2. The ability to test new patches on the platform
>> >> >> The latter is problematic, as many noticed in this thread, for those
>> >> >> whose customary environment does not include Windows.
>> >> >>
>> >> >> > If we implemented full "test-patch" support for Windows on trunk,
>> >> >> > would
>> >> >> > that
>> >> >> > fulfill both your items #1 and #2?  Please note that:
>> >> >> > a) Pushing the "Patch Available" button in Jira shall cause a
>> >> >> > pre-commit
>> >> >> > build to start within, I believe, 20 minutes.
>> >> >> > b) That build keeps logs for both java build and unit tests for
>> >> >> > several
>> >> >> > days, that are accessible to all viewers.
>> >> >>
>> >> >> In item #1 I mostly asking for the nightly build, which is simpler
>> >> >> than "test-patch". The latter would be ideal from the platform
>> >> >> support
>> >> >> viewpoint, but it is for the community to decide if we want to add
>> >> >> extra +3 hours to the build.
>> >> >> Nightly build in my understanding is triggered by the timer rather
>> >> >> than by Jira's "submit patch" button.  On Jenkins build
>> >> >> configuration
>> >> >> you can specify it under "Build periodically".
>> >> >>
>> >> >> > So, does this provide sufficient on-demand support that we don't
>> >> >> > have
>> >> >> > to
>> >> >> > implement a whole new on-demand VM support structure of some sort
>> >> >> > for
>> >> >> > #2
>> >> >> > (which would be an extraordinary and impractical demand)?
>> >> >>
>> >> >> I did not mention VMs. Item #2 means a build, which runs
>> >> >> "test-patch"
>> >> >> target with the file specified by a user (instead of a jira
>> >> >> attachment).
>> >> >> When user clicks "Build Now" link a box is displayed where the user
>> >> >> can enter the file path containing the patch. This can be specified
>> >> >> in
>> >> >> the Build Configuration under "This build is parameterized" by
>> >> >> choosing AddParameter / FileParameter. The build can run on the same
>> >> >> Windows machine as the nightly build.
>> >> >> Such build will let people test their patches for Windows on Jenkins
>> >> >> if they don't posses a license for the right version of Windows.
>> >> >> I hope this will not turn into extraordinary or impractical effort.
>> >> >>
>> >> >> Thanks,
>> >> >> --Konst
>> >> >>
>> >> >> > Thanks,
>> >> >> > --Matt
>> >> >> >
>> >> >> >
>> >> >> > On Fri, Mar 1, 2013 at 12:18 PM, Konstantin Shvachko
>> >> >> > <sh...@gmail.com>
>> >> >> > wrote:
>> >> >> >>
>> >> >> >> -1
>> >> >> >> We should have a CI infrastructure in place before we can commit
>> >> >> >> to
>> >> >> >> supporting Windows platform.
>> >> >> >>
>> >> >> >> Eric is right Win/Cygwin was supported since day one.
>> >> >> >> I had a Windows box under my desk running nightly builds back in
>> >> >> >> 2006-07.
>> >> >> >> People were irritated but I was filing windows bugs until 0.22
>> >> >> >> release.
>> >> >> >> Times changing and I am glad to see wider support for Win
>> >> >> >> platform.
>> >> >> >>
>> >> >> >> But in order to make it work you guys need to put the CI process
>> >> >> >> in
>> >> >> >> place
>> >> >> >>
>> >> >> >> 1. windows jenkins build: could be nightly or PreCommit.
>> >> >> >> - Nightly would mean that changes can be committed to trunk based
>> >> >> >> on
>> >> >> >> linux PreCommit build. And people will file bugs if the change
>> >> >> >> broke
>> >> >> >> Windows nightly build.
>> >> >> >> - PreCommit-win build will mean automatic reporting failed tests
>> >> >> >> to
>> >> >> >> respective jira blocking commits the same way as it is now with
>> >> >> >> linux
>> >> >> >> PreCommit builds.
>> >> >> >> We should discuss which way is more efficient for developers.
>> >> >> >>
>> >> >> >> 2. On-demand-windows Jenkins build.
>> >> >> >> I see it as a build to which I can attach my patch and the build
>> >> >> >> will
>> >> >> >> run my changes on a dedicated windows box.
>> >> >> >> That way people can test their changes without having personal
>> >> >> >> windows
>> >> >> >> nodes.
>> >> >> >>
>> >> >> >> I think this is the minimal set of requirement for us to be able
>> >> >> >> to
>> >> >> >> commit to the new platform.
>> >> >> >> Right now I see only one windows related build
>> >> >> >> https://builds.apache.org/view/Hadoop/job/Hadoop-1-win/
>> >> >> >> Which was failing since Sept 8, 2012 and did not run in the last
>> >> >> >> month.
>> >> >> >>
>> >> >> >> Thanks,
>> >> >> >> --Konst
>> >> >> >>
>> >> >> >> On Thu, Feb 28, 2013 at 8:47 PM, Eric Baldeschwieler
>> >> >> >> <er...@hortonworks.com> wrote:
>> >> >> >> > +1 (non-binding)
>> >> >> >> >
>> >> >> >> > A few of observations:
>> >> >> >> >
>> >> >> >> > - Windows has actually been a supported platform for Hadoop
>> >> >> >> > since
>> >> >> >> > 0.1
>> >> >> >> > .
>> >> >> >> > Doug championed supporting windows then and we've continued to
>> >> >> >> > do
>> >> >> >> > it
>> >> >> >> > with
>> >> >> >> > varying vigor over time.  To my knowledge we've never made a
>> >> >> >> > decision
>> >> >> >> > to
>> >> >> >> > drop windows support.  The change here is improving our support
>> >> >> >> > and
>> >> >> >> > dropping
>> >> >> >> > the requirement of cigwin.  We had Nutch windows users on the
>> >> >> >> > list
>> >> >> >> > in
>> >> >> >> > 2006
>> >> >> >> > and we've been supporting windows FS requirements since
>> >> >> >> > inception.
>> >> >> >> >
>> >> >> >> > - A little pragmatism will go a long way.  As a community we've
>> >> >> >> > got
>> >> >> >> > to
>> >> >> >> > stay committed to keeping hadoop simple (so it does work on
>> >> >> >> > many
>> >> >> >> > platforms)
>> >> >> >> > and extending it to take advantage of key emerging OS/hardware
>> >> >> >> > features,
>> >> >> >> > such as containers, new FSs, virtualization, flash ...  We
>> >> >> >> > should
>> >> >> >> > all
>> >> >> >> > plan
>> >> >> >> > to let new features & optimizations emerge that don't work
>> >> >> >> > everywhere, if
>> >> >> >> > they are compelling and central to hadoop's mission of being
>> >> >> >> > THE
>> >> >> >> > best
>> >> >> >> > fabric
>> >> >> >> > for storing and processing big data.
>> >> >> >> >
>> >> >> >> > - A UI project like KDE has to deal with the MANY differences
>> >> >> >> > between
>> >> >> >> > windows and linux UI APIs.  Hadoop faces no such complex
>> >> >> >> > challenge
>> >> >> >> > and hence
>> >> >> >> > can be maintained from a single codeline IMO.  It is mostly
>> >> >> >> > abstracted from
>> >> >> >> > the OS APIs via Java and our design choices.  Where it is not
>> >> >> >> > we
>> >> >> >> > can
>> >> >> >> > continue to add plugable abstractions.
>> >> >> >> >
>> >> >> >
>> >> >> >
>> >> >
>> >> >
>> >
>> >
>
>

Re: [Vote] Merge branch-trunk-win to trunk

[Matt Foley <mf...@hortonworks.com>]

Thanks.  I agree Windows -1's in test-patch should not block commits.

--Matt



On Mon, Mar 4, 2013 at 2:30 PM, Konstantin Shvachko <sh...@gmail.com>wrote:

> On Mon, Mar 4, 2013 at 12:22 PM, Matt Foley <mf...@hortonworks.com>
> wrote:
> > Konstantine, you have voted -1, and stated some requirements before
> you'll
> > withdraw that -1.  As I plan to do work to fulfill those requirements, I
> > want to make sure that what I'm proposing will, in fact, satisfy you.
> > That's why I'm asking, if we implement full "test-patch" integration for
> > Windows, does it seem to you that that would provide adequate support?
>
> Yes.
>
> > I have learned not to presume that my interpretation is correct.  My
> > interpretation of item #1 is that test-patch provides pre-commit build,
> so
> > it would satisfy item #1.  But rather than assuming that I am
> interpreting
> > it correctly, I simply want your agreement that it would, or if not,
> > clarification why it won't.
>
> I agree it will satisfy my item #1.
> I did not agree in my previous email, but I changed my mind based on
> the latest discussion. I have to explain why now.
> I was proposing nightly build because I did not want pre-commit build
> for Windows block commits to Linux. But if people are fine just ignoring
> -1s for the Windows part of the build it should be good.
>
> > Regarding item #2, it is also my interpretation that test-patch provides
> an
> > on-demand (perhaps 20-minutes deferred) Jenkins build and unit test, with
> > logs available to the developer, so it would satisfy item #2.  But rather
> > than assuming that I am interpreting it correctly, I simply want your
> > agreement that it would, or if not, clarification why it won't.
>
> It will satisfy my item #2 in the following way:
> I can duplicate your pre-commit build for Windows and add an input
> parameter, which would let people run the build on their patches
> chosen from local machine rather than attaching them to Jiras.
>
> Thanks,
> --Konstantin
>
> > In agile terms, you are the Owner of these requirements.  Please give me
> > owner feedback as to whether my proposed work sounds like it will satisfy
> > the requirements.
> >
> > Thank you,
> > --Matt
> >
> >
> > On Sun, Mar 3, 2013 at 12:16 PM, Konstantin Shvachko <
> shv.hadoop@gmail.com>
> > wrote:
> >>
> >> Didn't I explain in details what I am asking for?
> >>
> >> Thanks,
> >> --Konst
> >>
> >> On Sun, Mar 3, 2013 at 11:08 AM, Matt Foley <mf...@hortonworks.com>
> >> wrote:
> >> > Hi Konstantin,
> >> > I'd like to point out two things:
> >> > First, I already committed in this thread (email of Thu, Feb 28, 2013
> at
> >> > 6:01 PM) to providing CI for Windows builds.  So please stop acting
> like
> >> > I'm
> >> > resisting this idea or something.
> >> > Second, you didn't answer my question, you just kvetched about the
> >> > phrasing.
> >> > So I ask again:
> >> >
> >> > Will providing full "test-patch" integration (pre-commit build and
> unit
> >> > test
> >> > triggered by Jira "Patch Available" state) satisfy your request for
> >> > functionality #1 and #2?  Yes or no, please.
> >> >
> >> > Thanks,
> >> > --Matt
> >> >
> >> >
> >> > On Sat, Mar 2, 2013 at 7:32 PM, Konstantin Shvachko
> >> > <sh...@gmail.com>
> >> > wrote:
> >> >>
> >> >> Hi Matt,
> >> >>
> >> >> On Sat, Mar 2, 2013 at 12:32 PM, Matt Foley <mf...@hortonworks.com>
> >> >> wrote:
> >> >> > Konstantin,
> >> >> > I would like to explore what it would take to remove this perceived
> >> >> > impediment --
> >> >>
> >> >> Glad you decided to explore. Thank you.
> >> >>
> >> >> > although I reserve the right to argue that this is not
> >> >> > pre-requisite to merging the cross-platform support patch.
> >> >>
> >> >> It's your right indeed. So as mine to question what the platform
> >> >> support means for you, which I believe remained unclear.
> >> >> I do not impede the change as you should have noticed. My requirement
> >> >> comes from my perception of the support, which means to me exactly
> two
> >> >> things:
> >> >> 1. The ability to recognise the code is broken for the platform
> >> >> 2. The ability to test new patches on the platform
> >> >> The latter is problematic, as many noticed in this thread, for those
> >> >> whose customary environment does not include Windows.
> >> >>
> >> >> > If we implemented full "test-patch" support for Windows on trunk,
> >> >> > would
> >> >> > that
> >> >> > fulfill both your items #1 and #2?  Please note that:
> >> >> > a) Pushing the "Patch Available" button in Jira shall cause a
> >> >> > pre-commit
> >> >> > build to start within, I believe, 20 minutes.
> >> >> > b) That build keeps logs for both java build and unit tests for
> >> >> > several
> >> >> > days, that are accessible to all viewers.
> >> >>
> >> >> In item #1 I mostly asking for the nightly build, which is simpler
> >> >> than "test-patch". The latter would be ideal from the platform
> support
> >> >> viewpoint, but it is for the community to decide if we want to add
> >> >> extra +3 hours to the build.
> >> >> Nightly build in my understanding is triggered by the timer rather
> >> >> than by Jira's "submit patch" button.  On Jenkins build configuration
> >> >> you can specify it under "Build periodically".
> >> >>
> >> >> > So, does this provide sufficient on-demand support that we don't
> have
> >> >> > to
> >> >> > implement a whole new on-demand VM support structure of some sort
> for
> >> >> > #2
> >> >> > (which would be an extraordinary and impractical demand)?
> >> >>
> >> >> I did not mention VMs. Item #2 means a build, which runs "test-patch"
> >> >> target with the file specified by a user (instead of a jira
> >> >> attachment).
> >> >> When user clicks "Build Now" link a box is displayed where the user
> >> >> can enter the file path containing the patch. This can be specified
> in
> >> >> the Build Configuration under "This build is parameterized" by
> >> >> choosing AddParameter / FileParameter. The build can run on the same
> >> >> Windows machine as the nightly build.
> >> >> Such build will let people test their patches for Windows on Jenkins
> >> >> if they don't posses a license for the right version of Windows.
> >> >> I hope this will not turn into extraordinary or impractical effort.
> >> >>
> >> >> Thanks,
> >> >> --Konst
> >> >>
> >> >> > Thanks,
> >> >> > --Matt
> >> >> >
> >> >> >
> >> >> > On Fri, Mar 1, 2013 at 12:18 PM, Konstantin Shvachko
> >> >> > <sh...@gmail.com>
> >> >> > wrote:
> >> >> >>
> >> >> >> -1
> >> >> >> We should have a CI infrastructure in place before we can commit
> to
> >> >> >> supporting Windows platform.
> >> >> >>
> >> >> >> Eric is right Win/Cygwin was supported since day one.
> >> >> >> I had a Windows box under my desk running nightly builds back in
> >> >> >> 2006-07.
> >> >> >> People were irritated but I was filing windows bugs until 0.22
> >> >> >> release.
> >> >> >> Times changing and I am glad to see wider support for Win
> platform.
> >> >> >>
> >> >> >> But in order to make it work you guys need to put the CI process
> in
> >> >> >> place
> >> >> >>
> >> >> >> 1. windows jenkins build: could be nightly or PreCommit.
> >> >> >> - Nightly would mean that changes can be committed to trunk based
> on
> >> >> >> linux PreCommit build. And people will file bugs if the change
> broke
> >> >> >> Windows nightly build.
> >> >> >> - PreCommit-win build will mean automatic reporting failed tests
> to
> >> >> >> respective jira blocking commits the same way as it is now with
> >> >> >> linux
> >> >> >> PreCommit builds.
> >> >> >> We should discuss which way is more efficient for developers.
> >> >> >>
> >> >> >> 2. On-demand-windows Jenkins build.
> >> >> >> I see it as a build to which I can attach my patch and the build
> >> >> >> will
> >> >> >> run my changes on a dedicated windows box.
> >> >> >> That way people can test their changes without having personal
> >> >> >> windows
> >> >> >> nodes.
> >> >> >>
> >> >> >> I think this is the minimal set of requirement for us to be able
> to
> >> >> >> commit to the new platform.
> >> >> >> Right now I see only one windows related build
> >> >> >> https://builds.apache.org/view/Hadoop/job/Hadoop-1-win/
> >> >> >> Which was failing since Sept 8, 2012 and did not run in the last
> >> >> >> month.
> >> >> >>
> >> >> >> Thanks,
> >> >> >> --Konst
> >> >> >>
> >> >> >> On Thu, Feb 28, 2013 at 8:47 PM, Eric Baldeschwieler
> >> >> >> <er...@hortonworks.com> wrote:
> >> >> >> > +1 (non-binding)
> >> >> >> >
> >> >> >> > A few of observations:
> >> >> >> >
> >> >> >> > - Windows has actually been a supported platform for Hadoop
> since
> >> >> >> > 0.1
> >> >> >> > .
> >> >> >> > Doug championed supporting windows then and we've continued to
> do
> >> >> >> > it
> >> >> >> > with
> >> >> >> > varying vigor over time.  To my knowledge we've never made a
> >> >> >> > decision
> >> >> >> > to
> >> >> >> > drop windows support.  The change here is improving our support
> >> >> >> > and
> >> >> >> > dropping
> >> >> >> > the requirement of cigwin.  We had Nutch windows users on the
> list
> >> >> >> > in
> >> >> >> > 2006
> >> >> >> > and we've been supporting windows FS requirements since
> inception.
> >> >> >> >
> >> >> >> > - A little pragmatism will go a long way.  As a community we've
> >> >> >> > got
> >> >> >> > to
> >> >> >> > stay committed to keeping hadoop simple (so it does work on many
> >> >> >> > platforms)
> >> >> >> > and extending it to take advantage of key emerging OS/hardware
> >> >> >> > features,
> >> >> >> > such as containers, new FSs, virtualization, flash ...  We
> should
> >> >> >> > all
> >> >> >> > plan
> >> >> >> > to let new features & optimizations emerge that don't work
> >> >> >> > everywhere, if
> >> >> >> > they are compelling and central to hadoop's mission of being THE
> >> >> >> > best
> >> >> >> > fabric
> >> >> >> > for storing and processing big data.
> >> >> >> >
> >> >> >> > - A UI project like KDE has to deal with the MANY differences
> >> >> >> > between
> >> >> >> > windows and linux UI APIs.  Hadoop faces no such complex
> challenge
> >> >> >> > and hence
> >> >> >> > can be maintained from a single codeline IMO.  It is mostly
> >> >> >> > abstracted from
> >> >> >> > the OS APIs via Java and our design choices.  Where it is not we
> >> >> >> > can
> >> >> >> > continue to add plugable abstractions.
> >> >> >> >
> >> >> >
> >> >> >
> >> >
> >> >
> >
> >
>

Re: [Vote] Merge branch-trunk-win to trunk

[Matt Foley <mf...@hortonworks.com>]

Thanks.  I agree Windows -1's in test-patch should not block commits.

--Matt



On Mon, Mar 4, 2013 at 2:30 PM, Konstantin Shvachko <sh...@gmail.com>wrote:

> On Mon, Mar 4, 2013 at 12:22 PM, Matt Foley <mf...@hortonworks.com>
> wrote:
> > Konstantine, you have voted -1, and stated some requirements before
> you'll
> > withdraw that -1.  As I plan to do work to fulfill those requirements, I
> > want to make sure that what I'm proposing will, in fact, satisfy you.
> > That's why I'm asking, if we implement full "test-patch" integration for
> > Windows, does it seem to you that that would provide adequate support?
>
> Yes.
>
> > I have learned not to presume that my interpretation is correct.  My
> > interpretation of item #1 is that test-patch provides pre-commit build,
> so
> > it would satisfy item #1.  But rather than assuming that I am
> interpreting
> > it correctly, I simply want your agreement that it would, or if not,
> > clarification why it won't.
>
> I agree it will satisfy my item #1.
> I did not agree in my previous email, but I changed my mind based on
> the latest discussion. I have to explain why now.
> I was proposing nightly build because I did not want pre-commit build
> for Windows block commits to Linux. But if people are fine just ignoring
> -1s for the Windows part of the build it should be good.
>
> > Regarding item #2, it is also my interpretation that test-patch provides
> an
> > on-demand (perhaps 20-minutes deferred) Jenkins build and unit test, with
> > logs available to the developer, so it would satisfy item #2.  But rather
> > than assuming that I am interpreting it correctly, I simply want your
> > agreement that it would, or if not, clarification why it won't.
>
> It will satisfy my item #2 in the following way:
> I can duplicate your pre-commit build for Windows and add an input
> parameter, which would let people run the build on their patches
> chosen from local machine rather than attaching them to Jiras.
>
> Thanks,
> --Konstantin
>
> > In agile terms, you are the Owner of these requirements.  Please give me
> > owner feedback as to whether my proposed work sounds like it will satisfy
> > the requirements.
> >
> > Thank you,
> > --Matt
> >
> >
> > On Sun, Mar 3, 2013 at 12:16 PM, Konstantin Shvachko <
> shv.hadoop@gmail.com>
> > wrote:
> >>
> >> Didn't I explain in details what I am asking for?
> >>
> >> Thanks,
> >> --Konst
> >>
> >> On Sun, Mar 3, 2013 at 11:08 AM, Matt Foley <mf...@hortonworks.com>
> >> wrote:
> >> > Hi Konstantin,
> >> > I'd like to point out two things:
> >> > First, I already committed in this thread (email of Thu, Feb 28, 2013
> at
> >> > 6:01 PM) to providing CI for Windows builds.  So please stop acting
> like
> >> > I'm
> >> > resisting this idea or something.
> >> > Second, you didn't answer my question, you just kvetched about the
> >> > phrasing.
> >> > So I ask again:
> >> >
> >> > Will providing full "test-patch" integration (pre-commit build and
> unit
> >> > test
> >> > triggered by Jira "Patch Available" state) satisfy your request for
> >> > functionality #1 and #2?  Yes or no, please.
> >> >
> >> > Thanks,
> >> > --Matt
> >> >
> >> >
> >> > On Sat, Mar 2, 2013 at 7:32 PM, Konstantin Shvachko
> >> > <sh...@gmail.com>
> >> > wrote:
> >> >>
> >> >> Hi Matt,
> >> >>
> >> >> On Sat, Mar 2, 2013 at 12:32 PM, Matt Foley <mf...@hortonworks.com>
> >> >> wrote:
> >> >> > Konstantin,
> >> >> > I would like to explore what it would take to remove this perceived
> >> >> > impediment --
> >> >>
> >> >> Glad you decided to explore. Thank you.
> >> >>
> >> >> > although I reserve the right to argue that this is not
> >> >> > pre-requisite to merging the cross-platform support patch.
> >> >>
> >> >> It's your right indeed. So as mine to question what the platform
> >> >> support means for you, which I believe remained unclear.
> >> >> I do not impede the change as you should have noticed. My requirement
> >> >> comes from my perception of the support, which means to me exactly
> two
> >> >> things:
> >> >> 1. The ability to recognise the code is broken for the platform
> >> >> 2. The ability to test new patches on the platform
> >> >> The latter is problematic, as many noticed in this thread, for those
> >> >> whose customary environment does not include Windows.
> >> >>
> >> >> > If we implemented full "test-patch" support for Windows on trunk,
> >> >> > would
> >> >> > that
> >> >> > fulfill both your items #1 and #2?  Please note that:
> >> >> > a) Pushing the "Patch Available" button in Jira shall cause a
> >> >> > pre-commit
> >> >> > build to start within, I believe, 20 minutes.
> >> >> > b) That build keeps logs for both java build and unit tests for
> >> >> > several
> >> >> > days, that are accessible to all viewers.
> >> >>
> >> >> In item #1 I mostly asking for the nightly build, which is simpler
> >> >> than "test-patch". The latter would be ideal from the platform
> support
> >> >> viewpoint, but it is for the community to decide if we want to add
> >> >> extra +3 hours to the build.
> >> >> Nightly build in my understanding is triggered by the timer rather
> >> >> than by Jira's "submit patch" button.  On Jenkins build configuration
> >> >> you can specify it under "Build periodically".
> >> >>
> >> >> > So, does this provide sufficient on-demand support that we don't
> have
> >> >> > to
> >> >> > implement a whole new on-demand VM support structure of some sort
> for
> >> >> > #2
> >> >> > (which would be an extraordinary and impractical demand)?
> >> >>
> >> >> I did not mention VMs. Item #2 means a build, which runs "test-patch"
> >> >> target with the file specified by a user (instead of a jira
> >> >> attachment).
> >> >> When user clicks "Build Now" link a box is displayed where the user
> >> >> can enter the file path containing the patch. This can be specified
> in
> >> >> the Build Configuration under "This build is parameterized" by
> >> >> choosing AddParameter / FileParameter. The build can run on the same
> >> >> Windows machine as the nightly build.
> >> >> Such build will let people test their patches for Windows on Jenkins
> >> >> if they don't posses a license for the right version of Windows.
> >> >> I hope this will not turn into extraordinary or impractical effort.
> >> >>
> >> >> Thanks,
> >> >> --Konst
> >> >>
> >> >> > Thanks,
> >> >> > --Matt
> >> >> >
> >> >> >
> >> >> > On Fri, Mar 1, 2013 at 12:18 PM, Konstantin Shvachko
> >> >> > <sh...@gmail.com>
> >> >> > wrote:
> >> >> >>
> >> >> >> -1
> >> >> >> We should have a CI infrastructure in place before we can commit
> to
> >> >> >> supporting Windows platform.
> >> >> >>
> >> >> >> Eric is right Win/Cygwin was supported since day one.
> >> >> >> I had a Windows box under my desk running nightly builds back in
> >> >> >> 2006-07.
> >> >> >> People were irritated but I was filing windows bugs until 0.22
> >> >> >> release.
> >> >> >> Times changing and I am glad to see wider support for Win
> platform.
> >> >> >>
> >> >> >> But in order to make it work you guys need to put the CI process
> in
> >> >> >> place
> >> >> >>
> >> >> >> 1. windows jenkins build: could be nightly or PreCommit.
> >> >> >> - Nightly would mean that changes can be committed to trunk based
> on
> >> >> >> linux PreCommit build. And people will file bugs if the change
> broke
> >> >> >> Windows nightly build.
> >> >> >> - PreCommit-win build will mean automatic reporting failed tests
> to
> >> >> >> respective jira blocking commits the same way as it is now with
> >> >> >> linux
> >> >> >> PreCommit builds.
> >> >> >> We should discuss which way is more efficient for developers.
> >> >> >>
> >> >> >> 2. On-demand-windows Jenkins build.
> >> >> >> I see it as a build to which I can attach my patch and the build
> >> >> >> will
> >> >> >> run my changes on a dedicated windows box.
> >> >> >> That way people can test their changes without having personal
> >> >> >> windows
> >> >> >> nodes.
> >> >> >>
> >> >> >> I think this is the minimal set of requirement for us to be able
> to
> >> >> >> commit to the new platform.
> >> >> >> Right now I see only one windows related build
> >> >> >> https://builds.apache.org/view/Hadoop/job/Hadoop-1-win/
> >> >> >> Which was failing since Sept 8, 2012 and did not run in the last
> >> >> >> month.
> >> >> >>
> >> >> >> Thanks,
> >> >> >> --Konst
> >> >> >>
> >> >> >> On Thu, Feb 28, 2013 at 8:47 PM, Eric Baldeschwieler
> >> >> >> <er...@hortonworks.com> wrote:
> >> >> >> > +1 (non-binding)
> >> >> >> >
> >> >> >> > A few of observations:
> >> >> >> >
> >> >> >> > - Windows has actually been a supported platform for Hadoop
> since
> >> >> >> > 0.1
> >> >> >> > .
> >> >> >> > Doug championed supporting windows then and we've continued to
> do
> >> >> >> > it
> >> >> >> > with
> >> >> >> > varying vigor over time.  To my knowledge we've never made a
> >> >> >> > decision
> >> >> >> > to
> >> >> >> > drop windows support.  The change here is improving our support
> >> >> >> > and
> >> >> >> > dropping
> >> >> >> > the requirement of cigwin.  We had Nutch windows users on the
> list
> >> >> >> > in
> >> >> >> > 2006
> >> >> >> > and we've been supporting windows FS requirements since
> inception.
> >> >> >> >
> >> >> >> > - A little pragmatism will go a long way.  As a community we've
> >> >> >> > got
> >> >> >> > to
> >> >> >> > stay committed to keeping hadoop simple (so it does work on many
> >> >> >> > platforms)
> >> >> >> > and extending it to take advantage of key emerging OS/hardware
> >> >> >> > features,
> >> >> >> > such as containers, new FSs, virtualization, flash ...  We
> should
> >> >> >> > all
> >> >> >> > plan
> >> >> >> > to let new features & optimizations emerge that don't work
> >> >> >> > everywhere, if
> >> >> >> > they are compelling and central to hadoop's mission of being THE
> >> >> >> > best
> >> >> >> > fabric
> >> >> >> > for storing and processing big data.
> >> >> >> >
> >> >> >> > - A UI project like KDE has to deal with the MANY differences
> >> >> >> > between
> >> >> >> > windows and linux UI APIs.  Hadoop faces no such complex
> challenge
> >> >> >> > and hence
> >> >> >> > can be maintained from a single codeline IMO.  It is mostly
> >> >> >> > abstracted from
> >> >> >> > the OS APIs via Java and our design choices.  Where it is not we
> >> >> >> > can
> >> >> >> > continue to add plugable abstractions.
> >> >> >> >
> >> >> >
> >> >> >
> >> >
> >> >
> >
> >
>

Re: [Vote] Merge branch-trunk-win to trunk

[Matt Foley <mf...@hortonworks.com>]

Thanks.  I agree Windows -1's in test-patch should not block commits.

--Matt



On Mon, Mar 4, 2013 at 2:30 PM, Konstantin Shvachko <sh...@gmail.com>wrote:

> On Mon, Mar 4, 2013 at 12:22 PM, Matt Foley <mf...@hortonworks.com>
> wrote:
> > Konstantine, you have voted -1, and stated some requirements before
> you'll
> > withdraw that -1.  As I plan to do work to fulfill those requirements, I
> > want to make sure that what I'm proposing will, in fact, satisfy you.
> > That's why I'm asking, if we implement full "test-patch" integration for
> > Windows, does it seem to you that that would provide adequate support?
>
> Yes.
>
> > I have learned not to presume that my interpretation is correct.  My
> > interpretation of item #1 is that test-patch provides pre-commit build,
> so
> > it would satisfy item #1.  But rather than assuming that I am
> interpreting
> > it correctly, I simply want your agreement that it would, or if not,
> > clarification why it won't.
>
> I agree it will satisfy my item #1.
> I did not agree in my previous email, but I changed my mind based on
> the latest discussion. I have to explain why now.
> I was proposing nightly build because I did not want pre-commit build
> for Windows block commits to Linux. But if people are fine just ignoring
> -1s for the Windows part of the build it should be good.
>
> > Regarding item #2, it is also my interpretation that test-patch provides
> an
> > on-demand (perhaps 20-minutes deferred) Jenkins build and unit test, with
> > logs available to the developer, so it would satisfy item #2.  But rather
> > than assuming that I am interpreting it correctly, I simply want your
> > agreement that it would, or if not, clarification why it won't.
>
> It will satisfy my item #2 in the following way:
> I can duplicate your pre-commit build for Windows and add an input
> parameter, which would let people run the build on their patches
> chosen from local machine rather than attaching them to Jiras.
>
> Thanks,
> --Konstantin
>
> > In agile terms, you are the Owner of these requirements.  Please give me
> > owner feedback as to whether my proposed work sounds like it will satisfy
> > the requirements.
> >
> > Thank you,
> > --Matt
> >
> >
> > On Sun, Mar 3, 2013 at 12:16 PM, Konstantin Shvachko <
> shv.hadoop@gmail.com>
> > wrote:
> >>
> >> Didn't I explain in details what I am asking for?
> >>
> >> Thanks,
> >> --Konst
> >>
> >> On Sun, Mar 3, 2013 at 11:08 AM, Matt Foley <mf...@hortonworks.com>
> >> wrote:
> >> > Hi Konstantin,
> >> > I'd like to point out two things:
> >> > First, I already committed in this thread (email of Thu, Feb 28, 2013
> at
> >> > 6:01 PM) to providing CI for Windows builds.  So please stop acting
> like
> >> > I'm
> >> > resisting this idea or something.
> >> > Second, you didn't answer my question, you just kvetched about the
> >> > phrasing.
> >> > So I ask again:
> >> >
> >> > Will providing full "test-patch" integration (pre-commit build and
> unit
> >> > test
> >> > triggered by Jira "Patch Available" state) satisfy your request for
> >> > functionality #1 and #2?  Yes or no, please.
> >> >
> >> > Thanks,
> >> > --Matt
> >> >
> >> >
> >> > On Sat, Mar 2, 2013 at 7:32 PM, Konstantin Shvachko
> >> > <sh...@gmail.com>
> >> > wrote:
> >> >>
> >> >> Hi Matt,
> >> >>
> >> >> On Sat, Mar 2, 2013 at 12:32 PM, Matt Foley <mf...@hortonworks.com>
> >> >> wrote:
> >> >> > Konstantin,
> >> >> > I would like to explore what it would take to remove this perceived
> >> >> > impediment --
> >> >>
> >> >> Glad you decided to explore. Thank you.
> >> >>
> >> >> > although I reserve the right to argue that this is not
> >> >> > pre-requisite to merging the cross-platform support patch.
> >> >>
> >> >> It's your right indeed. So as mine to question what the platform
> >> >> support means for you, which I believe remained unclear.
> >> >> I do not impede the change as you should have noticed. My requirement
> >> >> comes from my perception of the support, which means to me exactly
> two
> >> >> things:
> >> >> 1. The ability to recognise the code is broken for the platform
> >> >> 2. The ability to test new patches on the platform
> >> >> The latter is problematic, as many noticed in this thread, for those
> >> >> whose customary environment does not include Windows.
> >> >>
> >> >> > If we implemented full "test-patch" support for Windows on trunk,
> >> >> > would
> >> >> > that
> >> >> > fulfill both your items #1 and #2?  Please note that:
> >> >> > a) Pushing the "Patch Available" button in Jira shall cause a
> >> >> > pre-commit
> >> >> > build to start within, I believe, 20 minutes.
> >> >> > b) That build keeps logs for both java build and unit tests for
> >> >> > several
> >> >> > days, that are accessible to all viewers.
> >> >>
> >> >> In item #1 I mostly asking for the nightly build, which is simpler
> >> >> than "test-patch". The latter would be ideal from the platform
> support
> >> >> viewpoint, but it is for the community to decide if we want to add
> >> >> extra +3 hours to the build.
> >> >> Nightly build in my understanding is triggered by the timer rather
> >> >> than by Jira's "submit patch" button.  On Jenkins build configuration
> >> >> you can specify it under "Build periodically".
> >> >>
> >> >> > So, does this provide sufficient on-demand support that we don't
> have
> >> >> > to
> >> >> > implement a whole new on-demand VM support structure of some sort
> for
> >> >> > #2
> >> >> > (which would be an extraordinary and impractical demand)?
> >> >>
> >> >> I did not mention VMs. Item #2 means a build, which runs "test-patch"
> >> >> target with the file specified by a user (instead of a jira
> >> >> attachment).
> >> >> When user clicks "Build Now" link a box is displayed where the user
> >> >> can enter the file path containing the patch. This can be specified
> in
> >> >> the Build Configuration under "This build is parameterized" by
> >> >> choosing AddParameter / FileParameter. The build can run on the same
> >> >> Windows machine as the nightly build.
> >> >> Such build will let people test their patches for Windows on Jenkins
> >> >> if they don't posses a license for the right version of Windows.
> >> >> I hope this will not turn into extraordinary or impractical effort.
> >> >>
> >> >> Thanks,
> >> >> --Konst
> >> >>
> >> >> > Thanks,
> >> >> > --Matt
> >> >> >
> >> >> >
> >> >> > On Fri, Mar 1, 2013 at 12:18 PM, Konstantin Shvachko
> >> >> > <sh...@gmail.com>
> >> >> > wrote:
> >> >> >>
> >> >> >> -1
> >> >> >> We should have a CI infrastructure in place before we can commit
> to
> >> >> >> supporting Windows platform.
> >> >> >>
> >> >> >> Eric is right Win/Cygwin was supported since day one.
> >> >> >> I had a Windows box under my desk running nightly builds back in
> >> >> >> 2006-07.
> >> >> >> People were irritated but I was filing windows bugs until 0.22
> >> >> >> release.
> >> >> >> Times changing and I am glad to see wider support for Win
> platform.
> >> >> >>
> >> >> >> But in order to make it work you guys need to put the CI process
> in
> >> >> >> place
> >> >> >>
> >> >> >> 1. windows jenkins build: could be nightly or PreCommit.
> >> >> >> - Nightly would mean that changes can be committed to trunk based
> on
> >> >> >> linux PreCommit build. And people will file bugs if the change
> broke
> >> >> >> Windows nightly build.
> >> >> >> - PreCommit-win build will mean automatic reporting failed tests
> to
> >> >> >> respective jira blocking commits the same way as it is now with
> >> >> >> linux
> >> >> >> PreCommit builds.
> >> >> >> We should discuss which way is more efficient for developers.
> >> >> >>
> >> >> >> 2. On-demand-windows Jenkins build.
> >> >> >> I see it as a build to which I can attach my patch and the build
> >> >> >> will
> >> >> >> run my changes on a dedicated windows box.
> >> >> >> That way people can test their changes without having personal
> >> >> >> windows
> >> >> >> nodes.
> >> >> >>
> >> >> >> I think this is the minimal set of requirement for us to be able
> to
> >> >> >> commit to the new platform.
> >> >> >> Right now I see only one windows related build
> >> >> >> https://builds.apache.org/view/Hadoop/job/Hadoop-1-win/
> >> >> >> Which was failing since Sept 8, 2012 and did not run in the last
> >> >> >> month.
> >> >> >>
> >> >> >> Thanks,
> >> >> >> --Konst
> >> >> >>
> >> >> >> On Thu, Feb 28, 2013 at 8:47 PM, Eric Baldeschwieler
> >> >> >> <er...@hortonworks.com> wrote:
> >> >> >> > +1 (non-binding)
> >> >> >> >
> >> >> >> > A few of observations:
> >> >> >> >
> >> >> >> > - Windows has actually been a supported platform for Hadoop
> since
> >> >> >> > 0.1
> >> >> >> > .
> >> >> >> > Doug championed supporting windows then and we've continued to
> do
> >> >> >> > it
> >> >> >> > with
> >> >> >> > varying vigor over time.  To my knowledge we've never made a
> >> >> >> > decision
> >> >> >> > to
> >> >> >> > drop windows support.  The change here is improving our support
> >> >> >> > and
> >> >> >> > dropping
> >> >> >> > the requirement of cigwin.  We had Nutch windows users on the
> list
> >> >> >> > in
> >> >> >> > 2006
> >> >> >> > and we've been supporting windows FS requirements since
> inception.
> >> >> >> >
> >> >> >> > - A little pragmatism will go a long way.  As a community we've
> >> >> >> > got
> >> >> >> > to
> >> >> >> > stay committed to keeping hadoop simple (so it does work on many
> >> >> >> > platforms)
> >> >> >> > and extending it to take advantage of key emerging OS/hardware
> >> >> >> > features,
> >> >> >> > such as containers, new FSs, virtualization, flash ...  We
> should
> >> >> >> > all
> >> >> >> > plan
> >> >> >> > to let new features & optimizations emerge that don't work
> >> >> >> > everywhere, if
> >> >> >> > they are compelling and central to hadoop's mission of being THE
> >> >> >> > best
> >> >> >> > fabric
> >> >> >> > for storing and processing big data.
> >> >> >> >
> >> >> >> > - A UI project like KDE has to deal with the MANY differences
> >> >> >> > between
> >> >> >> > windows and linux UI APIs.  Hadoop faces no such complex
> challenge
> >> >> >> > and hence
> >> >> >> > can be maintained from a single codeline IMO.  It is mostly
> >> >> >> > abstracted from
> >> >> >> > the OS APIs via Java and our design choices.  Where it is not we
> >> >> >> > can
> >> >> >> > continue to add plugable abstractions.
> >> >> >> >
> >> >> >
> >> >> >
> >> >
> >> >
> >
> >
>

Re: [Vote] Merge branch-trunk-win to trunk

[Matt Foley <mf...@hortonworks.com>]

Thanks.  I agree Windows -1's in test-patch should not block commits.

--Matt



On Mon, Mar 4, 2013 at 2:30 PM, Konstantin Shvachko <sh...@gmail.com>wrote:

> On Mon, Mar 4, 2013 at 12:22 PM, Matt Foley <mf...@hortonworks.com>
> wrote:
> > Konstantine, you have voted -1, and stated some requirements before
> you'll
> > withdraw that -1.  As I plan to do work to fulfill those requirements, I
> > want to make sure that what I'm proposing will, in fact, satisfy you.
> > That's why I'm asking, if we implement full "test-patch" integration for
> > Windows, does it seem to you that that would provide adequate support?
>
> Yes.
>
> > I have learned not to presume that my interpretation is correct.  My
> > interpretation of item #1 is that test-patch provides pre-commit build,
> so
> > it would satisfy item #1.  But rather than assuming that I am
> interpreting
> > it correctly, I simply want your agreement that it would, or if not,
> > clarification why it won't.
>
> I agree it will satisfy my item #1.
> I did not agree in my previous email, but I changed my mind based on
> the latest discussion. I have to explain why now.
> I was proposing nightly build because I did not want pre-commit build
> for Windows block commits to Linux. But if people are fine just ignoring
> -1s for the Windows part of the build it should be good.
>
> > Regarding item #2, it is also my interpretation that test-patch provides
> an
> > on-demand (perhaps 20-minutes deferred) Jenkins build and unit test, with
> > logs available to the developer, so it would satisfy item #2.  But rather
> > than assuming that I am interpreting it correctly, I simply want your
> > agreement that it would, or if not, clarification why it won't.
>
> It will satisfy my item #2 in the following way:
> I can duplicate your pre-commit build for Windows and add an input
> parameter, which would let people run the build on their patches
> chosen from local machine rather than attaching them to Jiras.
>
> Thanks,
> --Konstantin
>
> > In agile terms, you are the Owner of these requirements.  Please give me
> > owner feedback as to whether my proposed work sounds like it will satisfy
> > the requirements.
> >
> > Thank you,
> > --Matt
> >
> >
> > On Sun, Mar 3, 2013 at 12:16 PM, Konstantin Shvachko <
> shv.hadoop@gmail.com>
> > wrote:
> >>
> >> Didn't I explain in details what I am asking for?
> >>
> >> Thanks,
> >> --Konst
> >>
> >> On Sun, Mar 3, 2013 at 11:08 AM, Matt Foley <mf...@hortonworks.com>
> >> wrote:
> >> > Hi Konstantin,
> >> > I'd like to point out two things:
> >> > First, I already committed in this thread (email of Thu, Feb 28, 2013
> at
> >> > 6:01 PM) to providing CI for Windows builds.  So please stop acting
> like
> >> > I'm
> >> > resisting this idea or something.
> >> > Second, you didn't answer my question, you just kvetched about the
> >> > phrasing.
> >> > So I ask again:
> >> >
> >> > Will providing full "test-patch" integration (pre-commit build and
> unit
> >> > test
> >> > triggered by Jira "Patch Available" state) satisfy your request for
> >> > functionality #1 and #2?  Yes or no, please.
> >> >
> >> > Thanks,
> >> > --Matt
> >> >
> >> >
> >> > On Sat, Mar 2, 2013 at 7:32 PM, Konstantin Shvachko
> >> > <sh...@gmail.com>
> >> > wrote:
> >> >>
> >> >> Hi Matt,
> >> >>
> >> >> On Sat, Mar 2, 2013 at 12:32 PM, Matt Foley <mf...@hortonworks.com>
> >> >> wrote:
> >> >> > Konstantin,
> >> >> > I would like to explore what it would take to remove this perceived
> >> >> > impediment --
> >> >>
> >> >> Glad you decided to explore. Thank you.
> >> >>
> >> >> > although I reserve the right to argue that this is not
> >> >> > pre-requisite to merging the cross-platform support patch.
> >> >>
> >> >> It's your right indeed. So as mine to question what the platform
> >> >> support means for you, which I believe remained unclear.
> >> >> I do not impede the change as you should have noticed. My requirement
> >> >> comes from my perception of the support, which means to me exactly
> two
> >> >> things:
> >> >> 1. The ability to recognise the code is broken for the platform
> >> >> 2. The ability to test new patches on the platform
> >> >> The latter is problematic, as many noticed in this thread, for those
> >> >> whose customary environment does not include Windows.
> >> >>
> >> >> > If we implemented full "test-patch" support for Windows on trunk,
> >> >> > would
> >> >> > that
> >> >> > fulfill both your items #1 and #2?  Please note that:
> >> >> > a) Pushing the "Patch Available" button in Jira shall cause a
> >> >> > pre-commit
> >> >> > build to start within, I believe, 20 minutes.
> >> >> > b) That build keeps logs for both java build and unit tests for
> >> >> > several
> >> >> > days, that are accessible to all viewers.
> >> >>
> >> >> In item #1 I mostly asking for the nightly build, which is simpler
> >> >> than "test-patch". The latter would be ideal from the platform
> support
> >> >> viewpoint, but it is for the community to decide if we want to add
> >> >> extra +3 hours to the build.
> >> >> Nightly build in my understanding is triggered by the timer rather
> >> >> than by Jira's "submit patch" button.  On Jenkins build configuration
> >> >> you can specify it under "Build periodically".
> >> >>
> >> >> > So, does this provide sufficient on-demand support that we don't
> have
> >> >> > to
> >> >> > implement a whole new on-demand VM support structure of some sort
> for
> >> >> > #2
> >> >> > (which would be an extraordinary and impractical demand)?
> >> >>
> >> >> I did not mention VMs. Item #2 means a build, which runs "test-patch"
> >> >> target with the file specified by a user (instead of a jira
> >> >> attachment).
> >> >> When user clicks "Build Now" link a box is displayed where the user
> >> >> can enter the file path containing the patch. This can be specified
> in
> >> >> the Build Configuration under "This build is parameterized" by
> >> >> choosing AddParameter / FileParameter. The build can run on the same
> >> >> Windows machine as the nightly build.
> >> >> Such build will let people test their patches for Windows on Jenkins
> >> >> if they don't posses a license for the right version of Windows.
> >> >> I hope this will not turn into extraordinary or impractical effort.
> >> >>
> >> >> Thanks,
> >> >> --Konst
> >> >>
> >> >> > Thanks,
> >> >> > --Matt
> >> >> >
> >> >> >
> >> >> > On Fri, Mar 1, 2013 at 12:18 PM, Konstantin Shvachko
> >> >> > <sh...@gmail.com>
> >> >> > wrote:
> >> >> >>
> >> >> >> -1
> >> >> >> We should have a CI infrastructure in place before we can commit
> to
> >> >> >> supporting Windows platform.
> >> >> >>
> >> >> >> Eric is right Win/Cygwin was supported since day one.
> >> >> >> I had a Windows box under my desk running nightly builds back in
> >> >> >> 2006-07.
> >> >> >> People were irritated but I was filing windows bugs until 0.22
> >> >> >> release.
> >> >> >> Times changing and I am glad to see wider support for Win
> platform.
> >> >> >>
> >> >> >> But in order to make it work you guys need to put the CI process
> in
> >> >> >> place
> >> >> >>
> >> >> >> 1. windows jenkins build: could be nightly or PreCommit.
> >> >> >> - Nightly would mean that changes can be committed to trunk based
> on
> >> >> >> linux PreCommit build. And people will file bugs if the change
> broke
> >> >> >> Windows nightly build.
> >> >> >> - PreCommit-win build will mean automatic reporting failed tests
> to
> >> >> >> respective jira blocking commits the same way as it is now with
> >> >> >> linux
> >> >> >> PreCommit builds.
> >> >> >> We should discuss which way is more efficient for developers.
> >> >> >>
> >> >> >> 2. On-demand-windows Jenkins build.
> >> >> >> I see it as a build to which I can attach my patch and the build
> >> >> >> will
> >> >> >> run my changes on a dedicated windows box.
> >> >> >> That way people can test their changes without having personal
> >> >> >> windows
> >> >> >> nodes.
> >> >> >>
> >> >> >> I think this is the minimal set of requirement for us to be able
> to
> >> >> >> commit to the new platform.
> >> >> >> Right now I see only one windows related build
> >> >> >> https://builds.apache.org/view/Hadoop/job/Hadoop-1-win/
> >> >> >> Which was failing since Sept 8, 2012 and did not run in the last
> >> >> >> month.
> >> >> >>
> >> >> >> Thanks,
> >> >> >> --Konst
> >> >> >>
> >> >> >> On Thu, Feb 28, 2013 at 8:47 PM, Eric Baldeschwieler
> >> >> >> <er...@hortonworks.com> wrote:
> >> >> >> > +1 (non-binding)
> >> >> >> >
> >> >> >> > A few of observations:
> >> >> >> >
> >> >> >> > - Windows has actually been a supported platform for Hadoop
> since
> >> >> >> > 0.1
> >> >> >> > .
> >> >> >> > Doug championed supporting windows then and we've continued to
> do
> >> >> >> > it
> >> >> >> > with
> >> >> >> > varying vigor over time.  To my knowledge we've never made a
> >> >> >> > decision
> >> >> >> > to
> >> >> >> > drop windows support.  The change here is improving our support
> >> >> >> > and
> >> >> >> > dropping
> >> >> >> > the requirement of cigwin.  We had Nutch windows users on the
> list
> >> >> >> > in
> >> >> >> > 2006
> >> >> >> > and we've been supporting windows FS requirements since
> inception.
> >> >> >> >
> >> >> >> > - A little pragmatism will go a long way.  As a community we've
> >> >> >> > got
> >> >> >> > to
> >> >> >> > stay committed to keeping hadoop simple (so it does work on many
> >> >> >> > platforms)
> >> >> >> > and extending it to take advantage of key emerging OS/hardware
> >> >> >> > features,
> >> >> >> > such as containers, new FSs, virtualization, flash ...  We
> should
> >> >> >> > all
> >> >> >> > plan
> >> >> >> > to let new features & optimizations emerge that don't work
> >> >> >> > everywhere, if
> >> >> >> > they are compelling and central to hadoop's mission of being THE
> >> >> >> > best
> >> >> >> > fabric
> >> >> >> > for storing and processing big data.
> >> >> >> >
> >> >> >> > - A UI project like KDE has to deal with the MANY differences
> >> >> >> > between
> >> >> >> > windows and linux UI APIs.  Hadoop faces no such complex
> challenge
> >> >> >> > and hence
> >> >> >> > can be maintained from a single codeline IMO.  It is mostly
> >> >> >> > abstracted from
> >> >> >> > the OS APIs via Java and our design choices.  Where it is not we
> >> >> >> > can
> >> >> >> > continue to add plugable abstractions.
> >> >> >> >
> >> >> >
> >> >> >
> >> >
> >> >
> >
> >
>

Re: [Vote] Merge branch-trunk-win to trunk

[Konstantin Shvachko <sh...@gmail.com>]

On Mon, Mar 4, 2013 at 12:22 PM, Matt Foley <mf...@hortonworks.com> wrote:
> Konstantine, you have voted -1, and stated some requirements before you'll
> withdraw that -1.  As I plan to do work to fulfill those requirements, I
> want to make sure that what I'm proposing will, in fact, satisfy you.
> That's why I'm asking, if we implement full "test-patch" integration for
> Windows, does it seem to you that that would provide adequate support?

Yes.

> I have learned not to presume that my interpretation is correct.  My
> interpretation of item #1 is that test-patch provides pre-commit build, so
> it would satisfy item #1.  But rather than assuming that I am interpreting
> it correctly, I simply want your agreement that it would, or if not,
> clarification why it won't.

I agree it will satisfy my item #1.
I did not agree in my previous email, but I changed my mind based on
the latest discussion. I have to explain why now.
I was proposing nightly build because I did not want pre-commit build
for Windows block commits to Linux. But if people are fine just ignoring
-1s for the Windows part of the build it should be good.

> Regarding item #2, it is also my interpretation that test-patch provides an
> on-demand (perhaps 20-minutes deferred) Jenkins build and unit test, with
> logs available to the developer, so it would satisfy item #2.  But rather
> than assuming that I am interpreting it correctly, I simply want your
> agreement that it would, or if not, clarification why it won't.

It will satisfy my item #2 in the following way:
I can duplicate your pre-commit build for Windows and add an input
parameter, which would let people run the build on their patches
chosen from local machine rather than attaching them to Jiras.

Thanks,
--Konstantin

> In agile terms, you are the Owner of these requirements.  Please give me
> owner feedback as to whether my proposed work sounds like it will satisfy
> the requirements.
>
> Thank you,
> --Matt
>
>
> On Sun, Mar 3, 2013 at 12:16 PM, Konstantin Shvachko <sh...@gmail.com>
> wrote:
>>
>> Didn't I explain in details what I am asking for?
>>
>> Thanks,
>> --Konst
>>
>> On Sun, Mar 3, 2013 at 11:08 AM, Matt Foley <mf...@hortonworks.com>
>> wrote:
>> > Hi Konstantin,
>> > I'd like to point out two things:
>> > First, I already committed in this thread (email of Thu, Feb 28, 2013 at
>> > 6:01 PM) to providing CI for Windows builds.  So please stop acting like
>> > I'm
>> > resisting this idea or something.
>> > Second, you didn't answer my question, you just kvetched about the
>> > phrasing.
>> > So I ask again:
>> >
>> > Will providing full "test-patch" integration (pre-commit build and unit
>> > test
>> > triggered by Jira "Patch Available" state) satisfy your request for
>> > functionality #1 and #2?  Yes or no, please.
>> >
>> > Thanks,
>> > --Matt
>> >
>> >
>> > On Sat, Mar 2, 2013 at 7:32 PM, Konstantin Shvachko
>> > <sh...@gmail.com>
>> > wrote:
>> >>
>> >> Hi Matt,
>> >>
>> >> On Sat, Mar 2, 2013 at 12:32 PM, Matt Foley <mf...@hortonworks.com>
>> >> wrote:
>> >> > Konstantin,
>> >> > I would like to explore what it would take to remove this perceived
>> >> > impediment --
>> >>
>> >> Glad you decided to explore. Thank you.
>> >>
>> >> > although I reserve the right to argue that this is not
>> >> > pre-requisite to merging the cross-platform support patch.
>> >>
>> >> It's your right indeed. So as mine to question what the platform
>> >> support means for you, which I believe remained unclear.
>> >> I do not impede the change as you should have noticed. My requirement
>> >> comes from my perception of the support, which means to me exactly two
>> >> things:
>> >> 1. The ability to recognise the code is broken for the platform
>> >> 2. The ability to test new patches on the platform
>> >> The latter is problematic, as many noticed in this thread, for those
>> >> whose customary environment does not include Windows.
>> >>
>> >> > If we implemented full "test-patch" support for Windows on trunk,
>> >> > would
>> >> > that
>> >> > fulfill both your items #1 and #2?  Please note that:
>> >> > a) Pushing the "Patch Available" button in Jira shall cause a
>> >> > pre-commit
>> >> > build to start within, I believe, 20 minutes.
>> >> > b) That build keeps logs for both java build and unit tests for
>> >> > several
>> >> > days, that are accessible to all viewers.
>> >>
>> >> In item #1 I mostly asking for the nightly build, which is simpler
>> >> than "test-patch". The latter would be ideal from the platform support
>> >> viewpoint, but it is for the community to decide if we want to add
>> >> extra +3 hours to the build.
>> >> Nightly build in my understanding is triggered by the timer rather
>> >> than by Jira's "submit patch" button.  On Jenkins build configuration
>> >> you can specify it under "Build periodically".
>> >>
>> >> > So, does this provide sufficient on-demand support that we don't have
>> >> > to
>> >> > implement a whole new on-demand VM support structure of some sort for
>> >> > #2
>> >> > (which would be an extraordinary and impractical demand)?
>> >>
>> >> I did not mention VMs. Item #2 means a build, which runs "test-patch"
>> >> target with the file specified by a user (instead of a jira
>> >> attachment).
>> >> When user clicks "Build Now" link a box is displayed where the user
>> >> can enter the file path containing the patch. This can be specified in
>> >> the Build Configuration under "This build is parameterized" by
>> >> choosing AddParameter / FileParameter. The build can run on the same
>> >> Windows machine as the nightly build.
>> >> Such build will let people test their patches for Windows on Jenkins
>> >> if they don't posses a license for the right version of Windows.
>> >> I hope this will not turn into extraordinary or impractical effort.
>> >>
>> >> Thanks,
>> >> --Konst
>> >>
>> >> > Thanks,
>> >> > --Matt
>> >> >
>> >> >
>> >> > On Fri, Mar 1, 2013 at 12:18 PM, Konstantin Shvachko
>> >> > <sh...@gmail.com>
>> >> > wrote:
>> >> >>
>> >> >> -1
>> >> >> We should have a CI infrastructure in place before we can commit to
>> >> >> supporting Windows platform.
>> >> >>
>> >> >> Eric is right Win/Cygwin was supported since day one.
>> >> >> I had a Windows box under my desk running nightly builds back in
>> >> >> 2006-07.
>> >> >> People were irritated but I was filing windows bugs until 0.22
>> >> >> release.
>> >> >> Times changing and I am glad to see wider support for Win platform.
>> >> >>
>> >> >> But in order to make it work you guys need to put the CI process in
>> >> >> place
>> >> >>
>> >> >> 1. windows jenkins build: could be nightly or PreCommit.
>> >> >> - Nightly would mean that changes can be committed to trunk based on
>> >> >> linux PreCommit build. And people will file bugs if the change broke
>> >> >> Windows nightly build.
>> >> >> - PreCommit-win build will mean automatic reporting failed tests to
>> >> >> respective jira blocking commits the same way as it is now with
>> >> >> linux
>> >> >> PreCommit builds.
>> >> >> We should discuss which way is more efficient for developers.
>> >> >>
>> >> >> 2. On-demand-windows Jenkins build.
>> >> >> I see it as a build to which I can attach my patch and the build
>> >> >> will
>> >> >> run my changes on a dedicated windows box.
>> >> >> That way people can test their changes without having personal
>> >> >> windows
>> >> >> nodes.
>> >> >>
>> >> >> I think this is the minimal set of requirement for us to be able to
>> >> >> commit to the new platform.
>> >> >> Right now I see only one windows related build
>> >> >> https://builds.apache.org/view/Hadoop/job/Hadoop-1-win/
>> >> >> Which was failing since Sept 8, 2012 and did not run in the last
>> >> >> month.
>> >> >>
>> >> >> Thanks,
>> >> >> --Konst
>> >> >>
>> >> >> On Thu, Feb 28, 2013 at 8:47 PM, Eric Baldeschwieler
>> >> >> <er...@hortonworks.com> wrote:
>> >> >> > +1 (non-binding)
>> >> >> >
>> >> >> > A few of observations:
>> >> >> >
>> >> >> > - Windows has actually been a supported platform for Hadoop since
>> >> >> > 0.1
>> >> >> > .
>> >> >> > Doug championed supporting windows then and we've continued to do
>> >> >> > it
>> >> >> > with
>> >> >> > varying vigor over time.  To my knowledge we've never made a
>> >> >> > decision
>> >> >> > to
>> >> >> > drop windows support.  The change here is improving our support
>> >> >> > and
>> >> >> > dropping
>> >> >> > the requirement of cigwin.  We had Nutch windows users on the list
>> >> >> > in
>> >> >> > 2006
>> >> >> > and we've been supporting windows FS requirements since inception.
>> >> >> >
>> >> >> > - A little pragmatism will go a long way.  As a community we've
>> >> >> > got
>> >> >> > to
>> >> >> > stay committed to keeping hadoop simple (so it does work on many
>> >> >> > platforms)
>> >> >> > and extending it to take advantage of key emerging OS/hardware
>> >> >> > features,
>> >> >> > such as containers, new FSs, virtualization, flash ...  We should
>> >> >> > all
>> >> >> > plan
>> >> >> > to let new features & optimizations emerge that don't work
>> >> >> > everywhere, if
>> >> >> > they are compelling and central to hadoop's mission of being THE
>> >> >> > best
>> >> >> > fabric
>> >> >> > for storing and processing big data.
>> >> >> >
>> >> >> > - A UI project like KDE has to deal with the MANY differences
>> >> >> > between
>> >> >> > windows and linux UI APIs.  Hadoop faces no such complex challenge
>> >> >> > and hence
>> >> >> > can be maintained from a single codeline IMO.  It is mostly
>> >> >> > abstracted from
>> >> >> > the OS APIs via Java and our design choices.  Where it is not we
>> >> >> > can
>> >> >> > continue to add plugable abstractions.
>> >> >> >
>> >> >
>> >> >
>> >
>> >
>
>

Re: [Vote] Merge branch-trunk-win to trunk

[Konstantin Shvachko <sh...@gmail.com>]

On Mon, Mar 4, 2013 at 12:22 PM, Matt Foley <mf...@hortonworks.com> wrote:
> Konstantine, you have voted -1, and stated some requirements before you'll
> withdraw that -1.  As I plan to do work to fulfill those requirements, I
> want to make sure that what I'm proposing will, in fact, satisfy you.
> That's why I'm asking, if we implement full "test-patch" integration for
> Windows, does it seem to you that that would provide adequate support?

Yes.

> I have learned not to presume that my interpretation is correct.  My
> interpretation of item #1 is that test-patch provides pre-commit build, so
> it would satisfy item #1.  But rather than assuming that I am interpreting
> it correctly, I simply want your agreement that it would, or if not,
> clarification why it won't.

I agree it will satisfy my item #1.
I did not agree in my previous email, but I changed my mind based on
the latest discussion. I have to explain why now.
I was proposing nightly build because I did not want pre-commit build
for Windows block commits to Linux. But if people are fine just ignoring
-1s for the Windows part of the build it should be good.

> Regarding item #2, it is also my interpretation that test-patch provides an
> on-demand (perhaps 20-minutes deferred) Jenkins build and unit test, with
> logs available to the developer, so it would satisfy item #2.  But rather
> than assuming that I am interpreting it correctly, I simply want your
> agreement that it would, or if not, clarification why it won't.

It will satisfy my item #2 in the following way:
I can duplicate your pre-commit build for Windows and add an input
parameter, which would let people run the build on their patches
chosen from local machine rather than attaching them to Jiras.

Thanks,
--Konstantin

> In agile terms, you are the Owner of these requirements.  Please give me
> owner feedback as to whether my proposed work sounds like it will satisfy
> the requirements.
>
> Thank you,
> --Matt
>
>
> On Sun, Mar 3, 2013 at 12:16 PM, Konstantin Shvachko <sh...@gmail.com>
> wrote:
>>
>> Didn't I explain in details what I am asking for?
>>
>> Thanks,
>> --Konst
>>
>> On Sun, Mar 3, 2013 at 11:08 AM, Matt Foley <mf...@hortonworks.com>
>> wrote:
>> > Hi Konstantin,
>> > I'd like to point out two things:
>> > First, I already committed in this thread (email of Thu, Feb 28, 2013 at
>> > 6:01 PM) to providing CI for Windows builds.  So please stop acting like
>> > I'm
>> > resisting this idea or something.
>> > Second, you didn't answer my question, you just kvetched about the
>> > phrasing.
>> > So I ask again:
>> >
>> > Will providing full "test-patch" integration (pre-commit build and unit
>> > test
>> > triggered by Jira "Patch Available" state) satisfy your request for
>> > functionality #1 and #2?  Yes or no, please.
>> >
>> > Thanks,
>> > --Matt
>> >
>> >
>> > On Sat, Mar 2, 2013 at 7:32 PM, Konstantin Shvachko
>> > <sh...@gmail.com>
>> > wrote:
>> >>
>> >> Hi Matt,
>> >>
>> >> On Sat, Mar 2, 2013 at 12:32 PM, Matt Foley <mf...@hortonworks.com>
>> >> wrote:
>> >> > Konstantin,
>> >> > I would like to explore what it would take to remove this perceived
>> >> > impediment --
>> >>
>> >> Glad you decided to explore. Thank you.
>> >>
>> >> > although I reserve the right to argue that this is not
>> >> > pre-requisite to merging the cross-platform support patch.
>> >>
>> >> It's your right indeed. So as mine to question what the platform
>> >> support means for you, which I believe remained unclear.
>> >> I do not impede the change as you should have noticed. My requirement
>> >> comes from my perception of the support, which means to me exactly two
>> >> things:
>> >> 1. The ability to recognise the code is broken for the platform
>> >> 2. The ability to test new patches on the platform
>> >> The latter is problematic, as many noticed in this thread, for those
>> >> whose customary environment does not include Windows.
>> >>
>> >> > If we implemented full "test-patch" support for Windows on trunk,
>> >> > would
>> >> > that
>> >> > fulfill both your items #1 and #2?  Please note that:
>> >> > a) Pushing the "Patch Available" button in Jira shall cause a
>> >> > pre-commit
>> >> > build to start within, I believe, 20 minutes.
>> >> > b) That build keeps logs for both java build and unit tests for
>> >> > several
>> >> > days, that are accessible to all viewers.
>> >>
>> >> In item #1 I mostly asking for the nightly build, which is simpler
>> >> than "test-patch". The latter would be ideal from the platform support
>> >> viewpoint, but it is for the community to decide if we want to add
>> >> extra +3 hours to the build.
>> >> Nightly build in my understanding is triggered by the timer rather
>> >> than by Jira's "submit patch" button.  On Jenkins build configuration
>> >> you can specify it under "Build periodically".
>> >>
>> >> > So, does this provide sufficient on-demand support that we don't have
>> >> > to
>> >> > implement a whole new on-demand VM support structure of some sort for
>> >> > #2
>> >> > (which would be an extraordinary and impractical demand)?
>> >>
>> >> I did not mention VMs. Item #2 means a build, which runs "test-patch"
>> >> target with the file specified by a user (instead of a jira
>> >> attachment).
>> >> When user clicks "Build Now" link a box is displayed where the user
>> >> can enter the file path containing the patch. This can be specified in
>> >> the Build Configuration under "This build is parameterized" by
>> >> choosing AddParameter / FileParameter. The build can run on the same
>> >> Windows machine as the nightly build.
>> >> Such build will let people test their patches for Windows on Jenkins
>> >> if they don't posses a license for the right version of Windows.
>> >> I hope this will not turn into extraordinary or impractical effort.
>> >>
>> >> Thanks,
>> >> --Konst
>> >>
>> >> > Thanks,
>> >> > --Matt
>> >> >
>> >> >
>> >> > On Fri, Mar 1, 2013 at 12:18 PM, Konstantin Shvachko
>> >> > <sh...@gmail.com>
>> >> > wrote:
>> >> >>
>> >> >> -1
>> >> >> We should have a CI infrastructure in place before we can commit to
>> >> >> supporting Windows platform.
>> >> >>
>> >> >> Eric is right Win/Cygwin was supported since day one.
>> >> >> I had a Windows box under my desk running nightly builds back in
>> >> >> 2006-07.
>> >> >> People were irritated but I was filing windows bugs until 0.22
>> >> >> release.
>> >> >> Times changing and I am glad to see wider support for Win platform.
>> >> >>
>> >> >> But in order to make it work you guys need to put the CI process in
>> >> >> place
>> >> >>
>> >> >> 1. windows jenkins build: could be nightly or PreCommit.
>> >> >> - Nightly would mean that changes can be committed to trunk based on
>> >> >> linux PreCommit build. And people will file bugs if the change broke
>> >> >> Windows nightly build.
>> >> >> - PreCommit-win build will mean automatic reporting failed tests to
>> >> >> respective jira blocking commits the same way as it is now with
>> >> >> linux
>> >> >> PreCommit builds.
>> >> >> We should discuss which way is more efficient for developers.
>> >> >>
>> >> >> 2. On-demand-windows Jenkins build.
>> >> >> I see it as a build to which I can attach my patch and the build
>> >> >> will
>> >> >> run my changes on a dedicated windows box.
>> >> >> That way people can test their changes without having personal
>> >> >> windows
>> >> >> nodes.
>> >> >>
>> >> >> I think this is the minimal set of requirement for us to be able to
>> >> >> commit to the new platform.
>> >> >> Right now I see only one windows related build
>> >> >> https://builds.apache.org/view/Hadoop/job/Hadoop-1-win/
>> >> >> Which was failing since Sept 8, 2012 and did not run in the last
>> >> >> month.
>> >> >>
>> >> >> Thanks,
>> >> >> --Konst
>> >> >>
>> >> >> On Thu, Feb 28, 2013 at 8:47 PM, Eric Baldeschwieler
>> >> >> <er...@hortonworks.com> wrote:
>> >> >> > +1 (non-binding)
>> >> >> >
>> >> >> > A few of observations:
>> >> >> >
>> >> >> > - Windows has actually been a supported platform for Hadoop since
>> >> >> > 0.1
>> >> >> > .
>> >> >> > Doug championed supporting windows then and we've continued to do
>> >> >> > it
>> >> >> > with
>> >> >> > varying vigor over time.  To my knowledge we've never made a
>> >> >> > decision
>> >> >> > to
>> >> >> > drop windows support.  The change here is improving our support
>> >> >> > and
>> >> >> > dropping
>> >> >> > the requirement of cigwin.  We had Nutch windows users on the list
>> >> >> > in
>> >> >> > 2006
>> >> >> > and we've been supporting windows FS requirements since inception.
>> >> >> >
>> >> >> > - A little pragmatism will go a long way.  As a community we've
>> >> >> > got
>> >> >> > to
>> >> >> > stay committed to keeping hadoop simple (so it does work on many
>> >> >> > platforms)
>> >> >> > and extending it to take advantage of key emerging OS/hardware
>> >> >> > features,
>> >> >> > such as containers, new FSs, virtualization, flash ...  We should
>> >> >> > all
>> >> >> > plan
>> >> >> > to let new features & optimizations emerge that don't work
>> >> >> > everywhere, if
>> >> >> > they are compelling and central to hadoop's mission of being THE
>> >> >> > best
>> >> >> > fabric
>> >> >> > for storing and processing big data.
>> >> >> >
>> >> >> > - A UI project like KDE has to deal with the MANY differences
>> >> >> > between
>> >> >> > windows and linux UI APIs.  Hadoop faces no such complex challenge
>> >> >> > and hence
>> >> >> > can be maintained from a single codeline IMO.  It is mostly
>> >> >> > abstracted from
>> >> >> > the OS APIs via Java and our design choices.  Where it is not we
>> >> >> > can
>> >> >> > continue to add plugable abstractions.
>> >> >> >
>> >> >
>> >> >
>> >
>> >
>
>

Re: [Vote] Merge branch-trunk-win to trunk

[Konstantin Shvachko <sh...@gmail.com>]

On Mon, Mar 4, 2013 at 12:22 PM, Matt Foley <mf...@hortonworks.com> wrote:
> Konstantine, you have voted -1, and stated some requirements before you'll
> withdraw that -1.  As I plan to do work to fulfill those requirements, I
> want to make sure that what I'm proposing will, in fact, satisfy you.
> That's why I'm asking, if we implement full "test-patch" integration for
> Windows, does it seem to you that that would provide adequate support?

Yes.

> I have learned not to presume that my interpretation is correct.  My
> interpretation of item #1 is that test-patch provides pre-commit build, so
> it would satisfy item #1.  But rather than assuming that I am interpreting
> it correctly, I simply want your agreement that it would, or if not,
> clarification why it won't.

I agree it will satisfy my item #1.
I did not agree in my previous email, but I changed my mind based on
the latest discussion. I have to explain why now.
I was proposing nightly build because I did not want pre-commit build
for Windows block commits to Linux. But if people are fine just ignoring
-1s for the Windows part of the build it should be good.

> Regarding item #2, it is also my interpretation that test-patch provides an
> on-demand (perhaps 20-minutes deferred) Jenkins build and unit test, with
> logs available to the developer, so it would satisfy item #2.  But rather
> than assuming that I am interpreting it correctly, I simply want your
> agreement that it would, or if not, clarification why it won't.

It will satisfy my item #2 in the following way:
I can duplicate your pre-commit build for Windows and add an input
parameter, which would let people run the build on their patches
chosen from local machine rather than attaching them to Jiras.

Thanks,
--Konstantin

> In agile terms, you are the Owner of these requirements.  Please give me
> owner feedback as to whether my proposed work sounds like it will satisfy
> the requirements.
>
> Thank you,
> --Matt
>
>
> On Sun, Mar 3, 2013 at 12:16 PM, Konstantin Shvachko <sh...@gmail.com>
> wrote:
>>
>> Didn't I explain in details what I am asking for?
>>
>> Thanks,
>> --Konst
>>
>> On Sun, Mar 3, 2013 at 11:08 AM, Matt Foley <mf...@hortonworks.com>
>> wrote:
>> > Hi Konstantin,
>> > I'd like to point out two things:
>> > First, I already committed in this thread (email of Thu, Feb 28, 2013 at
>> > 6:01 PM) to providing CI for Windows builds.  So please stop acting like
>> > I'm
>> > resisting this idea or something.
>> > Second, you didn't answer my question, you just kvetched about the
>> > phrasing.
>> > So I ask again:
>> >
>> > Will providing full "test-patch" integration (pre-commit build and unit
>> > test
>> > triggered by Jira "Patch Available" state) satisfy your request for
>> > functionality #1 and #2?  Yes or no, please.
>> >
>> > Thanks,
>> > --Matt
>> >
>> >
>> > On Sat, Mar 2, 2013 at 7:32 PM, Konstantin Shvachko
>> > <sh...@gmail.com>
>> > wrote:
>> >>
>> >> Hi Matt,
>> >>
>> >> On Sat, Mar 2, 2013 at 12:32 PM, Matt Foley <mf...@hortonworks.com>
>> >> wrote:
>> >> > Konstantin,
>> >> > I would like to explore what it would take to remove this perceived
>> >> > impediment --
>> >>
>> >> Glad you decided to explore. Thank you.
>> >>
>> >> > although I reserve the right to argue that this is not
>> >> > pre-requisite to merging the cross-platform support patch.
>> >>
>> >> It's your right indeed. So as mine to question what the platform
>> >> support means for you, which I believe remained unclear.
>> >> I do not impede the change as you should have noticed. My requirement
>> >> comes from my perception of the support, which means to me exactly two
>> >> things:
>> >> 1. The ability to recognise the code is broken for the platform
>> >> 2. The ability to test new patches on the platform
>> >> The latter is problematic, as many noticed in this thread, for those
>> >> whose customary environment does not include Windows.
>> >>
>> >> > If we implemented full "test-patch" support for Windows on trunk,
>> >> > would
>> >> > that
>> >> > fulfill both your items #1 and #2?  Please note that:
>> >> > a) Pushing the "Patch Available" button in Jira shall cause a
>> >> > pre-commit
>> >> > build to start within, I believe, 20 minutes.
>> >> > b) That build keeps logs for both java build and unit tests for
>> >> > several
>> >> > days, that are accessible to all viewers.
>> >>
>> >> In item #1 I mostly asking for the nightly build, which is simpler
>> >> than "test-patch". The latter would be ideal from the platform support
>> >> viewpoint, but it is for the community to decide if we want to add
>> >> extra +3 hours to the build.
>> >> Nightly build in my understanding is triggered by the timer rather
>> >> than by Jira's "submit patch" button.  On Jenkins build configuration
>> >> you can specify it under "Build periodically".
>> >>
>> >> > So, does this provide sufficient on-demand support that we don't have
>> >> > to
>> >> > implement a whole new on-demand VM support structure of some sort for
>> >> > #2
>> >> > (which would be an extraordinary and impractical demand)?
>> >>
>> >> I did not mention VMs. Item #2 means a build, which runs "test-patch"
>> >> target with the file specified by a user (instead of a jira
>> >> attachment).
>> >> When user clicks "Build Now" link a box is displayed where the user
>> >> can enter the file path containing the patch. This can be specified in
>> >> the Build Configuration under "This build is parameterized" by
>> >> choosing AddParameter / FileParameter. The build can run on the same
>> >> Windows machine as the nightly build.
>> >> Such build will let people test their patches for Windows on Jenkins
>> >> if they don't posses a license for the right version of Windows.
>> >> I hope this will not turn into extraordinary or impractical effort.
>> >>
>> >> Thanks,
>> >> --Konst
>> >>
>> >> > Thanks,
>> >> > --Matt
>> >> >
>> >> >
>> >> > On Fri, Mar 1, 2013 at 12:18 PM, Konstantin Shvachko
>> >> > <sh...@gmail.com>
>> >> > wrote:
>> >> >>
>> >> >> -1
>> >> >> We should have a CI infrastructure in place before we can commit to
>> >> >> supporting Windows platform.
>> >> >>
>> >> >> Eric is right Win/Cygwin was supported since day one.
>> >> >> I had a Windows box under my desk running nightly builds back in
>> >> >> 2006-07.
>> >> >> People were irritated but I was filing windows bugs until 0.22
>> >> >> release.
>> >> >> Times changing and I am glad to see wider support for Win platform.
>> >> >>
>> >> >> But in order to make it work you guys need to put the CI process in
>> >> >> place
>> >> >>
>> >> >> 1. windows jenkins build: could be nightly or PreCommit.
>> >> >> - Nightly would mean that changes can be committed to trunk based on
>> >> >> linux PreCommit build. And people will file bugs if the change broke
>> >> >> Windows nightly build.
>> >> >> - PreCommit-win build will mean automatic reporting failed tests to
>> >> >> respective jira blocking commits the same way as it is now with
>> >> >> linux
>> >> >> PreCommit builds.
>> >> >> We should discuss which way is more efficient for developers.
>> >> >>
>> >> >> 2. On-demand-windows Jenkins build.
>> >> >> I see it as a build to which I can attach my patch and the build
>> >> >> will
>> >> >> run my changes on a dedicated windows box.
>> >> >> That way people can test their changes without having personal
>> >> >> windows
>> >> >> nodes.
>> >> >>
>> >> >> I think this is the minimal set of requirement for us to be able to
>> >> >> commit to the new platform.
>> >> >> Right now I see only one windows related build
>> >> >> https://builds.apache.org/view/Hadoop/job/Hadoop-1-win/
>> >> >> Which was failing since Sept 8, 2012 and did not run in the last
>> >> >> month.
>> >> >>
>> >> >> Thanks,
>> >> >> --Konst
>> >> >>
>> >> >> On Thu, Feb 28, 2013 at 8:47 PM, Eric Baldeschwieler
>> >> >> <er...@hortonworks.com> wrote:
>> >> >> > +1 (non-binding)
>> >> >> >
>> >> >> > A few of observations:
>> >> >> >
>> >> >> > - Windows has actually been a supported platform for Hadoop since
>> >> >> > 0.1
>> >> >> > .
>> >> >> > Doug championed supporting windows then and we've continued to do
>> >> >> > it
>> >> >> > with
>> >> >> > varying vigor over time.  To my knowledge we've never made a
>> >> >> > decision
>> >> >> > to
>> >> >> > drop windows support.  The change here is improving our support
>> >> >> > and
>> >> >> > dropping
>> >> >> > the requirement of cigwin.  We had Nutch windows users on the list
>> >> >> > in
>> >> >> > 2006
>> >> >> > and we've been supporting windows FS requirements since inception.
>> >> >> >
>> >> >> > - A little pragmatism will go a long way.  As a community we've
>> >> >> > got
>> >> >> > to
>> >> >> > stay committed to keeping hadoop simple (so it does work on many
>> >> >> > platforms)
>> >> >> > and extending it to take advantage of key emerging OS/hardware
>> >> >> > features,
>> >> >> > such as containers, new FSs, virtualization, flash ...  We should
>> >> >> > all
>> >> >> > plan
>> >> >> > to let new features & optimizations emerge that don't work
>> >> >> > everywhere, if
>> >> >> > they are compelling and central to hadoop's mission of being THE
>> >> >> > best
>> >> >> > fabric
>> >> >> > for storing and processing big data.
>> >> >> >
>> >> >> > - A UI project like KDE has to deal with the MANY differences
>> >> >> > between
>> >> >> > windows and linux UI APIs.  Hadoop faces no such complex challenge
>> >> >> > and hence
>> >> >> > can be maintained from a single codeline IMO.  It is mostly
>> >> >> > abstracted from
>> >> >> > the OS APIs via Java and our design choices.  Where it is not we
>> >> >> > can
>> >> >> > continue to add plugable abstractions.
>> >> >> >
>> >> >
>> >> >
>> >
>> >
>
>

Re: [Vote] Merge branch-trunk-win to trunk

[Konstantin Shvachko <sh...@gmail.com>]

On Mon, Mar 4, 2013 at 12:22 PM, Matt Foley <mf...@hortonworks.com> wrote:
> Konstantine, you have voted -1, and stated some requirements before you'll
> withdraw that -1.  As I plan to do work to fulfill those requirements, I
> want to make sure that what I'm proposing will, in fact, satisfy you.
> That's why I'm asking, if we implement full "test-patch" integration for
> Windows, does it seem to you that that would provide adequate support?

Yes.

> I have learned not to presume that my interpretation is correct.  My
> interpretation of item #1 is that test-patch provides pre-commit build, so
> it would satisfy item #1.  But rather than assuming that I am interpreting
> it correctly, I simply want your agreement that it would, or if not,
> clarification why it won't.

I agree it will satisfy my item #1.
I did not agree in my previous email, but I changed my mind based on
the latest discussion. I have to explain why now.
I was proposing nightly build because I did not want pre-commit build
for Windows block commits to Linux. But if people are fine just ignoring
-1s for the Windows part of the build it should be good.

> Regarding item #2, it is also my interpretation that test-patch provides an
> on-demand (perhaps 20-minutes deferred) Jenkins build and unit test, with
> logs available to the developer, so it would satisfy item #2.  But rather
> than assuming that I am interpreting it correctly, I simply want your
> agreement that it would, or if not, clarification why it won't.

It will satisfy my item #2 in the following way:
I can duplicate your pre-commit build for Windows and add an input
parameter, which would let people run the build on their patches
chosen from local machine rather than attaching them to Jiras.

Thanks,
--Konstantin

> In agile terms, you are the Owner of these requirements.  Please give me
> owner feedback as to whether my proposed work sounds like it will satisfy
> the requirements.
>
> Thank you,
> --Matt
>
>
> On Sun, Mar 3, 2013 at 12:16 PM, Konstantin Shvachko <sh...@gmail.com>
> wrote:
>>
>> Didn't I explain in details what I am asking for?
>>
>> Thanks,
>> --Konst
>>
>> On Sun, Mar 3, 2013 at 11:08 AM, Matt Foley <mf...@hortonworks.com>
>> wrote:
>> > Hi Konstantin,
>> > I'd like to point out two things:
>> > First, I already committed in this thread (email of Thu, Feb 28, 2013 at
>> > 6:01 PM) to providing CI for Windows builds.  So please stop acting like
>> > I'm
>> > resisting this idea or something.
>> > Second, you didn't answer my question, you just kvetched about the
>> > phrasing.
>> > So I ask again:
>> >
>> > Will providing full "test-patch" integration (pre-commit build and unit
>> > test
>> > triggered by Jira "Patch Available" state) satisfy your request for
>> > functionality #1 and #2?  Yes or no, please.
>> >
>> > Thanks,
>> > --Matt
>> >
>> >
>> > On Sat, Mar 2, 2013 at 7:32 PM, Konstantin Shvachko
>> > <sh...@gmail.com>
>> > wrote:
>> >>
>> >> Hi Matt,
>> >>
>> >> On Sat, Mar 2, 2013 at 12:32 PM, Matt Foley <mf...@hortonworks.com>
>> >> wrote:
>> >> > Konstantin,
>> >> > I would like to explore what it would take to remove this perceived
>> >> > impediment --
>> >>
>> >> Glad you decided to explore. Thank you.
>> >>
>> >> > although I reserve the right to argue that this is not
>> >> > pre-requisite to merging the cross-platform support patch.
>> >>
>> >> It's your right indeed. So as mine to question what the platform
>> >> support means for you, which I believe remained unclear.
>> >> I do not impede the change as you should have noticed. My requirement
>> >> comes from my perception of the support, which means to me exactly two
>> >> things:
>> >> 1. The ability to recognise the code is broken for the platform
>> >> 2. The ability to test new patches on the platform
>> >> The latter is problematic, as many noticed in this thread, for those
>> >> whose customary environment does not include Windows.
>> >>
>> >> > If we implemented full "test-patch" support for Windows on trunk,
>> >> > would
>> >> > that
>> >> > fulfill both your items #1 and #2?  Please note that:
>> >> > a) Pushing the "Patch Available" button in Jira shall cause a
>> >> > pre-commit
>> >> > build to start within, I believe, 20 minutes.
>> >> > b) That build keeps logs for both java build and unit tests for
>> >> > several
>> >> > days, that are accessible to all viewers.
>> >>
>> >> In item #1 I mostly asking for the nightly build, which is simpler
>> >> than "test-patch". The latter would be ideal from the platform support
>> >> viewpoint, but it is for the community to decide if we want to add
>> >> extra +3 hours to the build.
>> >> Nightly build in my understanding is triggered by the timer rather
>> >> than by Jira's "submit patch" button.  On Jenkins build configuration
>> >> you can specify it under "Build periodically".
>> >>
>> >> > So, does this provide sufficient on-demand support that we don't have
>> >> > to
>> >> > implement a whole new on-demand VM support structure of some sort for
>> >> > #2
>> >> > (which would be an extraordinary and impractical demand)?
>> >>
>> >> I did not mention VMs. Item #2 means a build, which runs "test-patch"
>> >> target with the file specified by a user (instead of a jira
>> >> attachment).
>> >> When user clicks "Build Now" link a box is displayed where the user
>> >> can enter the file path containing the patch. This can be specified in
>> >> the Build Configuration under "This build is parameterized" by
>> >> choosing AddParameter / FileParameter. The build can run on the same
>> >> Windows machine as the nightly build.
>> >> Such build will let people test their patches for Windows on Jenkins
>> >> if they don't posses a license for the right version of Windows.
>> >> I hope this will not turn into extraordinary or impractical effort.
>> >>
>> >> Thanks,
>> >> --Konst
>> >>
>> >> > Thanks,
>> >> > --Matt
>> >> >
>> >> >
>> >> > On Fri, Mar 1, 2013 at 12:18 PM, Konstantin Shvachko
>> >> > <sh...@gmail.com>
>> >> > wrote:
>> >> >>
>> >> >> -1
>> >> >> We should have a CI infrastructure in place before we can commit to
>> >> >> supporting Windows platform.
>> >> >>
>> >> >> Eric is right Win/Cygwin was supported since day one.
>> >> >> I had a Windows box under my desk running nightly builds back in
>> >> >> 2006-07.
>> >> >> People were irritated but I was filing windows bugs until 0.22
>> >> >> release.
>> >> >> Times changing and I am glad to see wider support for Win platform.
>> >> >>
>> >> >> But in order to make it work you guys need to put the CI process in
>> >> >> place
>> >> >>
>> >> >> 1. windows jenkins build: could be nightly or PreCommit.
>> >> >> - Nightly would mean that changes can be committed to trunk based on
>> >> >> linux PreCommit build. And people will file bugs if the change broke
>> >> >> Windows nightly build.
>> >> >> - PreCommit-win build will mean automatic reporting failed tests to
>> >> >> respective jira blocking commits the same way as it is now with
>> >> >> linux
>> >> >> PreCommit builds.
>> >> >> We should discuss which way is more efficient for developers.
>> >> >>
>> >> >> 2. On-demand-windows Jenkins build.
>> >> >> I see it as a build to which I can attach my patch and the build
>> >> >> will
>> >> >> run my changes on a dedicated windows box.
>> >> >> That way people can test their changes without having personal
>> >> >> windows
>> >> >> nodes.
>> >> >>
>> >> >> I think this is the minimal set of requirement for us to be able to
>> >> >> commit to the new platform.
>> >> >> Right now I see only one windows related build
>> >> >> https://builds.apache.org/view/Hadoop/job/Hadoop-1-win/
>> >> >> Which was failing since Sept 8, 2012 and did not run in the last
>> >> >> month.
>> >> >>
>> >> >> Thanks,
>> >> >> --Konst
>> >> >>
>> >> >> On Thu, Feb 28, 2013 at 8:47 PM, Eric Baldeschwieler
>> >> >> <er...@hortonworks.com> wrote:
>> >> >> > +1 (non-binding)
>> >> >> >
>> >> >> > A few of observations:
>> >> >> >
>> >> >> > - Windows has actually been a supported platform for Hadoop since
>> >> >> > 0.1
>> >> >> > .
>> >> >> > Doug championed supporting windows then and we've continued to do
>> >> >> > it
>> >> >> > with
>> >> >> > varying vigor over time.  To my knowledge we've never made a
>> >> >> > decision
>> >> >> > to
>> >> >> > drop windows support.  The change here is improving our support
>> >> >> > and
>> >> >> > dropping
>> >> >> > the requirement of cigwin.  We had Nutch windows users on the list
>> >> >> > in
>> >> >> > 2006
>> >> >> > and we've been supporting windows FS requirements since inception.
>> >> >> >
>> >> >> > - A little pragmatism will go a long way.  As a community we've
>> >> >> > got
>> >> >> > to
>> >> >> > stay committed to keeping hadoop simple (so it does work on many
>> >> >> > platforms)
>> >> >> > and extending it to take advantage of key emerging OS/hardware
>> >> >> > features,
>> >> >> > such as containers, new FSs, virtualization, flash ...  We should
>> >> >> > all
>> >> >> > plan
>> >> >> > to let new features & optimizations emerge that don't work
>> >> >> > everywhere, if
>> >> >> > they are compelling and central to hadoop's mission of being THE
>> >> >> > best
>> >> >> > fabric
>> >> >> > for storing and processing big data.
>> >> >> >
>> >> >> > - A UI project like KDE has to deal with the MANY differences
>> >> >> > between
>> >> >> > windows and linux UI APIs.  Hadoop faces no such complex challenge
>> >> >> > and hence
>> >> >> > can be maintained from a single codeline IMO.  It is mostly
>> >> >> > abstracted from
>> >> >> > the OS APIs via Java and our design choices.  Where it is not we
>> >> >> > can
>> >> >> > continue to add plugable abstractions.
>> >> >> >
>> >> >
>> >> >
>> >
>> >
>
>

Re: [Vote] Merge branch-trunk-win to trunk

[Matt Foley <mf...@hortonworks.com>]

Konstantine, you have voted -1, and stated some requirements before you'll
withdraw that -1.  As I plan to do work to fulfill those requirements, I
want to make sure that what I'm proposing will, in fact, satisfy you.
 That's why I'm asking, if we implement full "test-patch" integration for
Windows, does it seem to you that that would provide adequate support?

I have learned not to presume that my interpretation is correct.  My
interpretation of item #1 is that test-patch provides pre-commit build, so
it would satisfy item #1.  But rather than assuming that I am interpreting
it correctly, I simply want your agreement that it would, or if not,
clarification why it won't.

Regarding item #2, it is also my interpretation that test-patch provides an
on-demand (perhaps 20-minutes deferred) Jenkins build and unit test, with
logs available to the developer, so it would satisfy item #2.  But rather
than assuming that I am interpreting it correctly, I simply want your
agreement that it would, or if not, clarification why it won't.

In agile terms, you are the Owner of these requirements.  Please give me
owner feedback as to whether my proposed work sounds like it will satisfy
the requirements.

Thank you,
--Matt


On Sun, Mar 3, 2013 at 12:16 PM, Konstantin Shvachko
<sh...@gmail.com>wrote:

> Didn't I explain in details what I am asking for?
>
> Thanks,
> --Konst
>
> On Sun, Mar 3, 2013 at 11:08 AM, Matt Foley <mf...@hortonworks.com>
> wrote:
> > Hi Konstantin,
> > I'd like to point out two things:
> > First, I already committed in this thread (email of Thu, Feb 28, 2013 at
> > 6:01 PM) to providing CI for Windows builds.  So please stop acting like
> I'm
> > resisting this idea or something.
> > Second, you didn't answer my question, you just kvetched about the
> phrasing.
> > So I ask again:
> >
> > Will providing full "test-patch" integration (pre-commit build and unit
> test
> > triggered by Jira "Patch Available" state) satisfy your request for
> > functionality #1 and #2?  Yes or no, please.
> >
> > Thanks,
> > --Matt
> >
> >
> > On Sat, Mar 2, 2013 at 7:32 PM, Konstantin Shvachko <
> shv.hadoop@gmail.com>
> > wrote:
> >>
> >> Hi Matt,
> >>
> >> On Sat, Mar 2, 2013 at 12:32 PM, Matt Foley <mf...@hortonworks.com>
> >> wrote:
> >> > Konstantin,
> >> > I would like to explore what it would take to remove this perceived
> >> > impediment --
> >>
> >> Glad you decided to explore. Thank you.
> >>
> >> > although I reserve the right to argue that this is not
> >> > pre-requisite to merging the cross-platform support patch.
> >>
> >> It's your right indeed. So as mine to question what the platform
> >> support means for you, which I believe remained unclear.
> >> I do not impede the change as you should have noticed. My requirement
> >> comes from my perception of the support, which means to me exactly two
> >> things:
> >> 1. The ability to recognise the code is broken for the platform
> >> 2. The ability to test new patches on the platform
> >> The latter is problematic, as many noticed in this thread, for those
> >> whose customary environment does not include Windows.
> >>
> >> > If we implemented full "test-patch" support for Windows on trunk,
> would
> >> > that
> >> > fulfill both your items #1 and #2?  Please note that:
> >> > a) Pushing the "Patch Available" button in Jira shall cause a
> pre-commit
> >> > build to start within, I believe, 20 minutes.
> >> > b) That build keeps logs for both java build and unit tests for
> several
> >> > days, that are accessible to all viewers.
> >>
> >> In item #1 I mostly asking for the nightly build, which is simpler
> >> than "test-patch". The latter would be ideal from the platform support
> >> viewpoint, but it is for the community to decide if we want to add
> >> extra +3 hours to the build.
> >> Nightly build in my understanding is triggered by the timer rather
> >> than by Jira's "submit patch" button.  On Jenkins build configuration
> >> you can specify it under "Build periodically".
> >>
> >> > So, does this provide sufficient on-demand support that we don't have
> to
> >> > implement a whole new on-demand VM support structure of some sort for
> #2
> >> > (which would be an extraordinary and impractical demand)?
> >>
> >> I did not mention VMs. Item #2 means a build, which runs "test-patch"
> >> target with the file specified by a user (instead of a jira
> >> attachment).
> >> When user clicks "Build Now" link a box is displayed where the user
> >> can enter the file path containing the patch. This can be specified in
> >> the Build Configuration under "This build is parameterized" by
> >> choosing AddParameter / FileParameter. The build can run on the same
> >> Windows machine as the nightly build.
> >> Such build will let people test their patches for Windows on Jenkins
> >> if they don't posses a license for the right version of Windows.
> >> I hope this will not turn into extraordinary or impractical effort.
> >>
> >> Thanks,
> >> --Konst
> >>
> >> > Thanks,
> >> > --Matt
> >> >
> >> >
> >> > On Fri, Mar 1, 2013 at 12:18 PM, Konstantin Shvachko
> >> > <sh...@gmail.com>
> >> > wrote:
> >> >>
> >> >> -1
> >> >> We should have a CI infrastructure in place before we can commit to
> >> >> supporting Windows platform.
> >> >>
> >> >> Eric is right Win/Cygwin was supported since day one.
> >> >> I had a Windows box under my desk running nightly builds back in
> >> >> 2006-07.
> >> >> People were irritated but I was filing windows bugs until 0.22
> release.
> >> >> Times changing and I am glad to see wider support for Win platform.
> >> >>
> >> >> But in order to make it work you guys need to put the CI process in
> >> >> place
> >> >>
> >> >> 1. windows jenkins build: could be nightly or PreCommit.
> >> >> - Nightly would mean that changes can be committed to trunk based on
> >> >> linux PreCommit build. And people will file bugs if the change broke
> >> >> Windows nightly build.
> >> >> - PreCommit-win build will mean automatic reporting failed tests to
> >> >> respective jira blocking commits the same way as it is now with linux
> >> >> PreCommit builds.
> >> >> We should discuss which way is more efficient for developers.
> >> >>
> >> >> 2. On-demand-windows Jenkins build.
> >> >> I see it as a build to which I can attach my patch and the build will
> >> >> run my changes on a dedicated windows box.
> >> >> That way people can test their changes without having personal
> windows
> >> >> nodes.
> >> >>
> >> >> I think this is the minimal set of requirement for us to be able to
> >> >> commit to the new platform.
> >> >> Right now I see only one windows related build
> >> >> https://builds.apache.org/view/Hadoop/job/Hadoop-1-win/
> >> >> Which was failing since Sept 8, 2012 and did not run in the last
> month.
> >> >>
> >> >> Thanks,
> >> >> --Konst
> >> >>
> >> >> On Thu, Feb 28, 2013 at 8:47 PM, Eric Baldeschwieler
> >> >> <er...@hortonworks.com> wrote:
> >> >> > +1 (non-binding)
> >> >> >
> >> >> > A few of observations:
> >> >> >
> >> >> > - Windows has actually been a supported platform for Hadoop since
> 0.1
> >> >> > .
> >> >> > Doug championed supporting windows then and we've continued to do
> it
> >> >> > with
> >> >> > varying vigor over time.  To my knowledge we've never made a
> decision
> >> >> > to
> >> >> > drop windows support.  The change here is improving our support and
> >> >> > dropping
> >> >> > the requirement of cigwin.  We had Nutch windows users on the list
> in
> >> >> > 2006
> >> >> > and we've been supporting windows FS requirements since inception.
> >> >> >
> >> >> > - A little pragmatism will go a long way.  As a community we've got
> >> >> > to
> >> >> > stay committed to keeping hadoop simple (so it does work on many
> >> >> > platforms)
> >> >> > and extending it to take advantage of key emerging OS/hardware
> >> >> > features,
> >> >> > such as containers, new FSs, virtualization, flash ...  We should
> all
> >> >> > plan
> >> >> > to let new features & optimizations emerge that don't work
> >> >> > everywhere, if
> >> >> > they are compelling and central to hadoop's mission of being THE
> best
> >> >> > fabric
> >> >> > for storing and processing big data.
> >> >> >
> >> >> > - A UI project like KDE has to deal with the MANY differences
> between
> >> >> > windows and linux UI APIs.  Hadoop faces no such complex challenge
> >> >> > and hence
> >> >> > can be maintained from a single codeline IMO.  It is mostly
> >> >> > abstracted from
> >> >> > the OS APIs via Java and our design choices.  Where it is not we
> can
> >> >> > continue to add plugable abstractions.
> >> >> >
> >> >
> >> >
> >
> >
>

Re: [Vote] Merge branch-trunk-win to trunk

[Matt Foley <mf...@hortonworks.com>]

Konstantine, you have voted -1, and stated some requirements before you'll
withdraw that -1.  As I plan to do work to fulfill those requirements, I
want to make sure that what I'm proposing will, in fact, satisfy you.
 That's why I'm asking, if we implement full "test-patch" integration for
Windows, does it seem to you that that would provide adequate support?

I have learned not to presume that my interpretation is correct.  My
interpretation of item #1 is that test-patch provides pre-commit build, so
it would satisfy item #1.  But rather than assuming that I am interpreting
it correctly, I simply want your agreement that it would, or if not,
clarification why it won't.

Regarding item #2, it is also my interpretation that test-patch provides an
on-demand (perhaps 20-minutes deferred) Jenkins build and unit test, with
logs available to the developer, so it would satisfy item #2.  But rather
than assuming that I am interpreting it correctly, I simply want your
agreement that it would, or if not, clarification why it won't.

In agile terms, you are the Owner of these requirements.  Please give me
owner feedback as to whether my proposed work sounds like it will satisfy
the requirements.

Thank you,
--Matt


On Sun, Mar 3, 2013 at 12:16 PM, Konstantin Shvachko
<sh...@gmail.com>wrote:

> Didn't I explain in details what I am asking for?
>
> Thanks,
> --Konst
>
> On Sun, Mar 3, 2013 at 11:08 AM, Matt Foley <mf...@hortonworks.com>
> wrote:
> > Hi Konstantin,
> > I'd like to point out two things:
> > First, I already committed in this thread (email of Thu, Feb 28, 2013 at
> > 6:01 PM) to providing CI for Windows builds.  So please stop acting like
> I'm
> > resisting this idea or something.
> > Second, you didn't answer my question, you just kvetched about the
> phrasing.
> > So I ask again:
> >
> > Will providing full "test-patch" integration (pre-commit build and unit
> test
> > triggered by Jira "Patch Available" state) satisfy your request for
> > functionality #1 and #2?  Yes or no, please.
> >
> > Thanks,
> > --Matt
> >
> >
> > On Sat, Mar 2, 2013 at 7:32 PM, Konstantin Shvachko <
> shv.hadoop@gmail.com>
> > wrote:
> >>
> >> Hi Matt,
> >>
> >> On Sat, Mar 2, 2013 at 12:32 PM, Matt Foley <mf...@hortonworks.com>
> >> wrote:
> >> > Konstantin,
> >> > I would like to explore what it would take to remove this perceived
> >> > impediment --
> >>
> >> Glad you decided to explore. Thank you.
> >>
> >> > although I reserve the right to argue that this is not
> >> > pre-requisite to merging the cross-platform support patch.
> >>
> >> It's your right indeed. So as mine to question what the platform
> >> support means for you, which I believe remained unclear.
> >> I do not impede the change as you should have noticed. My requirement
> >> comes from my perception of the support, which means to me exactly two
> >> things:
> >> 1. The ability to recognise the code is broken for the platform
> >> 2. The ability to test new patches on the platform
> >> The latter is problematic, as many noticed in this thread, for those
> >> whose customary environment does not include Windows.
> >>
> >> > If we implemented full "test-patch" support for Windows on trunk,
> would
> >> > that
> >> > fulfill both your items #1 and #2?  Please note that:
> >> > a) Pushing the "Patch Available" button in Jira shall cause a
> pre-commit
> >> > build to start within, I believe, 20 minutes.
> >> > b) That build keeps logs for both java build and unit tests for
> several
> >> > days, that are accessible to all viewers.
> >>
> >> In item #1 I mostly asking for the nightly build, which is simpler
> >> than "test-patch". The latter would be ideal from the platform support
> >> viewpoint, but it is for the community to decide if we want to add
> >> extra +3 hours to the build.
> >> Nightly build in my understanding is triggered by the timer rather
> >> than by Jira's "submit patch" button.  On Jenkins build configuration
> >> you can specify it under "Build periodically".
> >>
> >> > So, does this provide sufficient on-demand support that we don't have
> to
> >> > implement a whole new on-demand VM support structure of some sort for
> #2
> >> > (which would be an extraordinary and impractical demand)?
> >>
> >> I did not mention VMs. Item #2 means a build, which runs "test-patch"
> >> target with the file specified by a user (instead of a jira
> >> attachment).
> >> When user clicks "Build Now" link a box is displayed where the user
> >> can enter the file path containing the patch. This can be specified in
> >> the Build Configuration under "This build is parameterized" by
> >> choosing AddParameter / FileParameter. The build can run on the same
> >> Windows machine as the nightly build.
> >> Such build will let people test their patches for Windows on Jenkins
> >> if they don't posses a license for the right version of Windows.
> >> I hope this will not turn into extraordinary or impractical effort.
> >>
> >> Thanks,
> >> --Konst
> >>
> >> > Thanks,
> >> > --Matt
> >> >
> >> >
> >> > On Fri, Mar 1, 2013 at 12:18 PM, Konstantin Shvachko
> >> > <sh...@gmail.com>
> >> > wrote:
> >> >>
> >> >> -1
> >> >> We should have a CI infrastructure in place before we can commit to
> >> >> supporting Windows platform.
> >> >>
> >> >> Eric is right Win/Cygwin was supported since day one.
> >> >> I had a Windows box under my desk running nightly builds back in
> >> >> 2006-07.
> >> >> People were irritated but I was filing windows bugs until 0.22
> release.
> >> >> Times changing and I am glad to see wider support for Win platform.
> >> >>
> >> >> But in order to make it work you guys need to put the CI process in
> >> >> place
> >> >>
> >> >> 1. windows jenkins build: could be nightly or PreCommit.
> >> >> - Nightly would mean that changes can be committed to trunk based on
> >> >> linux PreCommit build. And people will file bugs if the change broke
> >> >> Windows nightly build.
> >> >> - PreCommit-win build will mean automatic reporting failed tests to
> >> >> respective jira blocking commits the same way as it is now with linux
> >> >> PreCommit builds.
> >> >> We should discuss which way is more efficient for developers.
> >> >>
> >> >> 2. On-demand-windows Jenkins build.
> >> >> I see it as a build to which I can attach my patch and the build will
> >> >> run my changes on a dedicated windows box.
> >> >> That way people can test their changes without having personal
> windows
> >> >> nodes.
> >> >>
> >> >> I think this is the minimal set of requirement for us to be able to
> >> >> commit to the new platform.
> >> >> Right now I see only one windows related build
> >> >> https://builds.apache.org/view/Hadoop/job/Hadoop-1-win/
> >> >> Which was failing since Sept 8, 2012 and did not run in the last
> month.
> >> >>
> >> >> Thanks,
> >> >> --Konst
> >> >>
> >> >> On Thu, Feb 28, 2013 at 8:47 PM, Eric Baldeschwieler
> >> >> <er...@hortonworks.com> wrote:
> >> >> > +1 (non-binding)
> >> >> >
> >> >> > A few of observations:
> >> >> >
> >> >> > - Windows has actually been a supported platform for Hadoop since
> 0.1
> >> >> > .
> >> >> > Doug championed supporting windows then and we've continued to do
> it
> >> >> > with
> >> >> > varying vigor over time.  To my knowledge we've never made a
> decision
> >> >> > to
> >> >> > drop windows support.  The change here is improving our support and
> >> >> > dropping
> >> >> > the requirement of cigwin.  We had Nutch windows users on the list
> in
> >> >> > 2006
> >> >> > and we've been supporting windows FS requirements since inception.
> >> >> >
> >> >> > - A little pragmatism will go a long way.  As a community we've got
> >> >> > to
> >> >> > stay committed to keeping hadoop simple (so it does work on many
> >> >> > platforms)
> >> >> > and extending it to take advantage of key emerging OS/hardware
> >> >> > features,
> >> >> > such as containers, new FSs, virtualization, flash ...  We should
> all
> >> >> > plan
> >> >> > to let new features & optimizations emerge that don't work
> >> >> > everywhere, if
> >> >> > they are compelling and central to hadoop's mission of being THE
> best
> >> >> > fabric
> >> >> > for storing and processing big data.
> >> >> >
> >> >> > - A UI project like KDE has to deal with the MANY differences
> between
> >> >> > windows and linux UI APIs.  Hadoop faces no such complex challenge
> >> >> > and hence
> >> >> > can be maintained from a single codeline IMO.  It is mostly
> >> >> > abstracted from
> >> >> > the OS APIs via Java and our design choices.  Where it is not we
> can
> >> >> > continue to add plugable abstractions.
> >> >> >
> >> >
> >> >
> >
> >
>

Re: [Vote] Merge branch-trunk-win to trunk

[Matt Foley <mf...@hortonworks.com>]

Konstantine, you have voted -1, and stated some requirements before you'll
withdraw that -1.  As I plan to do work to fulfill those requirements, I
want to make sure that what I'm proposing will, in fact, satisfy you.
 That's why I'm asking, if we implement full "test-patch" integration for
Windows, does it seem to you that that would provide adequate support?

I have learned not to presume that my interpretation is correct.  My
interpretation of item #1 is that test-patch provides pre-commit build, so
it would satisfy item #1.  But rather than assuming that I am interpreting
it correctly, I simply want your agreement that it would, or if not,
clarification why it won't.

Regarding item #2, it is also my interpretation that test-patch provides an
on-demand (perhaps 20-minutes deferred) Jenkins build and unit test, with
logs available to the developer, so it would satisfy item #2.  But rather
than assuming that I am interpreting it correctly, I simply want your
agreement that it would, or if not, clarification why it won't.

In agile terms, you are the Owner of these requirements.  Please give me
owner feedback as to whether my proposed work sounds like it will satisfy
the requirements.

Thank you,
--Matt


On Sun, Mar 3, 2013 at 12:16 PM, Konstantin Shvachko
<sh...@gmail.com>wrote:

> Didn't I explain in details what I am asking for?
>
> Thanks,
> --Konst
>
> On Sun, Mar 3, 2013 at 11:08 AM, Matt Foley <mf...@hortonworks.com>
> wrote:
> > Hi Konstantin,
> > I'd like to point out two things:
> > First, I already committed in this thread (email of Thu, Feb 28, 2013 at
> > 6:01 PM) to providing CI for Windows builds.  So please stop acting like
> I'm
> > resisting this idea or something.
> > Second, you didn't answer my question, you just kvetched about the
> phrasing.
> > So I ask again:
> >
> > Will providing full "test-patch" integration (pre-commit build and unit
> test
> > triggered by Jira "Patch Available" state) satisfy your request for
> > functionality #1 and #2?  Yes or no, please.
> >
> > Thanks,
> > --Matt
> >
> >
> > On Sat, Mar 2, 2013 at 7:32 PM, Konstantin Shvachko <
> shv.hadoop@gmail.com>
> > wrote:
> >>
> >> Hi Matt,
> >>
> >> On Sat, Mar 2, 2013 at 12:32 PM, Matt Foley <mf...@hortonworks.com>
> >> wrote:
> >> > Konstantin,
> >> > I would like to explore what it would take to remove this perceived
> >> > impediment --
> >>
> >> Glad you decided to explore. Thank you.
> >>
> >> > although I reserve the right to argue that this is not
> >> > pre-requisite to merging the cross-platform support patch.
> >>
> >> It's your right indeed. So as mine to question what the platform
> >> support means for you, which I believe remained unclear.
> >> I do not impede the change as you should have noticed. My requirement
> >> comes from my perception of the support, which means to me exactly two
> >> things:
> >> 1. The ability to recognise the code is broken for the platform
> >> 2. The ability to test new patches on the platform
> >> The latter is problematic, as many noticed in this thread, for those
> >> whose customary environment does not include Windows.
> >>
> >> > If we implemented full "test-patch" support for Windows on trunk,
> would
> >> > that
> >> > fulfill both your items #1 and #2?  Please note that:
> >> > a) Pushing the "Patch Available" button in Jira shall cause a
> pre-commit
> >> > build to start within, I believe, 20 minutes.
> >> > b) That build keeps logs for both java build and unit tests for
> several
> >> > days, that are accessible to all viewers.
> >>
> >> In item #1 I mostly asking for the nightly build, which is simpler
> >> than "test-patch". The latter would be ideal from the platform support
> >> viewpoint, but it is for the community to decide if we want to add
> >> extra +3 hours to the build.
> >> Nightly build in my understanding is triggered by the timer rather
> >> than by Jira's "submit patch" button.  On Jenkins build configuration
> >> you can specify it under "Build periodically".
> >>
> >> > So, does this provide sufficient on-demand support that we don't have
> to
> >> > implement a whole new on-demand VM support structure of some sort for
> #2
> >> > (which would be an extraordinary and impractical demand)?
> >>
> >> I did not mention VMs. Item #2 means a build, which runs "test-patch"
> >> target with the file specified by a user (instead of a jira
> >> attachment).
> >> When user clicks "Build Now" link a box is displayed where the user
> >> can enter the file path containing the patch. This can be specified in
> >> the Build Configuration under "This build is parameterized" by
> >> choosing AddParameter / FileParameter. The build can run on the same
> >> Windows machine as the nightly build.
> >> Such build will let people test their patches for Windows on Jenkins
> >> if they don't posses a license for the right version of Windows.
> >> I hope this will not turn into extraordinary or impractical effort.
> >>
> >> Thanks,
> >> --Konst
> >>
> >> > Thanks,
> >> > --Matt
> >> >
> >> >
> >> > On Fri, Mar 1, 2013 at 12:18 PM, Konstantin Shvachko
> >> > <sh...@gmail.com>
> >> > wrote:
> >> >>
> >> >> -1
> >> >> We should have a CI infrastructure in place before we can commit to
> >> >> supporting Windows platform.
> >> >>
> >> >> Eric is right Win/Cygwin was supported since day one.
> >> >> I had a Windows box under my desk running nightly builds back in
> >> >> 2006-07.
> >> >> People were irritated but I was filing windows bugs until 0.22
> release.
> >> >> Times changing and I am glad to see wider support for Win platform.
> >> >>
> >> >> But in order to make it work you guys need to put the CI process in
> >> >> place
> >> >>
> >> >> 1. windows jenkins build: could be nightly or PreCommit.
> >> >> - Nightly would mean that changes can be committed to trunk based on
> >> >> linux PreCommit build. And people will file bugs if the change broke
> >> >> Windows nightly build.
> >> >> - PreCommit-win build will mean automatic reporting failed tests to
> >> >> respective jira blocking commits the same way as it is now with linux
> >> >> PreCommit builds.
> >> >> We should discuss which way is more efficient for developers.
> >> >>
> >> >> 2. On-demand-windows Jenkins build.
> >> >> I see it as a build to which I can attach my patch and the build will
> >> >> run my changes on a dedicated windows box.
> >> >> That way people can test their changes without having personal
> windows
> >> >> nodes.
> >> >>
> >> >> I think this is the minimal set of requirement for us to be able to
> >> >> commit to the new platform.
> >> >> Right now I see only one windows related build
> >> >> https://builds.apache.org/view/Hadoop/job/Hadoop-1-win/
> >> >> Which was failing since Sept 8, 2012 and did not run in the last
> month.
> >> >>
> >> >> Thanks,
> >> >> --Konst
> >> >>
> >> >> On Thu, Feb 28, 2013 at 8:47 PM, Eric Baldeschwieler
> >> >> <er...@hortonworks.com> wrote:
> >> >> > +1 (non-binding)
> >> >> >
> >> >> > A few of observations:
> >> >> >
> >> >> > - Windows has actually been a supported platform for Hadoop since
> 0.1
> >> >> > .
> >> >> > Doug championed supporting windows then and we've continued to do
> it
> >> >> > with
> >> >> > varying vigor over time.  To my knowledge we've never made a
> decision
> >> >> > to
> >> >> > drop windows support.  The change here is improving our support and
> >> >> > dropping
> >> >> > the requirement of cigwin.  We had Nutch windows users on the list
> in
> >> >> > 2006
> >> >> > and we've been supporting windows FS requirements since inception.
> >> >> >
> >> >> > - A little pragmatism will go a long way.  As a community we've got
> >> >> > to
> >> >> > stay committed to keeping hadoop simple (so it does work on many
> >> >> > platforms)
> >> >> > and extending it to take advantage of key emerging OS/hardware
> >> >> > features,
> >> >> > such as containers, new FSs, virtualization, flash ...  We should
> all
> >> >> > plan
> >> >> > to let new features & optimizations emerge that don't work
> >> >> > everywhere, if
> >> >> > they are compelling and central to hadoop's mission of being THE
> best
> >> >> > fabric
> >> >> > for storing and processing big data.
> >> >> >
> >> >> > - A UI project like KDE has to deal with the MANY differences
> between
> >> >> > windows and linux UI APIs.  Hadoop faces no such complex challenge
> >> >> > and hence
> >> >> > can be maintained from a single codeline IMO.  It is mostly
> >> >> > abstracted from
> >> >> > the OS APIs via Java and our design choices.  Where it is not we
> can
> >> >> > continue to add plugable abstractions.
> >> >> >
> >> >
> >> >
> >
> >
>

Re: [Vote] Merge branch-trunk-win to trunk

[Konstantin Shvachko <sh...@gmail.com>]

Didn't I explain in details what I am asking for?

Thanks,
--Konst

On Sun, Mar 3, 2013 at 11:08 AM, Matt Foley <mf...@hortonworks.com> wrote:
> Hi Konstantin,
> I'd like to point out two things:
> First, I already committed in this thread (email of Thu, Feb 28, 2013 at
> 6:01 PM) to providing CI for Windows builds.  So please stop acting like I'm
> resisting this idea or something.
> Second, you didn't answer my question, you just kvetched about the phrasing.
> So I ask again:
>
> Will providing full "test-patch" integration (pre-commit build and unit test
> triggered by Jira "Patch Available" state) satisfy your request for
> functionality #1 and #2?  Yes or no, please.
>
> Thanks,
> --Matt
>
>
> On Sat, Mar 2, 2013 at 7:32 PM, Konstantin Shvachko <sh...@gmail.com>
> wrote:
>>
>> Hi Matt,
>>
>> On Sat, Mar 2, 2013 at 12:32 PM, Matt Foley <mf...@hortonworks.com>
>> wrote:
>> > Konstantin,
>> > I would like to explore what it would take to remove this perceived
>> > impediment --
>>
>> Glad you decided to explore. Thank you.
>>
>> > although I reserve the right to argue that this is not
>> > pre-requisite to merging the cross-platform support patch.
>>
>> It's your right indeed. So as mine to question what the platform
>> support means for you, which I believe remained unclear.
>> I do not impede the change as you should have noticed. My requirement
>> comes from my perception of the support, which means to me exactly two
>> things:
>> 1. The ability to recognise the code is broken for the platform
>> 2. The ability to test new patches on the platform
>> The latter is problematic, as many noticed in this thread, for those
>> whose customary environment does not include Windows.
>>
>> > If we implemented full "test-patch" support for Windows on trunk, would
>> > that
>> > fulfill both your items #1 and #2?  Please note that:
>> > a) Pushing the "Patch Available" button in Jira shall cause a pre-commit
>> > build to start within, I believe, 20 minutes.
>> > b) That build keeps logs for both java build and unit tests for several
>> > days, that are accessible to all viewers.
>>
>> In item #1 I mostly asking for the nightly build, which is simpler
>> than "test-patch". The latter would be ideal from the platform support
>> viewpoint, but it is for the community to decide if we want to add
>> extra +3 hours to the build.
>> Nightly build in my understanding is triggered by the timer rather
>> than by Jira's "submit patch" button.  On Jenkins build configuration
>> you can specify it under "Build periodically".
>>
>> > So, does this provide sufficient on-demand support that we don't have to
>> > implement a whole new on-demand VM support structure of some sort for #2
>> > (which would be an extraordinary and impractical demand)?
>>
>> I did not mention VMs. Item #2 means a build, which runs "test-patch"
>> target with the file specified by a user (instead of a jira
>> attachment).
>> When user clicks "Build Now" link a box is displayed where the user
>> can enter the file path containing the patch. This can be specified in
>> the Build Configuration under "This build is parameterized" by
>> choosing AddParameter / FileParameter. The build can run on the same
>> Windows machine as the nightly build.
>> Such build will let people test their patches for Windows on Jenkins
>> if they don't posses a license for the right version of Windows.
>> I hope this will not turn into extraordinary or impractical effort.
>>
>> Thanks,
>> --Konst
>>
>> > Thanks,
>> > --Matt
>> >
>> >
>> > On Fri, Mar 1, 2013 at 12:18 PM, Konstantin Shvachko
>> > <sh...@gmail.com>
>> > wrote:
>> >>
>> >> -1
>> >> We should have a CI infrastructure in place before we can commit to
>> >> supporting Windows platform.
>> >>
>> >> Eric is right Win/Cygwin was supported since day one.
>> >> I had a Windows box under my desk running nightly builds back in
>> >> 2006-07.
>> >> People were irritated but I was filing windows bugs until 0.22 release.
>> >> Times changing and I am glad to see wider support for Win platform.
>> >>
>> >> But in order to make it work you guys need to put the CI process in
>> >> place
>> >>
>> >> 1. windows jenkins build: could be nightly or PreCommit.
>> >> - Nightly would mean that changes can be committed to trunk based on
>> >> linux PreCommit build. And people will file bugs if the change broke
>> >> Windows nightly build.
>> >> - PreCommit-win build will mean automatic reporting failed tests to
>> >> respective jira blocking commits the same way as it is now with linux
>> >> PreCommit builds.
>> >> We should discuss which way is more efficient for developers.
>> >>
>> >> 2. On-demand-windows Jenkins build.
>> >> I see it as a build to which I can attach my patch and the build will
>> >> run my changes on a dedicated windows box.
>> >> That way people can test their changes without having personal windows
>> >> nodes.
>> >>
>> >> I think this is the minimal set of requirement for us to be able to
>> >> commit to the new platform.
>> >> Right now I see only one windows related build
>> >> https://builds.apache.org/view/Hadoop/job/Hadoop-1-win/
>> >> Which was failing since Sept 8, 2012 and did not run in the last month.
>> >>
>> >> Thanks,
>> >> --Konst
>> >>
>> >> On Thu, Feb 28, 2013 at 8:47 PM, Eric Baldeschwieler
>> >> <er...@hortonworks.com> wrote:
>> >> > +1 (non-binding)
>> >> >
>> >> > A few of observations:
>> >> >
>> >> > - Windows has actually been a supported platform for Hadoop since 0.1
>> >> > .
>> >> > Doug championed supporting windows then and we've continued to do it
>> >> > with
>> >> > varying vigor over time.  To my knowledge we've never made a decision
>> >> > to
>> >> > drop windows support.  The change here is improving our support and
>> >> > dropping
>> >> > the requirement of cigwin.  We had Nutch windows users on the list in
>> >> > 2006
>> >> > and we've been supporting windows FS requirements since inception.
>> >> >
>> >> > - A little pragmatism will go a long way.  As a community we've got
>> >> > to
>> >> > stay committed to keeping hadoop simple (so it does work on many
>> >> > platforms)
>> >> > and extending it to take advantage of key emerging OS/hardware
>> >> > features,
>> >> > such as containers, new FSs, virtualization, flash ...  We should all
>> >> > plan
>> >> > to let new features & optimizations emerge that don't work
>> >> > everywhere, if
>> >> > they are compelling and central to hadoop's mission of being THE best
>> >> > fabric
>> >> > for storing and processing big data.
>> >> >
>> >> > - A UI project like KDE has to deal with the MANY differences between
>> >> > windows and linux UI APIs.  Hadoop faces no such complex challenge
>> >> > and hence
>> >> > can be maintained from a single codeline IMO.  It is mostly
>> >> > abstracted from
>> >> > the OS APIs via Java and our design choices.  Where it is not we can
>> >> > continue to add plugable abstractions.
>> >> >
>> >
>> >
>
>

Re: [Vote] Merge branch-trunk-win to trunk

[Konstantin Shvachko <sh...@gmail.com>]

Didn't I explain in details what I am asking for?

Thanks,
--Konst

On Sun, Mar 3, 2013 at 11:08 AM, Matt Foley <mf...@hortonworks.com> wrote:
> Hi Konstantin,
> I'd like to point out two things:
> First, I already committed in this thread (email of Thu, Feb 28, 2013 at
> 6:01 PM) to providing CI for Windows builds.  So please stop acting like I'm
> resisting this idea or something.
> Second, you didn't answer my question, you just kvetched about the phrasing.
> So I ask again:
>
> Will providing full "test-patch" integration (pre-commit build and unit test
> triggered by Jira "Patch Available" state) satisfy your request for
> functionality #1 and #2?  Yes or no, please.
>
> Thanks,
> --Matt
>
>
> On Sat, Mar 2, 2013 at 7:32 PM, Konstantin Shvachko <sh...@gmail.com>
> wrote:
>>
>> Hi Matt,
>>
>> On Sat, Mar 2, 2013 at 12:32 PM, Matt Foley <mf...@hortonworks.com>
>> wrote:
>> > Konstantin,
>> > I would like to explore what it would take to remove this perceived
>> > impediment --
>>
>> Glad you decided to explore. Thank you.
>>
>> > although I reserve the right to argue that this is not
>> > pre-requisite to merging the cross-platform support patch.
>>
>> It's your right indeed. So as mine to question what the platform
>> support means for you, which I believe remained unclear.
>> I do not impede the change as you should have noticed. My requirement
>> comes from my perception of the support, which means to me exactly two
>> things:
>> 1. The ability to recognise the code is broken for the platform
>> 2. The ability to test new patches on the platform
>> The latter is problematic, as many noticed in this thread, for those
>> whose customary environment does not include Windows.
>>
>> > If we implemented full "test-patch" support for Windows on trunk, would
>> > that
>> > fulfill both your items #1 and #2?  Please note that:
>> > a) Pushing the "Patch Available" button in Jira shall cause a pre-commit
>> > build to start within, I believe, 20 minutes.
>> > b) That build keeps logs for both java build and unit tests for several
>> > days, that are accessible to all viewers.
>>
>> In item #1 I mostly asking for the nightly build, which is simpler
>> than "test-patch". The latter would be ideal from the platform support
>> viewpoint, but it is for the community to decide if we want to add
>> extra +3 hours to the build.
>> Nightly build in my understanding is triggered by the timer rather
>> than by Jira's "submit patch" button.  On Jenkins build configuration
>> you can specify it under "Build periodically".
>>
>> > So, does this provide sufficient on-demand support that we don't have to
>> > implement a whole new on-demand VM support structure of some sort for #2
>> > (which would be an extraordinary and impractical demand)?
>>
>> I did not mention VMs. Item #2 means a build, which runs "test-patch"
>> target with the file specified by a user (instead of a jira
>> attachment).
>> When user clicks "Build Now" link a box is displayed where the user
>> can enter the file path containing the patch. This can be specified in
>> the Build Configuration under "This build is parameterized" by
>> choosing AddParameter / FileParameter. The build can run on the same
>> Windows machine as the nightly build.
>> Such build will let people test their patches for Windows on Jenkins
>> if they don't posses a license for the right version of Windows.
>> I hope this will not turn into extraordinary or impractical effort.
>>
>> Thanks,
>> --Konst
>>
>> > Thanks,
>> > --Matt
>> >
>> >
>> > On Fri, Mar 1, 2013 at 12:18 PM, Konstantin Shvachko
>> > <sh...@gmail.com>
>> > wrote:
>> >>
>> >> -1
>> >> We should have a CI infrastructure in place before we can commit to
>> >> supporting Windows platform.
>> >>
>> >> Eric is right Win/Cygwin was supported since day one.
>> >> I had a Windows box under my desk running nightly builds back in
>> >> 2006-07.
>> >> People were irritated but I was filing windows bugs until 0.22 release.
>> >> Times changing and I am glad to see wider support for Win platform.
>> >>
>> >> But in order to make it work you guys need to put the CI process in
>> >> place
>> >>
>> >> 1. windows jenkins build: could be nightly or PreCommit.
>> >> - Nightly would mean that changes can be committed to trunk based on
>> >> linux PreCommit build. And people will file bugs if the change broke
>> >> Windows nightly build.
>> >> - PreCommit-win build will mean automatic reporting failed tests to
>> >> respective jira blocking commits the same way as it is now with linux
>> >> PreCommit builds.
>> >> We should discuss which way is more efficient for developers.
>> >>
>> >> 2. On-demand-windows Jenkins build.
>> >> I see it as a build to which I can attach my patch and the build will
>> >> run my changes on a dedicated windows box.
>> >> That way people can test their changes without having personal windows
>> >> nodes.
>> >>
>> >> I think this is the minimal set of requirement for us to be able to
>> >> commit to the new platform.
>> >> Right now I see only one windows related build
>> >> https://builds.apache.org/view/Hadoop/job/Hadoop-1-win/
>> >> Which was failing since Sept 8, 2012 and did not run in the last month.
>> >>
>> >> Thanks,
>> >> --Konst
>> >>
>> >> On Thu, Feb 28, 2013 at 8:47 PM, Eric Baldeschwieler
>> >> <er...@hortonworks.com> wrote:
>> >> > +1 (non-binding)
>> >> >
>> >> > A few of observations:
>> >> >
>> >> > - Windows has actually been a supported platform for Hadoop since 0.1
>> >> > .
>> >> > Doug championed supporting windows then and we've continued to do it
>> >> > with
>> >> > varying vigor over time.  To my knowledge we've never made a decision
>> >> > to
>> >> > drop windows support.  The change here is improving our support and
>> >> > dropping
>> >> > the requirement of cigwin.  We had Nutch windows users on the list in
>> >> > 2006
>> >> > and we've been supporting windows FS requirements since inception.
>> >> >
>> >> > - A little pragmatism will go a long way.  As a community we've got
>> >> > to
>> >> > stay committed to keeping hadoop simple (so it does work on many
>> >> > platforms)
>> >> > and extending it to take advantage of key emerging OS/hardware
>> >> > features,
>> >> > such as containers, new FSs, virtualization, flash ...  We should all
>> >> > plan
>> >> > to let new features & optimizations emerge that don't work
>> >> > everywhere, if
>> >> > they are compelling and central to hadoop's mission of being THE best
>> >> > fabric
>> >> > for storing and processing big data.
>> >> >
>> >> > - A UI project like KDE has to deal with the MANY differences between
>> >> > windows and linux UI APIs.  Hadoop faces no such complex challenge
>> >> > and hence
>> >> > can be maintained from a single codeline IMO.  It is mostly
>> >> > abstracted from
>> >> > the OS APIs via Java and our design choices.  Where it is not we can
>> >> > continue to add plugable abstractions.
>> >> >
>> >
>> >
>
>

Re: [Vote] Merge branch-trunk-win to trunk

[Konstantin Shvachko <sh...@gmail.com>]

Didn't I explain in details what I am asking for?

Thanks,
--Konst

On Sun, Mar 3, 2013 at 11:08 AM, Matt Foley <mf...@hortonworks.com> wrote:
> Hi Konstantin,
> I'd like to point out two things:
> First, I already committed in this thread (email of Thu, Feb 28, 2013 at
> 6:01 PM) to providing CI for Windows builds.  So please stop acting like I'm
> resisting this idea or something.
> Second, you didn't answer my question, you just kvetched about the phrasing.
> So I ask again:
>
> Will providing full "test-patch" integration (pre-commit build and unit test
> triggered by Jira "Patch Available" state) satisfy your request for
> functionality #1 and #2?  Yes or no, please.
>
> Thanks,
> --Matt
>
>
> On Sat, Mar 2, 2013 at 7:32 PM, Konstantin Shvachko <sh...@gmail.com>
> wrote:
>>
>> Hi Matt,
>>
>> On Sat, Mar 2, 2013 at 12:32 PM, Matt Foley <mf...@hortonworks.com>
>> wrote:
>> > Konstantin,
>> > I would like to explore what it would take to remove this perceived
>> > impediment --
>>
>> Glad you decided to explore. Thank you.
>>
>> > although I reserve the right to argue that this is not
>> > pre-requisite to merging the cross-platform support patch.
>>
>> It's your right indeed. So as mine to question what the platform
>> support means for you, which I believe remained unclear.
>> I do not impede the change as you should have noticed. My requirement
>> comes from my perception of the support, which means to me exactly two
>> things:
>> 1. The ability to recognise the code is broken for the platform
>> 2. The ability to test new patches on the platform
>> The latter is problematic, as many noticed in this thread, for those
>> whose customary environment does not include Windows.
>>
>> > If we implemented full "test-patch" support for Windows on trunk, would
>> > that
>> > fulfill both your items #1 and #2?  Please note that:
>> > a) Pushing the "Patch Available" button in Jira shall cause a pre-commit
>> > build to start within, I believe, 20 minutes.
>> > b) That build keeps logs for both java build and unit tests for several
>> > days, that are accessible to all viewers.
>>
>> In item #1 I mostly asking for the nightly build, which is simpler
>> than "test-patch". The latter would be ideal from the platform support
>> viewpoint, but it is for the community to decide if we want to add
>> extra +3 hours to the build.
>> Nightly build in my understanding is triggered by the timer rather
>> than by Jira's "submit patch" button.  On Jenkins build configuration
>> you can specify it under "Build periodically".
>>
>> > So, does this provide sufficient on-demand support that we don't have to
>> > implement a whole new on-demand VM support structure of some sort for #2
>> > (which would be an extraordinary and impractical demand)?
>>
>> I did not mention VMs. Item #2 means a build, which runs "test-patch"
>> target with the file specified by a user (instead of a jira
>> attachment).
>> When user clicks "Build Now" link a box is displayed where the user
>> can enter the file path containing the patch. This can be specified in
>> the Build Configuration under "This build is parameterized" by
>> choosing AddParameter / FileParameter. The build can run on the same
>> Windows machine as the nightly build.
>> Such build will let people test their patches for Windows on Jenkins
>> if they don't posses a license for the right version of Windows.
>> I hope this will not turn into extraordinary or impractical effort.
>>
>> Thanks,
>> --Konst
>>
>> > Thanks,
>> > --Matt
>> >
>> >
>> > On Fri, Mar 1, 2013 at 12:18 PM, Konstantin Shvachko
>> > <sh...@gmail.com>
>> > wrote:
>> >>
>> >> -1
>> >> We should have a CI infrastructure in place before we can commit to
>> >> supporting Windows platform.
>> >>
>> >> Eric is right Win/Cygwin was supported since day one.
>> >> I had a Windows box under my desk running nightly builds back in
>> >> 2006-07.
>> >> People were irritated but I was filing windows bugs until 0.22 release.
>> >> Times changing and I am glad to see wider support for Win platform.
>> >>
>> >> But in order to make it work you guys need to put the CI process in
>> >> place
>> >>
>> >> 1. windows jenkins build: could be nightly or PreCommit.
>> >> - Nightly would mean that changes can be committed to trunk based on
>> >> linux PreCommit build. And people will file bugs if the change broke
>> >> Windows nightly build.
>> >> - PreCommit-win build will mean automatic reporting failed tests to
>> >> respective jira blocking commits the same way as it is now with linux
>> >> PreCommit builds.
>> >> We should discuss which way is more efficient for developers.
>> >>
>> >> 2. On-demand-windows Jenkins build.
>> >> I see it as a build to which I can attach my patch and the build will
>> >> run my changes on a dedicated windows box.
>> >> That way people can test their changes without having personal windows
>> >> nodes.
>> >>
>> >> I think this is the minimal set of requirement for us to be able to
>> >> commit to the new platform.
>> >> Right now I see only one windows related build
>> >> https://builds.apache.org/view/Hadoop/job/Hadoop-1-win/
>> >> Which was failing since Sept 8, 2012 and did not run in the last month.
>> >>
>> >> Thanks,
>> >> --Konst
>> >>
>> >> On Thu, Feb 28, 2013 at 8:47 PM, Eric Baldeschwieler
>> >> <er...@hortonworks.com> wrote:
>> >> > +1 (non-binding)
>> >> >
>> >> > A few of observations:
>> >> >
>> >> > - Windows has actually been a supported platform for Hadoop since 0.1
>> >> > .
>> >> > Doug championed supporting windows then and we've continued to do it
>> >> > with
>> >> > varying vigor over time.  To my knowledge we've never made a decision
>> >> > to
>> >> > drop windows support.  The change here is improving our support and
>> >> > dropping
>> >> > the requirement of cigwin.  We had Nutch windows users on the list in
>> >> > 2006
>> >> > and we've been supporting windows FS requirements since inception.
>> >> >
>> >> > - A little pragmatism will go a long way.  As a community we've got
>> >> > to
>> >> > stay committed to keeping hadoop simple (so it does work on many
>> >> > platforms)
>> >> > and extending it to take advantage of key emerging OS/hardware
>> >> > features,
>> >> > such as containers, new FSs, virtualization, flash ...  We should all
>> >> > plan
>> >> > to let new features & optimizations emerge that don't work
>> >> > everywhere, if
>> >> > they are compelling and central to hadoop's mission of being THE best
>> >> > fabric
>> >> > for storing and processing big data.
>> >> >
>> >> > - A UI project like KDE has to deal with the MANY differences between
>> >> > windows and linux UI APIs.  Hadoop faces no such complex challenge
>> >> > and hence
>> >> > can be maintained from a single codeline IMO.  It is mostly
>> >> > abstracted from
>> >> > the OS APIs via Java and our design choices.  Where it is not we can
>> >> > continue to add plugable abstractions.
>> >> >
>> >
>> >
>
>

Re: [Vote] Merge branch-trunk-win to trunk

[Konstantin Shvachko <sh...@gmail.com>]

Didn't I explain in details what I am asking for?

Thanks,
--Konst

On Sun, Mar 3, 2013 at 11:08 AM, Matt Foley <mf...@hortonworks.com> wrote:
> Hi Konstantin,
> I'd like to point out two things:
> First, I already committed in this thread (email of Thu, Feb 28, 2013 at
> 6:01 PM) to providing CI for Windows builds.  So please stop acting like I'm
> resisting this idea or something.
> Second, you didn't answer my question, you just kvetched about the phrasing.
> So I ask again:
>
> Will providing full "test-patch" integration (pre-commit build and unit test
> triggered by Jira "Patch Available" state) satisfy your request for
> functionality #1 and #2?  Yes or no, please.
>
> Thanks,
> --Matt
>
>
> On Sat, Mar 2, 2013 at 7:32 PM, Konstantin Shvachko <sh...@gmail.com>
> wrote:
>>
>> Hi Matt,
>>
>> On Sat, Mar 2, 2013 at 12:32 PM, Matt Foley <mf...@hortonworks.com>
>> wrote:
>> > Konstantin,
>> > I would like to explore what it would take to remove this perceived
>> > impediment --
>>
>> Glad you decided to explore. Thank you.
>>
>> > although I reserve the right to argue that this is not
>> > pre-requisite to merging the cross-platform support patch.
>>
>> It's your right indeed. So as mine to question what the platform
>> support means for you, which I believe remained unclear.
>> I do not impede the change as you should have noticed. My requirement
>> comes from my perception of the support, which means to me exactly two
>> things:
>> 1. The ability to recognise the code is broken for the platform
>> 2. The ability to test new patches on the platform
>> The latter is problematic, as many noticed in this thread, for those
>> whose customary environment does not include Windows.
>>
>> > If we implemented full "test-patch" support for Windows on trunk, would
>> > that
>> > fulfill both your items #1 and #2?  Please note that:
>> > a) Pushing the "Patch Available" button in Jira shall cause a pre-commit
>> > build to start within, I believe, 20 minutes.
>> > b) That build keeps logs for both java build and unit tests for several
>> > days, that are accessible to all viewers.
>>
>> In item #1 I mostly asking for the nightly build, which is simpler
>> than "test-patch". The latter would be ideal from the platform support
>> viewpoint, but it is for the community to decide if we want to add
>> extra +3 hours to the build.
>> Nightly build in my understanding is triggered by the timer rather
>> than by Jira's "submit patch" button.  On Jenkins build configuration
>> you can specify it under "Build periodically".
>>
>> > So, does this provide sufficient on-demand support that we don't have to
>> > implement a whole new on-demand VM support structure of some sort for #2
>> > (which would be an extraordinary and impractical demand)?
>>
>> I did not mention VMs. Item #2 means a build, which runs "test-patch"
>> target with the file specified by a user (instead of a jira
>> attachment).
>> When user clicks "Build Now" link a box is displayed where the user
>> can enter the file path containing the patch. This can be specified in
>> the Build Configuration under "This build is parameterized" by
>> choosing AddParameter / FileParameter. The build can run on the same
>> Windows machine as the nightly build.
>> Such build will let people test their patches for Windows on Jenkins
>> if they don't posses a license for the right version of Windows.
>> I hope this will not turn into extraordinary or impractical effort.
>>
>> Thanks,
>> --Konst
>>
>> > Thanks,
>> > --Matt
>> >
>> >
>> > On Fri, Mar 1, 2013 at 12:18 PM, Konstantin Shvachko
>> > <sh...@gmail.com>
>> > wrote:
>> >>
>> >> -1
>> >> We should have a CI infrastructure in place before we can commit to
>> >> supporting Windows platform.
>> >>
>> >> Eric is right Win/Cygwin was supported since day one.
>> >> I had a Windows box under my desk running nightly builds back in
>> >> 2006-07.
>> >> People were irritated but I was filing windows bugs until 0.22 release.
>> >> Times changing and I am glad to see wider support for Win platform.
>> >>
>> >> But in order to make it work you guys need to put the CI process in
>> >> place
>> >>
>> >> 1. windows jenkins build: could be nightly or PreCommit.
>> >> - Nightly would mean that changes can be committed to trunk based on
>> >> linux PreCommit build. And people will file bugs if the change broke
>> >> Windows nightly build.
>> >> - PreCommit-win build will mean automatic reporting failed tests to
>> >> respective jira blocking commits the same way as it is now with linux
>> >> PreCommit builds.
>> >> We should discuss which way is more efficient for developers.
>> >>
>> >> 2. On-demand-windows Jenkins build.
>> >> I see it as a build to which I can attach my patch and the build will
>> >> run my changes on a dedicated windows box.
>> >> That way people can test their changes without having personal windows
>> >> nodes.
>> >>
>> >> I think this is the minimal set of requirement for us to be able to
>> >> commit to the new platform.
>> >> Right now I see only one windows related build
>> >> https://builds.apache.org/view/Hadoop/job/Hadoop-1-win/
>> >> Which was failing since Sept 8, 2012 and did not run in the last month.
>> >>
>> >> Thanks,
>> >> --Konst
>> >>
>> >> On Thu, Feb 28, 2013 at 8:47 PM, Eric Baldeschwieler
>> >> <er...@hortonworks.com> wrote:
>> >> > +1 (non-binding)
>> >> >
>> >> > A few of observations:
>> >> >
>> >> > - Windows has actually been a supported platform for Hadoop since 0.1
>> >> > .
>> >> > Doug championed supporting windows then and we've continued to do it
>> >> > with
>> >> > varying vigor over time.  To my knowledge we've never made a decision
>> >> > to
>> >> > drop windows support.  The change here is improving our support and
>> >> > dropping
>> >> > the requirement of cigwin.  We had Nutch windows users on the list in
>> >> > 2006
>> >> > and we've been supporting windows FS requirements since inception.
>> >> >
>> >> > - A little pragmatism will go a long way.  As a community we've got
>> >> > to
>> >> > stay committed to keeping hadoop simple (so it does work on many
>> >> > platforms)
>> >> > and extending it to take advantage of key emerging OS/hardware
>> >> > features,
>> >> > such as containers, new FSs, virtualization, flash ...  We should all
>> >> > plan
>> >> > to let new features & optimizations emerge that don't work
>> >> > everywhere, if
>> >> > they are compelling and central to hadoop's mission of being THE best
>> >> > fabric
>> >> > for storing and processing big data.
>> >> >
>> >> > - A UI project like KDE has to deal with the MANY differences between
>> >> > windows and linux UI APIs.  Hadoop faces no such complex challenge
>> >> > and hence
>> >> > can be maintained from a single codeline IMO.  It is mostly
>> >> > abstracted from
>> >> > the OS APIs via Java and our design choices.  Where it is not we can
>> >> > continue to add plugable abstractions.
>> >> >
>> >
>> >
>
>

Re: [Vote] Merge branch-trunk-win to trunk

[Matt Foley <mf...@hortonworks.com>]

Hi Konstantin,
I'd like to point out two things:
First, I already committed in this thread (email of Thu, Feb 28, 2013 at
6:01 PM) to providing CI for Windows builds.  So please stop acting like
I'm resisting this idea or something.
Second, you didn't answer my question, you just kvetched about the
phrasing.  So I ask again:

Will providing full "test-patch" integration (pre-commit build and unit
test triggered by Jira "Patch Available" state) satisfy your request for
functionality #1 and #2?  Yes or no, please.

Thanks,
--Matt


On Sat, Mar 2, 2013 at 7:32 PM, Konstantin Shvachko <sh...@gmail.com>wrote:

> Hi Matt,
>
> On Sat, Mar 2, 2013 at 12:32 PM, Matt Foley <mf...@hortonworks.com>
> wrote:
> > Konstantin,
> > I would like to explore what it would take to remove this perceived
> > impediment --
>
> Glad you decided to explore. Thank you.
>
> > although I reserve the right to argue that this is not
> > pre-requisite to merging the cross-platform support patch.
>
> It's your right indeed. So as mine to question what the platform
> support means for you, which I believe remained unclear.
> I do not impede the change as you should have noticed. My requirement
> comes from my perception of the support, which means to me exactly two
> things:
> 1. The ability to recognise the code is broken for the platform
> 2. The ability to test new patches on the platform
> The latter is problematic, as many noticed in this thread, for those
> whose customary environment does not include Windows.
>
> > If we implemented full "test-patch" support for Windows on trunk, would
> that
> > fulfill both your items #1 and #2?  Please note that:
> > a) Pushing the "Patch Available" button in Jira shall cause a pre-commit
> > build to start within, I believe, 20 minutes.
> > b) That build keeps logs for both java build and unit tests for several
> > days, that are accessible to all viewers.
>
> In item #1 I mostly asking for the nightly build, which is simpler
> than "test-patch". The latter would be ideal from the platform support
> viewpoint, but it is for the community to decide if we want to add
> extra +3 hours to the build.
> Nightly build in my understanding is triggered by the timer rather
> than by Jira's "submit patch" button.  On Jenkins build configuration
> you can specify it under "Build periodically".
>
> > So, does this provide sufficient on-demand support that we don't have to
> > implement a whole new on-demand VM support structure of some sort for #2
> > (which would be an extraordinary and impractical demand)?
>
> I did not mention VMs. Item #2 means a build, which runs "test-patch"
> target with the file specified by a user (instead of a jira
> attachment).
> When user clicks "Build Now" link a box is displayed where the user
> can enter the file path containing the patch. This can be specified in
> the Build Configuration under "This build is parameterized" by
> choosing AddParameter / FileParameter. The build can run on the same
> Windows machine as the nightly build.
> Such build will let people test their patches for Windows on Jenkins
> if they don't posses a license for the right version of Windows.
> I hope this will not turn into extraordinary or impractical effort.
>
> Thanks,
> --Konst
>
> > Thanks,
> > --Matt
> >
> >
> > On Fri, Mar 1, 2013 at 12:18 PM, Konstantin Shvachko <
> shv.hadoop@gmail.com>
> > wrote:
> >>
> >> -1
> >> We should have a CI infrastructure in place before we can commit to
> >> supporting Windows platform.
> >>
> >> Eric is right Win/Cygwin was supported since day one.
> >> I had a Windows box under my desk running nightly builds back in
> 2006-07.
> >> People were irritated but I was filing windows bugs until 0.22 release.
> >> Times changing and I am glad to see wider support for Win platform.
> >>
> >> But in order to make it work you guys need to put the CI process in
> place
> >>
> >> 1. windows jenkins build: could be nightly or PreCommit.
> >> - Nightly would mean that changes can be committed to trunk based on
> >> linux PreCommit build. And people will file bugs if the change broke
> >> Windows nightly build.
> >> - PreCommit-win build will mean automatic reporting failed tests to
> >> respective jira blocking commits the same way as it is now with linux
> >> PreCommit builds.
> >> We should discuss which way is more efficient for developers.
> >>
> >> 2. On-demand-windows Jenkins build.
> >> I see it as a build to which I can attach my patch and the build will
> >> run my changes on a dedicated windows box.
> >> That way people can test their changes without having personal windows
> >> nodes.
> >>
> >> I think this is the minimal set of requirement for us to be able to
> >> commit to the new platform.
> >> Right now I see only one windows related build
> >> https://builds.apache.org/view/Hadoop/job/Hadoop-1-win/
> >> Which was failing since Sept 8, 2012 and did not run in the last month.
> >>
> >> Thanks,
> >> --Konst
> >>
> >> On Thu, Feb 28, 2013 at 8:47 PM, Eric Baldeschwieler
> >> <er...@hortonworks.com> wrote:
> >> > +1 (non-binding)
> >> >
> >> > A few of observations:
> >> >
> >> > - Windows has actually been a supported platform for Hadoop since 0.1
> .
> >> > Doug championed supporting windows then and we've continued to do it
> with
> >> > varying vigor over time.  To my knowledge we've never made a decision
> to
> >> > drop windows support.  The change here is improving our support and
> dropping
> >> > the requirement of cigwin.  We had Nutch windows users on the list in
> 2006
> >> > and we've been supporting windows FS requirements since inception.
> >> >
> >> > - A little pragmatism will go a long way.  As a community we've got to
> >> > stay committed to keeping hadoop simple (so it does work on many
> platforms)
> >> > and extending it to take advantage of key emerging OS/hardware
> features,
> >> > such as containers, new FSs, virtualization, flash ...  We should all
> plan
> >> > to let new features & optimizations emerge that don't work
> everywhere, if
> >> > they are compelling and central to hadoop's mission of being THE best
> fabric
> >> > for storing and processing big data.
> >> >
> >> > - A UI project like KDE has to deal with the MANY differences between
> >> > windows and linux UI APIs.  Hadoop faces no such complex challenge
> and hence
> >> > can be maintained from a single codeline IMO.  It is mostly
> abstracted from
> >> > the OS APIs via Java and our design choices.  Where it is not we can
> >> > continue to add plugable abstractions.
> >> >
> >
> >
>

Re: [Vote] Merge branch-trunk-win to trunk

[Matt Foley <mf...@hortonworks.com>]

Hi Konstantin,
I'd like to point out two things:
First, I already committed in this thread (email of Thu, Feb 28, 2013 at
6:01 PM) to providing CI for Windows builds.  So please stop acting like
I'm resisting this idea or something.
Second, you didn't answer my question, you just kvetched about the
phrasing.  So I ask again:

Will providing full "test-patch" integration (pre-commit build and unit
test triggered by Jira "Patch Available" state) satisfy your request for
functionality #1 and #2?  Yes or no, please.

Thanks,
--Matt


On Sat, Mar 2, 2013 at 7:32 PM, Konstantin Shvachko <sh...@gmail.com>wrote:

> Hi Matt,
>
> On Sat, Mar 2, 2013 at 12:32 PM, Matt Foley <mf...@hortonworks.com>
> wrote:
> > Konstantin,
> > I would like to explore what it would take to remove this perceived
> > impediment --
>
> Glad you decided to explore. Thank you.
>
> > although I reserve the right to argue that this is not
> > pre-requisite to merging the cross-platform support patch.
>
> It's your right indeed. So as mine to question what the platform
> support means for you, which I believe remained unclear.
> I do not impede the change as you should have noticed. My requirement
> comes from my perception of the support, which means to me exactly two
> things:
> 1. The ability to recognise the code is broken for the platform
> 2. The ability to test new patches on the platform
> The latter is problematic, as many noticed in this thread, for those
> whose customary environment does not include Windows.
>
> > If we implemented full "test-patch" support for Windows on trunk, would
> that
> > fulfill both your items #1 and #2?  Please note that:
> > a) Pushing the "Patch Available" button in Jira shall cause a pre-commit
> > build to start within, I believe, 20 minutes.
> > b) That build keeps logs for both java build and unit tests for several
> > days, that are accessible to all viewers.
>
> In item #1 I mostly asking for the nightly build, which is simpler
> than "test-patch". The latter would be ideal from the platform support
> viewpoint, but it is for the community to decide if we want to add
> extra +3 hours to the build.
> Nightly build in my understanding is triggered by the timer rather
> than by Jira's "submit patch" button.  On Jenkins build configuration
> you can specify it under "Build periodically".
>
> > So, does this provide sufficient on-demand support that we don't have to
> > implement a whole new on-demand VM support structure of some sort for #2
> > (which would be an extraordinary and impractical demand)?
>
> I did not mention VMs. Item #2 means a build, which runs "test-patch"
> target with the file specified by a user (instead of a jira
> attachment).
> When user clicks "Build Now" link a box is displayed where the user
> can enter the file path containing the patch. This can be specified in
> the Build Configuration under "This build is parameterized" by
> choosing AddParameter / FileParameter. The build can run on the same
> Windows machine as the nightly build.
> Such build will let people test their patches for Windows on Jenkins
> if they don't posses a license for the right version of Windows.
> I hope this will not turn into extraordinary or impractical effort.
>
> Thanks,
> --Konst
>
> > Thanks,
> > --Matt
> >
> >
> > On Fri, Mar 1, 2013 at 12:18 PM, Konstantin Shvachko <
> shv.hadoop@gmail.com>
> > wrote:
> >>
> >> -1
> >> We should have a CI infrastructure in place before we can commit to
> >> supporting Windows platform.
> >>
> >> Eric is right Win/Cygwin was supported since day one.
> >> I had a Windows box under my desk running nightly builds back in
> 2006-07.
> >> People were irritated but I was filing windows bugs until 0.22 release.
> >> Times changing and I am glad to see wider support for Win platform.
> >>
> >> But in order to make it work you guys need to put the CI process in
> place
> >>
> >> 1. windows jenkins build: could be nightly or PreCommit.
> >> - Nightly would mean that changes can be committed to trunk based on
> >> linux PreCommit build. And people will file bugs if the change broke
> >> Windows nightly build.
> >> - PreCommit-win build will mean automatic reporting failed tests to
> >> respective jira blocking commits the same way as it is now with linux
> >> PreCommit builds.
> >> We should discuss which way is more efficient for developers.
> >>
> >> 2. On-demand-windows Jenkins build.
> >> I see it as a build to which I can attach my patch and the build will
> >> run my changes on a dedicated windows box.
> >> That way people can test their changes without having personal windows
> >> nodes.
> >>
> >> I think this is the minimal set of requirement for us to be able to
> >> commit to the new platform.
> >> Right now I see only one windows related build
> >> https://builds.apache.org/view/Hadoop/job/Hadoop-1-win/
> >> Which was failing since Sept 8, 2012 and did not run in the last month.
> >>
> >> Thanks,
> >> --Konst
> >>
> >> On Thu, Feb 28, 2013 at 8:47 PM, Eric Baldeschwieler
> >> <er...@hortonworks.com> wrote:
> >> > +1 (non-binding)
> >> >
> >> > A few of observations:
> >> >
> >> > - Windows has actually been a supported platform for Hadoop since 0.1
> .
> >> > Doug championed supporting windows then and we've continued to do it
> with
> >> > varying vigor over time.  To my knowledge we've never made a decision
> to
> >> > drop windows support.  The change here is improving our support and
> dropping
> >> > the requirement of cigwin.  We had Nutch windows users on the list in
> 2006
> >> > and we've been supporting windows FS requirements since inception.
> >> >
> >> > - A little pragmatism will go a long way.  As a community we've got to
> >> > stay committed to keeping hadoop simple (so it does work on many
> platforms)
> >> > and extending it to take advantage of key emerging OS/hardware
> features,
> >> > such as containers, new FSs, virtualization, flash ...  We should all
> plan
> >> > to let new features & optimizations emerge that don't work
> everywhere, if
> >> > they are compelling and central to hadoop's mission of being THE best
> fabric
> >> > for storing and processing big data.
> >> >
> >> > - A UI project like KDE has to deal with the MANY differences between
> >> > windows and linux UI APIs.  Hadoop faces no such complex challenge
> and hence
> >> > can be maintained from a single codeline IMO.  It is mostly
> abstracted from
> >> > the OS APIs via Java and our design choices.  Where it is not we can
> >> > continue to add plugable abstractions.
> >> >
> >
> >
>

Re: [Vote] Merge branch-trunk-win to trunk

[Matt Foley <mf...@hortonworks.com>]

Hi Konstantin,
I'd like to point out two things:
First, I already committed in this thread (email of Thu, Feb 28, 2013 at
6:01 PM) to providing CI for Windows builds.  So please stop acting like
I'm resisting this idea or something.
Second, you didn't answer my question, you just kvetched about the
phrasing.  So I ask again:

Will providing full "test-patch" integration (pre-commit build and unit
test triggered by Jira "Patch Available" state) satisfy your request for
functionality #1 and #2?  Yes or no, please.

Thanks,
--Matt


On Sat, Mar 2, 2013 at 7:32 PM, Konstantin Shvachko <sh...@gmail.com>wrote:

> Hi Matt,
>
> On Sat, Mar 2, 2013 at 12:32 PM, Matt Foley <mf...@hortonworks.com>
> wrote:
> > Konstantin,
> > I would like to explore what it would take to remove this perceived
> > impediment --
>
> Glad you decided to explore. Thank you.
>
> > although I reserve the right to argue that this is not
> > pre-requisite to merging the cross-platform support patch.
>
> It's your right indeed. So as mine to question what the platform
> support means for you, which I believe remained unclear.
> I do not impede the change as you should have noticed. My requirement
> comes from my perception of the support, which means to me exactly two
> things:
> 1. The ability to recognise the code is broken for the platform
> 2. The ability to test new patches on the platform
> The latter is problematic, as many noticed in this thread, for those
> whose customary environment does not include Windows.
>
> > If we implemented full "test-patch" support for Windows on trunk, would
> that
> > fulfill both your items #1 and #2?  Please note that:
> > a) Pushing the "Patch Available" button in Jira shall cause a pre-commit
> > build to start within, I believe, 20 minutes.
> > b) That build keeps logs for both java build and unit tests for several
> > days, that are accessible to all viewers.
>
> In item #1 I mostly asking for the nightly build, which is simpler
> than "test-patch". The latter would be ideal from the platform support
> viewpoint, but it is for the community to decide if we want to add
> extra +3 hours to the build.
> Nightly build in my understanding is triggered by the timer rather
> than by Jira's "submit patch" button.  On Jenkins build configuration
> you can specify it under "Build periodically".
>
> > So, does this provide sufficient on-demand support that we don't have to
> > implement a whole new on-demand VM support structure of some sort for #2
> > (which would be an extraordinary and impractical demand)?
>
> I did not mention VMs. Item #2 means a build, which runs "test-patch"
> target with the file specified by a user (instead of a jira
> attachment).
> When user clicks "Build Now" link a box is displayed where the user
> can enter the file path containing the patch. This can be specified in
> the Build Configuration under "This build is parameterized" by
> choosing AddParameter / FileParameter. The build can run on the same
> Windows machine as the nightly build.
> Such build will let people test their patches for Windows on Jenkins
> if they don't posses a license for the right version of Windows.
> I hope this will not turn into extraordinary or impractical effort.
>
> Thanks,
> --Konst
>
> > Thanks,
> > --Matt
> >
> >
> > On Fri, Mar 1, 2013 at 12:18 PM, Konstantin Shvachko <
> shv.hadoop@gmail.com>
> > wrote:
> >>
> >> -1
> >> We should have a CI infrastructure in place before we can commit to
> >> supporting Windows platform.
> >>
> >> Eric is right Win/Cygwin was supported since day one.
> >> I had a Windows box under my desk running nightly builds back in
> 2006-07.
> >> People were irritated but I was filing windows bugs until 0.22 release.
> >> Times changing and I am glad to see wider support for Win platform.
> >>
> >> But in order to make it work you guys need to put the CI process in
> place
> >>
> >> 1. windows jenkins build: could be nightly or PreCommit.
> >> - Nightly would mean that changes can be committed to trunk based on
> >> linux PreCommit build. And people will file bugs if the change broke
> >> Windows nightly build.
> >> - PreCommit-win build will mean automatic reporting failed tests to
> >> respective jira blocking commits the same way as it is now with linux
> >> PreCommit builds.
> >> We should discuss which way is more efficient for developers.
> >>
> >> 2. On-demand-windows Jenkins build.
> >> I see it as a build to which I can attach my patch and the build will
> >> run my changes on a dedicated windows box.
> >> That way people can test their changes without having personal windows
> >> nodes.
> >>
> >> I think this is the minimal set of requirement for us to be able to
> >> commit to the new platform.
> >> Right now I see only one windows related build
> >> https://builds.apache.org/view/Hadoop/job/Hadoop-1-win/
> >> Which was failing since Sept 8, 2012 and did not run in the last month.
> >>
> >> Thanks,
> >> --Konst
> >>
> >> On Thu, Feb 28, 2013 at 8:47 PM, Eric Baldeschwieler
> >> <er...@hortonworks.com> wrote:
> >> > +1 (non-binding)
> >> >
> >> > A few of observations:
> >> >
> >> > - Windows has actually been a supported platform for Hadoop since 0.1
> .
> >> > Doug championed supporting windows then and we've continued to do it
> with
> >> > varying vigor over time.  To my knowledge we've never made a decision
> to
> >> > drop windows support.  The change here is improving our support and
> dropping
> >> > the requirement of cigwin.  We had Nutch windows users on the list in
> 2006
> >> > and we've been supporting windows FS requirements since inception.
> >> >
> >> > - A little pragmatism will go a long way.  As a community we've got to
> >> > stay committed to keeping hadoop simple (so it does work on many
> platforms)
> >> > and extending it to take advantage of key emerging OS/hardware
> features,
> >> > such as containers, new FSs, virtualization, flash ...  We should all
> plan
> >> > to let new features & optimizations emerge that don't work
> everywhere, if
> >> > they are compelling and central to hadoop's mission of being THE best
> fabric
> >> > for storing and processing big data.
> >> >
> >> > - A UI project like KDE has to deal with the MANY differences between
> >> > windows and linux UI APIs.  Hadoop faces no such complex challenge
> and hence
> >> > can be maintained from a single codeline IMO.  It is mostly
> abstracted from
> >> > the OS APIs via Java and our design choices.  Where it is not we can
> >> > continue to add plugable abstractions.
> >> >
> >
> >
>

Re: [Vote] Merge branch-trunk-win to trunk

[Matt Foley <mf...@hortonworks.com>]

Added to the Jira to modify http://wiki.apache.org/hadoop/HowToContribute to
document this decision.


On Mon, Mar 4, 2013 at 5:42 PM, Harsh J <ha...@cloudera.com> wrote:

> Thanks Suresh. Regarding where; we can state it on
> http://wiki.apache.org/hadoop/HowToContribute in the test-patch
> section perhaps.
>
> +1 on the merge.
>
> On Mon, Mar 4, 2013 at 11:39 PM, Suresh Srinivas <su...@hortonworks.com>
> wrote:
> > On Sun, Mar 3, 2013 at 8:50 PM, Harsh J <ha...@cloudera.com> wrote:
> >
> >> Have we agreed (and stated it somewhere proper) that a -1 obtained for
> >> a Windows CI build for a test-patch will not block the ongoing work
> >> (unless it is Windows specific) and patches may still be committed to
> >> trunk despite that?
> >>
> >
> > This thread is long and possibly hard to follow. Yes, I and several
> others
> > have
> > stated that for now it is okay to commit even if Windows precommit build
> > posts -1.
> >
> >>
> >> I'm +1 if someone can assert and add the above into the formal
> >> guidelines. I'd still prefer that Windows does its releases separately
> >> as that ensures more quality for its audience and better testing
> >> periods (and wouldn't block anything), but we can come to that iff we
> >> are unable to maintain the currently proposed model.
> >
> >
> > Which do you think is the right place to add this?
> >
> > At this time we are voting for merging into trunk. I prefer having a
> single
> > release
> > that supports both Linux and windows. Based on working on Windows support
> > I think this is doable and should not hold up releases for Linux.
>
>
>
> --
> Harsh J
>

Re: [Vote] Merge branch-trunk-win to trunk

[Harsh J <ha...@cloudera.com>]

Thanks Suresh. Regarding where; we can state it on
http://wiki.apache.org/hadoop/HowToContribute in the test-patch
section perhaps.

+1 on the merge.

On Mon, Mar 4, 2013 at 11:39 PM, Suresh Srinivas <su...@hortonworks.com> wrote:
> On Sun, Mar 3, 2013 at 8:50 PM, Harsh J <ha...@cloudera.com> wrote:
>
>> Have we agreed (and stated it somewhere proper) that a -1 obtained for
>> a Windows CI build for a test-patch will not block the ongoing work
>> (unless it is Windows specific) and patches may still be committed to
>> trunk despite that?
>>
>
> This thread is long and possibly hard to follow. Yes, I and several others
> have
> stated that for now it is okay to commit even if Windows precommit build
> posts -1.
>
>>
>> I'm +1 if someone can assert and add the above into the formal
>> guidelines. I'd still prefer that Windows does its releases separately
>> as that ensures more quality for its audience and better testing
>> periods (and wouldn't block anything), but we can come to that iff we
>> are unable to maintain the currently proposed model.
>
>
> Which do you think is the right place to add this?
>
> At this time we are voting for merging into trunk. I prefer having a single
> release
> that supports both Linux and windows. Based on working on Windows support
> I think this is doable and should not hold up releases for Linux.



--
Harsh J

Re: [Vote] Merge branch-trunk-win to trunk

[Suresh Srinivas <su...@hortonworks.com>]

On Sun, Mar 3, 2013 at 8:50 PM, Harsh J <ha...@cloudera.com> wrote:

> Have we agreed (and stated it somewhere proper) that a -1 obtained for
> a Windows CI build for a test-patch will not block the ongoing work
> (unless it is Windows specific) and patches may still be committed to
> trunk despite that?
>

This thread is long and possibly hard to follow. Yes, I and several others
have
stated that for now it is okay to commit even if Windows precommit build
posts -1.

>
> I'm +1 if someone can assert and add the above into the formal
> guidelines. I'd still prefer that Windows does its releases separately
> as that ensures more quality for its audience and better testing
> periods (and wouldn't block anything), but we can come to that iff we
> are unable to maintain the currently proposed model.


Which do you think is the right place to add this?

At this time we are voting for merging into trunk. I prefer having a single
release
that supports both Linux and windows. Based on working on Windows support
I think this is doable and should not hold up releases for Linux.

Re: [Vote] Merge branch-trunk-win to trunk

[Harsh J <ha...@cloudera.com>]

Have we agreed (and stated it somewhere proper) that a -1 obtained for
a Windows CI build for a test-patch will not block the ongoing work
(unless it is Windows specific) and patches may still be committed to
trunk despite that?

I'm +1 if someone can assert and add the above into the formal
guidelines. I'd still prefer that Windows does its releases separately
as that ensures more quality for its audience and better testing
periods (and wouldn't block anything), but we can come to that iff we
are unable to maintain the currently proposed model.

On Mon, Mar 4, 2013 at 7:39 AM, Tsuyoshi OZAWA <oz...@gmail.com> wrote:
> +1 (non-binding),
>
> Windows support is attractive for lots users.
> From point a view from Hadoop developer, Matt said that CI supports
> cross platform testing, and it's quite reasonable condition to merge.
>
> Thanks,
> Tsuyoshi



--
Harsh J

Re: [Vote] Merge branch-trunk-win to trunk

[Tsuyoshi OZAWA <oz...@gmail.com>]

+1 (non-binding),

Windows support is attractive for lots users.
>From point a view from Hadoop developer, Matt said that CI supports
cross platform testing, and it's quite reasonable condition to merge.

Thanks,
Tsuyoshi

Re: [Vote] Merge branch-trunk-win to trunk

[Konstantin Shvachko <sh...@gmail.com>]

Hi Matt,

On Sat, Mar 2, 2013 at 12:32 PM, Matt Foley <mf...@hortonworks.com> wrote:
> Konstantin,
> I would like to explore what it would take to remove this perceived
> impediment --

Glad you decided to explore. Thank you.

> although I reserve the right to argue that this is not
> pre-requisite to merging the cross-platform support patch.

It's your right indeed. So as mine to question what the platform
support means for you, which I believe remained unclear.
I do not impede the change as you should have noticed. My requirement
comes from my perception of the support, which means to me exactly two
things:
1. The ability to recognise the code is broken for the platform
2. The ability to test new patches on the platform
The latter is problematic, as many noticed in this thread, for those
whose customary environment does not include Windows.

> If we implemented full "test-patch" support for Windows on trunk, would that
> fulfill both your items #1 and #2?  Please note that:
> a) Pushing the "Patch Available" button in Jira shall cause a pre-commit
> build to start within, I believe, 20 minutes.
> b) That build keeps logs for both java build and unit tests for several
> days, that are accessible to all viewers.

In item #1 I mostly asking for the nightly build, which is simpler
than "test-patch". The latter would be ideal from the platform support
viewpoint, but it is for the community to decide if we want to add
extra +3 hours to the build.
Nightly build in my understanding is triggered by the timer rather
than by Jira's "submit patch" button.  On Jenkins build configuration
you can specify it under "Build periodically".

> So, does this provide sufficient on-demand support that we don't have to
> implement a whole new on-demand VM support structure of some sort for #2
> (which would be an extraordinary and impractical demand)?

I did not mention VMs. Item #2 means a build, which runs "test-patch"
target with the file specified by a user (instead of a jira
attachment).
When user clicks "Build Now" link a box is displayed where the user
can enter the file path containing the patch. This can be specified in
the Build Configuration under "This build is parameterized" by
choosing AddParameter / FileParameter. The build can run on the same
Windows machine as the nightly build.
Such build will let people test their patches for Windows on Jenkins
if they don't posses a license for the right version of Windows.
I hope this will not turn into extraordinary or impractical effort.

Thanks,
--Konst

> Thanks,
> --Matt
>
>
> On Fri, Mar 1, 2013 at 12:18 PM, Konstantin Shvachko <sh...@gmail.com>
> wrote:
>>
>> -1
>> We should have a CI infrastructure in place before we can commit to
>> supporting Windows platform.
>>
>> Eric is right Win/Cygwin was supported since day one.
>> I had a Windows box under my desk running nightly builds back in 2006-07.
>> People were irritated but I was filing windows bugs until 0.22 release.
>> Times changing and I am glad to see wider support for Win platform.
>>
>> But in order to make it work you guys need to put the CI process in place
>>
>> 1. windows jenkins build: could be nightly or PreCommit.
>> - Nightly would mean that changes can be committed to trunk based on
>> linux PreCommit build. And people will file bugs if the change broke
>> Windows nightly build.
>> - PreCommit-win build will mean automatic reporting failed tests to
>> respective jira blocking commits the same way as it is now with linux
>> PreCommit builds.
>> We should discuss which way is more efficient for developers.
>>
>> 2. On-demand-windows Jenkins build.
>> I see it as a build to which I can attach my patch and the build will
>> run my changes on a dedicated windows box.
>> That way people can test their changes without having personal windows
>> nodes.
>>
>> I think this is the minimal set of requirement for us to be able to
>> commit to the new platform.
>> Right now I see only one windows related build
>> https://builds.apache.org/view/Hadoop/job/Hadoop-1-win/
>> Which was failing since Sept 8, 2012 and did not run in the last month.
>>
>> Thanks,
>> --Konst
>>
>> On Thu, Feb 28, 2013 at 8:47 PM, Eric Baldeschwieler
>> <er...@hortonworks.com> wrote:
>> > +1 (non-binding)
>> >
>> > A few of observations:
>> >
>> > - Windows has actually been a supported platform for Hadoop since 0.1 .
>> > Doug championed supporting windows then and we've continued to do it with
>> > varying vigor over time.  To my knowledge we've never made a decision to
>> > drop windows support.  The change here is improving our support and dropping
>> > the requirement of cigwin.  We had Nutch windows users on the list in 2006
>> > and we've been supporting windows FS requirements since inception.
>> >
>> > - A little pragmatism will go a long way.  As a community we've got to
>> > stay committed to keeping hadoop simple (so it does work on many platforms)
>> > and extending it to take advantage of key emerging OS/hardware features,
>> > such as containers, new FSs, virtualization, flash ...  We should all plan
>> > to let new features & optimizations emerge that don't work everywhere, if
>> > they are compelling and central to hadoop's mission of being THE best fabric
>> > for storing and processing big data.
>> >
>> > - A UI project like KDE has to deal with the MANY differences between
>> > windows and linux UI APIs.  Hadoop faces no such complex challenge and hence
>> > can be maintained from a single codeline IMO.  It is mostly abstracted from
>> > the OS APIs via Java and our design choices.  Where it is not we can
>> > continue to add plugable abstractions.
>> >
>
>

Re: [Vote] Merge branch-trunk-win to trunk

[Konstantin Shvachko <sh...@gmail.com>]

Hi Matt,

On Sat, Mar 2, 2013 at 12:32 PM, Matt Foley <mf...@hortonworks.com> wrote:
> Konstantin,
> I would like to explore what it would take to remove this perceived
> impediment --

Glad you decided to explore. Thank you.

> although I reserve the right to argue that this is not
> pre-requisite to merging the cross-platform support patch.

It's your right indeed. So as mine to question what the platform
support means for you, which I believe remained unclear.
I do not impede the change as you should have noticed. My requirement
comes from my perception of the support, which means to me exactly two
things:
1. The ability to recognise the code is broken for the platform
2. The ability to test new patches on the platform
The latter is problematic, as many noticed in this thread, for those
whose customary environment does not include Windows.

> If we implemented full "test-patch" support for Windows on trunk, would that
> fulfill both your items #1 and #2?  Please note that:
> a) Pushing the "Patch Available" button in Jira shall cause a pre-commit
> build to start within, I believe, 20 minutes.
> b) That build keeps logs for both java build and unit tests for several
> days, that are accessible to all viewers.

In item #1 I mostly asking for the nightly build, which is simpler
than "test-patch". The latter would be ideal from the platform support
viewpoint, but it is for the community to decide if we want to add
extra +3 hours to the build.
Nightly build in my understanding is triggered by the timer rather
than by Jira's "submit patch" button.  On Jenkins build configuration
you can specify it under "Build periodically".

> So, does this provide sufficient on-demand support that we don't have to
> implement a whole new on-demand VM support structure of some sort for #2
> (which would be an extraordinary and impractical demand)?

I did not mention VMs. Item #2 means a build, which runs "test-patch"
target with the file specified by a user (instead of a jira
attachment).
When user clicks "Build Now" link a box is displayed where the user
can enter the file path containing the patch. This can be specified in
the Build Configuration under "This build is parameterized" by
choosing AddParameter / FileParameter. The build can run on the same
Windows machine as the nightly build.
Such build will let people test their patches for Windows on Jenkins
if they don't posses a license for the right version of Windows.
I hope this will not turn into extraordinary or impractical effort.

Thanks,
--Konst

> Thanks,
> --Matt
>
>
> On Fri, Mar 1, 2013 at 12:18 PM, Konstantin Shvachko <sh...@gmail.com>
> wrote:
>>
>> -1
>> We should have a CI infrastructure in place before we can commit to
>> supporting Windows platform.
>>
>> Eric is right Win/Cygwin was supported since day one.
>> I had a Windows box under my desk running nightly builds back in 2006-07.
>> People were irritated but I was filing windows bugs until 0.22 release.
>> Times changing and I am glad to see wider support for Win platform.
>>
>> But in order to make it work you guys need to put the CI process in place
>>
>> 1. windows jenkins build: could be nightly or PreCommit.
>> - Nightly would mean that changes can be committed to trunk based on
>> linux PreCommit build. And people will file bugs if the change broke
>> Windows nightly build.
>> - PreCommit-win build will mean automatic reporting failed tests to
>> respective jira blocking commits the same way as it is now with linux
>> PreCommit builds.
>> We should discuss which way is more efficient for developers.
>>
>> 2. On-demand-windows Jenkins build.
>> I see it as a build to which I can attach my patch and the build will
>> run my changes on a dedicated windows box.
>> That way people can test their changes without having personal windows
>> nodes.
>>
>> I think this is the minimal set of requirement for us to be able to
>> commit to the new platform.
>> Right now I see only one windows related build
>> https://builds.apache.org/view/Hadoop/job/Hadoop-1-win/
>> Which was failing since Sept 8, 2012 and did not run in the last month.
>>
>> Thanks,
>> --Konst
>>
>> On Thu, Feb 28, 2013 at 8:47 PM, Eric Baldeschwieler
>> <er...@hortonworks.com> wrote:
>> > +1 (non-binding)
>> >
>> > A few of observations:
>> >
>> > - Windows has actually been a supported platform for Hadoop since 0.1 .
>> > Doug championed supporting windows then and we've continued to do it with
>> > varying vigor over time.  To my knowledge we've never made a decision to
>> > drop windows support.  The change here is improving our support and dropping
>> > the requirement of cigwin.  We had Nutch windows users on the list in 2006
>> > and we've been supporting windows FS requirements since inception.
>> >
>> > - A little pragmatism will go a long way.  As a community we've got to
>> > stay committed to keeping hadoop simple (so it does work on many platforms)
>> > and extending it to take advantage of key emerging OS/hardware features,
>> > such as containers, new FSs, virtualization, flash ...  We should all plan
>> > to let new features & optimizations emerge that don't work everywhere, if
>> > they are compelling and central to hadoop's mission of being THE best fabric
>> > for storing and processing big data.
>> >
>> > - A UI project like KDE has to deal with the MANY differences between
>> > windows and linux UI APIs.  Hadoop faces no such complex challenge and hence
>> > can be maintained from a single codeline IMO.  It is mostly abstracted from
>> > the OS APIs via Java and our design choices.  Where it is not we can
>> > continue to add plugable abstractions.
>> >
>
>

Re: [Vote] Merge branch-trunk-win to trunk

[Konstantin Shvachko <sh...@gmail.com>]

Hi Matt,

On Sat, Mar 2, 2013 at 12:32 PM, Matt Foley <mf...@hortonworks.com> wrote:
> Konstantin,
> I would like to explore what it would take to remove this perceived
> impediment --

Glad you decided to explore. Thank you.

> although I reserve the right to argue that this is not
> pre-requisite to merging the cross-platform support patch.

It's your right indeed. So as mine to question what the platform
support means for you, which I believe remained unclear.
I do not impede the change as you should have noticed. My requirement
comes from my perception of the support, which means to me exactly two
things:
1. The ability to recognise the code is broken for the platform
2. The ability to test new patches on the platform
The latter is problematic, as many noticed in this thread, for those
whose customary environment does not include Windows.

> If we implemented full "test-patch" support for Windows on trunk, would that
> fulfill both your items #1 and #2?  Please note that:
> a) Pushing the "Patch Available" button in Jira shall cause a pre-commit
> build to start within, I believe, 20 minutes.
> b) That build keeps logs for both java build and unit tests for several
> days, that are accessible to all viewers.

In item #1 I mostly asking for the nightly build, which is simpler
than "test-patch". The latter would be ideal from the platform support
viewpoint, but it is for the community to decide if we want to add
extra +3 hours to the build.
Nightly build in my understanding is triggered by the timer rather
than by Jira's "submit patch" button.  On Jenkins build configuration
you can specify it under "Build periodically".

> So, does this provide sufficient on-demand support that we don't have to
> implement a whole new on-demand VM support structure of some sort for #2
> (which would be an extraordinary and impractical demand)?

I did not mention VMs. Item #2 means a build, which runs "test-patch"
target with the file specified by a user (instead of a jira
attachment).
When user clicks "Build Now" link a box is displayed where the user
can enter the file path containing the patch. This can be specified in
the Build Configuration under "This build is parameterized" by
choosing AddParameter / FileParameter. The build can run on the same
Windows machine as the nightly build.
Such build will let people test their patches for Windows on Jenkins
if they don't posses a license for the right version of Windows.
I hope this will not turn into extraordinary or impractical effort.

Thanks,
--Konst

> Thanks,
> --Matt
>
>
> On Fri, Mar 1, 2013 at 12:18 PM, Konstantin Shvachko <sh...@gmail.com>
> wrote:
>>
>> -1
>> We should have a CI infrastructure in place before we can commit to
>> supporting Windows platform.
>>
>> Eric is right Win/Cygwin was supported since day one.
>> I had a Windows box under my desk running nightly builds back in 2006-07.
>> People were irritated but I was filing windows bugs until 0.22 release.
>> Times changing and I am glad to see wider support for Win platform.
>>
>> But in order to make it work you guys need to put the CI process in place
>>
>> 1. windows jenkins build: could be nightly or PreCommit.
>> - Nightly would mean that changes can be committed to trunk based on
>> linux PreCommit build. And people will file bugs if the change broke
>> Windows nightly build.
>> - PreCommit-win build will mean automatic reporting failed tests to
>> respective jira blocking commits the same way as it is now with linux
>> PreCommit builds.
>> We should discuss which way is more efficient for developers.
>>
>> 2. On-demand-windows Jenkins build.
>> I see it as a build to which I can attach my patch and the build will
>> run my changes on a dedicated windows box.
>> That way people can test their changes without having personal windows
>> nodes.
>>
>> I think this is the minimal set of requirement for us to be able to
>> commit to the new platform.
>> Right now I see only one windows related build
>> https://builds.apache.org/view/Hadoop/job/Hadoop-1-win/
>> Which was failing since Sept 8, 2012 and did not run in the last month.
>>
>> Thanks,
>> --Konst
>>
>> On Thu, Feb 28, 2013 at 8:47 PM, Eric Baldeschwieler
>> <er...@hortonworks.com> wrote:
>> > +1 (non-binding)
>> >
>> > A few of observations:
>> >
>> > - Windows has actually been a supported platform for Hadoop since 0.1 .
>> > Doug championed supporting windows then and we've continued to do it with
>> > varying vigor over time.  To my knowledge we've never made a decision to
>> > drop windows support.  The change here is improving our support and dropping
>> > the requirement of cigwin.  We had Nutch windows users on the list in 2006
>> > and we've been supporting windows FS requirements since inception.
>> >
>> > - A little pragmatism will go a long way.  As a community we've got to
>> > stay committed to keeping hadoop simple (so it does work on many platforms)
>> > and extending it to take advantage of key emerging OS/hardware features,
>> > such as containers, new FSs, virtualization, flash ...  We should all plan
>> > to let new features & optimizations emerge that don't work everywhere, if
>> > they are compelling and central to hadoop's mission of being THE best fabric
>> > for storing and processing big data.
>> >
>> > - A UI project like KDE has to deal with the MANY differences between
>> > windows and linux UI APIs.  Hadoop faces no such complex challenge and hence
>> > can be maintained from a single codeline IMO.  It is mostly abstracted from
>> > the OS APIs via Java and our design choices.  Where it is not we can
>> > continue to add plugable abstractions.
>> >
>
>

Re: [Vote] Merge branch-trunk-win to trunk

[Matt Foley <mf...@hortonworks.com>]

Konstantin,
I would like to explore what it would take to remove this perceived
impediment -- although I reserve the right to argue that this is not
pre-requisite to merging the cross-platform support patch.

If we implemented full "test-patch" support for Windows on trunk, would
that fulfill both your items #1 and #2?  Please note that:
a) Pushing the "Patch Available" button in Jira shall cause a pre-commit
build to start within, I believe, 20 minutes.
b) That build keeps logs for both java build and unit tests for several
days, that are accessible to all viewers.

So, does this provide sufficient on-demand support that we don't have to
implement a whole new on-demand VM support structure of some sort for #2
(which would be an extraordinary and impractical demand)?

Thanks,
--Matt


On Fri, Mar 1, 2013 at 12:18 PM, Konstantin Shvachko
<sh...@gmail.com>wrote:

> -1
> We should have a CI infrastructure in place before we can commit to
> supporting Windows platform.
>
> Eric is right Win/Cygwin was supported since day one.
> I had a Windows box under my desk running nightly builds back in 2006-07.
> People were irritated but I was filing windows bugs until 0.22 release.
> Times changing and I am glad to see wider support for Win platform.
>
> But in order to make it work you guys need to put the CI process in place
>
> 1. windows jenkins build: could be nightly or PreCommit.
> - Nightly would mean that changes can be committed to trunk based on
> linux PreCommit build. And people will file bugs if the change broke
> Windows nightly build.
> - PreCommit-win build will mean automatic reporting failed tests to
> respective jira blocking commits the same way as it is now with linux
> PreCommit builds.
> We should discuss which way is more efficient for developers.
>
> 2. On-demand-windows Jenkins build.
> I see it as a build to which I can attach my patch and the build will
> run my changes on a dedicated windows box.
> That way people can test their changes without having personal windows
> nodes.
>
> I think this is the minimal set of requirement for us to be able to
> commit to the new platform.
> Right now I see only one windows related build
> https://builds.apache.org/view/Hadoop/job/Hadoop-1-win/
> Which was failing since Sept 8, 2012 and did not run in the last month.
>
> Thanks,
> --Konst
>
> On Thu, Feb 28, 2013 at 8:47 PM, Eric Baldeschwieler
> <er...@hortonworks.com> wrote:
> > +1 (non-binding)
> >
> > A few of observations:
> >
> > - Windows has actually been a supported platform for Hadoop since 0.1 .
>  Doug championed supporting windows then and we've continued to do it with
> varying vigor over time.  To my knowledge we've never made a decision to
> drop windows support.  The change here is improving our support and
> dropping the requirement of cigwin.  We had Nutch windows users on the list
> in 2006 and we've been supporting windows FS requirements since inception.
> >
> > - A little pragmatism will go a long way.  As a community we've got to
> stay committed to keeping hadoop simple (so it does work on many platforms)
> and extending it to take advantage of key emerging OS/hardware features,
> such as containers, new FSs, virtualization, flash ...  We should all plan
> to let new features & optimizations emerge that don't work everywhere, if
> they are compelling and central to hadoop's mission of being THE best
> fabric for storing and processing big data.
> >
> > - A UI project like KDE has to deal with the MANY differences between
> windows and linux UI APIs.  Hadoop faces no such complex challenge and
> hence can be maintained from a single codeline IMO.  It is mostly
> abstracted from the OS APIs via Java and our design choices.  Where it is
> not we can continue to add plugable abstractions.
> >
>

Re: [Vote] Merge branch-trunk-win to trunk

[Chris Douglas <cd...@apache.org>]

Konstantin-

There's no debate on the necessity of CI and related infrastructure to
support the platform well. Suresh outlined the support to effect this
here: http://s.apache.org/s1

Is the commitment to establish this infrastructure after the merge
sufficient? -C

On Fri, Mar 1, 2013 at 12:18 PM, Konstantin Shvachko
<sh...@gmail.com> wrote:
> -1
> We should have a CI infrastructure in place before we can commit to
> supporting Windows platform.
>
> Eric is right Win/Cygwin was supported since day one.
> I had a Windows box under my desk running nightly builds back in 2006-07.
> People were irritated but I was filing windows bugs until 0.22 release.
> Times changing and I am glad to see wider support for Win platform.
>
> But in order to make it work you guys need to put the CI process in place
>
> 1. windows jenkins build: could be nightly or PreCommit.
> - Nightly would mean that changes can be committed to trunk based on
> linux PreCommit build. And people will file bugs if the change broke
> Windows nightly build.
> - PreCommit-win build will mean automatic reporting failed tests to
> respective jira blocking commits the same way as it is now with linux
> PreCommit builds.
> We should discuss which way is more efficient for developers.
>
> 2. On-demand-windows Jenkins build.
> I see it as a build to which I can attach my patch and the build will
> run my changes on a dedicated windows box.
> That way people can test their changes without having personal windows nodes.
>
> I think this is the minimal set of requirement for us to be able to
> commit to the new platform.
> Right now I see only one windows related build
> https://builds.apache.org/view/Hadoop/job/Hadoop-1-win/
> Which was failing since Sept 8, 2012 and did not run in the last month.
>
> Thanks,
> --Konst
>
> On Thu, Feb 28, 2013 at 8:47 PM, Eric Baldeschwieler
> <er...@hortonworks.com> wrote:
>> +1 (non-binding)
>>
>> A few of observations:
>>
>> - Windows has actually been a supported platform for Hadoop since 0.1 .  Doug championed supporting windows then and we've continued to do it with varying vigor over time.  To my knowledge we've never made a decision to drop windows support.  The change here is improving our support and dropping the requirement of cigwin.  We had Nutch windows users on the list in 2006 and we've been supporting windows FS requirements since inception.
>>
>> - A little pragmatism will go a long way.  As a community we've got to stay committed to keeping hadoop simple (so it does work on many platforms) and extending it to take advantage of key emerging OS/hardware features, such as containers, new FSs, virtualization, flash ...  We should all plan to let new features & optimizations emerge that don't work everywhere, if they are compelling and central to hadoop's mission of being THE best fabric for storing and processing big data.
>>
>> - A UI project like KDE has to deal with the MANY differences between windows and linux UI APIs.  Hadoop faces no such complex challenge and hence can be maintained from a single codeline IMO.  It is mostly abstracted from the OS APIs via Java and our design choices.  Where it is not we can continue to add plugable abstractions.
>>

Re: [Vote] Merge branch-trunk-win to trunk

[Matt Foley <mf...@hortonworks.com>]

Konstantin,
I would like to explore what it would take to remove this perceived
impediment -- although I reserve the right to argue that this is not
pre-requisite to merging the cross-platform support patch.

If we implemented full "test-patch" support for Windows on trunk, would
that fulfill both your items #1 and #2?  Please note that:
a) Pushing the "Patch Available" button in Jira shall cause a pre-commit
build to start within, I believe, 20 minutes.
b) That build keeps logs for both java build and unit tests for several
days, that are accessible to all viewers.

So, does this provide sufficient on-demand support that we don't have to
implement a whole new on-demand VM support structure of some sort for #2
(which would be an extraordinary and impractical demand)?

Thanks,
--Matt


On Fri, Mar 1, 2013 at 12:18 PM, Konstantin Shvachko
<sh...@gmail.com>wrote:

> -1
> We should have a CI infrastructure in place before we can commit to
> supporting Windows platform.
>
> Eric is right Win/Cygwin was supported since day one.
> I had a Windows box under my desk running nightly builds back in 2006-07.
> People were irritated but I was filing windows bugs until 0.22 release.
> Times changing and I am glad to see wider support for Win platform.
>
> But in order to make it work you guys need to put the CI process in place
>
> 1. windows jenkins build: could be nightly or PreCommit.
> - Nightly would mean that changes can be committed to trunk based on
> linux PreCommit build. And people will file bugs if the change broke
> Windows nightly build.
> - PreCommit-win build will mean automatic reporting failed tests to
> respective jira blocking commits the same way as it is now with linux
> PreCommit builds.
> We should discuss which way is more efficient for developers.
>
> 2. On-demand-windows Jenkins build.
> I see it as a build to which I can attach my patch and the build will
> run my changes on a dedicated windows box.
> That way people can test their changes without having personal windows
> nodes.
>
> I think this is the minimal set of requirement for us to be able to
> commit to the new platform.
> Right now I see only one windows related build
> https://builds.apache.org/view/Hadoop/job/Hadoop-1-win/
> Which was failing since Sept 8, 2012 and did not run in the last month.
>
> Thanks,
> --Konst
>
> On Thu, Feb 28, 2013 at 8:47 PM, Eric Baldeschwieler
> <er...@hortonworks.com> wrote:
> > +1 (non-binding)
> >
> > A few of observations:
> >
> > - Windows has actually been a supported platform for Hadoop since 0.1 .
>  Doug championed supporting windows then and we've continued to do it with
> varying vigor over time.  To my knowledge we've never made a decision to
> drop windows support.  The change here is improving our support and
> dropping the requirement of cigwin.  We had Nutch windows users on the list
> in 2006 and we've been supporting windows FS requirements since inception.
> >
> > - A little pragmatism will go a long way.  As a community we've got to
> stay committed to keeping hadoop simple (so it does work on many platforms)
> and extending it to take advantage of key emerging OS/hardware features,
> such as containers, new FSs, virtualization, flash ...  We should all plan
> to let new features & optimizations emerge that don't work everywhere, if
> they are compelling and central to hadoop's mission of being THE best
> fabric for storing and processing big data.
> >
> > - A UI project like KDE has to deal with the MANY differences between
> windows and linux UI APIs.  Hadoop faces no such complex challenge and
> hence can be maintained from a single codeline IMO.  It is mostly
> abstracted from the OS APIs via Java and our design choices.  Where it is
> not we can continue to add plugable abstractions.
> >
>

Re: [Vote] Merge branch-trunk-win to trunk

[Matt Foley <mf...@hortonworks.com>]

Konstantin,
I would like to explore what it would take to remove this perceived
impediment -- although I reserve the right to argue that this is not
pre-requisite to merging the cross-platform support patch.

If we implemented full "test-patch" support for Windows on trunk, would
that fulfill both your items #1 and #2?  Please note that:
a) Pushing the "Patch Available" button in Jira shall cause a pre-commit
build to start within, I believe, 20 minutes.
b) That build keeps logs for both java build and unit tests for several
days, that are accessible to all viewers.

So, does this provide sufficient on-demand support that we don't have to
implement a whole new on-demand VM support structure of some sort for #2
(which would be an extraordinary and impractical demand)?

Thanks,
--Matt


On Fri, Mar 1, 2013 at 12:18 PM, Konstantin Shvachko
<sh...@gmail.com>wrote:

> -1
> We should have a CI infrastructure in place before we can commit to
> supporting Windows platform.
>
> Eric is right Win/Cygwin was supported since day one.
> I had a Windows box under my desk running nightly builds back in 2006-07.
> People were irritated but I was filing windows bugs until 0.22 release.
> Times changing and I am glad to see wider support for Win platform.
>
> But in order to make it work you guys need to put the CI process in place
>
> 1. windows jenkins build: could be nightly or PreCommit.
> - Nightly would mean that changes can be committed to trunk based on
> linux PreCommit build. And people will file bugs if the change broke
> Windows nightly build.
> - PreCommit-win build will mean automatic reporting failed tests to
> respective jira blocking commits the same way as it is now with linux
> PreCommit builds.
> We should discuss which way is more efficient for developers.
>
> 2. On-demand-windows Jenkins build.
> I see it as a build to which I can attach my patch and the build will
> run my changes on a dedicated windows box.
> That way people can test their changes without having personal windows
> nodes.
>
> I think this is the minimal set of requirement for us to be able to
> commit to the new platform.
> Right now I see only one windows related build
> https://builds.apache.org/view/Hadoop/job/Hadoop-1-win/
> Which was failing since Sept 8, 2012 and did not run in the last month.
>
> Thanks,
> --Konst
>
> On Thu, Feb 28, 2013 at 8:47 PM, Eric Baldeschwieler
> <er...@hortonworks.com> wrote:
> > +1 (non-binding)
> >
> > A few of observations:
> >
> > - Windows has actually been a supported platform for Hadoop since 0.1 .
>  Doug championed supporting windows then and we've continued to do it with
> varying vigor over time.  To my knowledge we've never made a decision to
> drop windows support.  The change here is improving our support and
> dropping the requirement of cigwin.  We had Nutch windows users on the list
> in 2006 and we've been supporting windows FS requirements since inception.
> >
> > - A little pragmatism will go a long way.  As a community we've got to
> stay committed to keeping hadoop simple (so it does work on many platforms)
> and extending it to take advantage of key emerging OS/hardware features,
> such as containers, new FSs, virtualization, flash ...  We should all plan
> to let new features & optimizations emerge that don't work everywhere, if
> they are compelling and central to hadoop's mission of being THE best
> fabric for storing and processing big data.
> >
> > - A UI project like KDE has to deal with the MANY differences between
> windows and linux UI APIs.  Hadoop faces no such complex challenge and
> hence can be maintained from a single codeline IMO.  It is mostly
> abstracted from the OS APIs via Java and our design choices.  Where it is
> not we can continue to add plugable abstractions.
> >
>

Re: [Vote] Merge branch-trunk-win to trunk

[Matt Foley <mf...@hortonworks.com>]

Konstantin,
I would like to explore what it would take to remove this perceived
impediment -- although I reserve the right to argue that this is not
pre-requisite to merging the cross-platform support patch.

If we implemented full "test-patch" support for Windows on trunk, would
that fulfill both your items #1 and #2?  Please note that:
a) Pushing the "Patch Available" button in Jira shall cause a pre-commit
build to start within, I believe, 20 minutes.
b) That build keeps logs for both java build and unit tests for several
days, that are accessible to all viewers.

So, does this provide sufficient on-demand support that we don't have to
implement a whole new on-demand VM support structure of some sort for #2
(which would be an extraordinary and impractical demand)?

Thanks,
--Matt


On Fri, Mar 1, 2013 at 12:18 PM, Konstantin Shvachko
<sh...@gmail.com>wrote:

> -1
> We should have a CI infrastructure in place before we can commit to
> supporting Windows platform.
>
> Eric is right Win/Cygwin was supported since day one.
> I had a Windows box under my desk running nightly builds back in 2006-07.
> People were irritated but I was filing windows bugs until 0.22 release.
> Times changing and I am glad to see wider support for Win platform.
>
> But in order to make it work you guys need to put the CI process in place
>
> 1. windows jenkins build: could be nightly or PreCommit.
> - Nightly would mean that changes can be committed to trunk based on
> linux PreCommit build. And people will file bugs if the change broke
> Windows nightly build.
> - PreCommit-win build will mean automatic reporting failed tests to
> respective jira blocking commits the same way as it is now with linux
> PreCommit builds.
> We should discuss which way is more efficient for developers.
>
> 2. On-demand-windows Jenkins build.
> I see it as a build to which I can attach my patch and the build will
> run my changes on a dedicated windows box.
> That way people can test their changes without having personal windows
> nodes.
>
> I think this is the minimal set of requirement for us to be able to
> commit to the new platform.
> Right now I see only one windows related build
> https://builds.apache.org/view/Hadoop/job/Hadoop-1-win/
> Which was failing since Sept 8, 2012 and did not run in the last month.
>
> Thanks,
> --Konst
>
> On Thu, Feb 28, 2013 at 8:47 PM, Eric Baldeschwieler
> <er...@hortonworks.com> wrote:
> > +1 (non-binding)
> >
> > A few of observations:
> >
> > - Windows has actually been a supported platform for Hadoop since 0.1 .
>  Doug championed supporting windows then and we've continued to do it with
> varying vigor over time.  To my knowledge we've never made a decision to
> drop windows support.  The change here is improving our support and
> dropping the requirement of cigwin.  We had Nutch windows users on the list
> in 2006 and we've been supporting windows FS requirements since inception.
> >
> > - A little pragmatism will go a long way.  As a community we've got to
> stay committed to keeping hadoop simple (so it does work on many platforms)
> and extending it to take advantage of key emerging OS/hardware features,
> such as containers, new FSs, virtualization, flash ...  We should all plan
> to let new features & optimizations emerge that don't work everywhere, if
> they are compelling and central to hadoop's mission of being THE best
> fabric for storing and processing big data.
> >
> > - A UI project like KDE has to deal with the MANY differences between
> windows and linux UI APIs.  Hadoop faces no such complex challenge and
> hence can be maintained from a single codeline IMO.  It is mostly
> abstracted from the OS APIs via Java and our design choices.  Where it is
> not we can continue to add plugable abstractions.
> >
>

Re: [Vote] Merge branch-trunk-win to trunk

[Chris Douglas <cd...@apache.org>]

Konstantin-

There's no debate on the necessity of CI and related infrastructure to
support the platform well. Suresh outlined the support to effect this
here: http://s.apache.org/s1

Is the commitment to establish this infrastructure after the merge
sufficient? -C

On Fri, Mar 1, 2013 at 12:18 PM, Konstantin Shvachko
<sh...@gmail.com> wrote:
> -1
> We should have a CI infrastructure in place before we can commit to
> supporting Windows platform.
>
> Eric is right Win/Cygwin was supported since day one.
> I had a Windows box under my desk running nightly builds back in 2006-07.
> People were irritated but I was filing windows bugs until 0.22 release.
> Times changing and I am glad to see wider support for Win platform.
>
> But in order to make it work you guys need to put the CI process in place
>
> 1. windows jenkins build: could be nightly or PreCommit.
> - Nightly would mean that changes can be committed to trunk based on
> linux PreCommit build. And people will file bugs if the change broke
> Windows nightly build.
> - PreCommit-win build will mean automatic reporting failed tests to
> respective jira blocking commits the same way as it is now with linux
> PreCommit builds.
> We should discuss which way is more efficient for developers.
>
> 2. On-demand-windows Jenkins build.
> I see it as a build to which I can attach my patch and the build will
> run my changes on a dedicated windows box.
> That way people can test their changes without having personal windows nodes.
>
> I think this is the minimal set of requirement for us to be able to
> commit to the new platform.
> Right now I see only one windows related build
> https://builds.apache.org/view/Hadoop/job/Hadoop-1-win/
> Which was failing since Sept 8, 2012 and did not run in the last month.
>
> Thanks,
> --Konst
>
> On Thu, Feb 28, 2013 at 8:47 PM, Eric Baldeschwieler
> <er...@hortonworks.com> wrote:
>> +1 (non-binding)
>>
>> A few of observations:
>>
>> - Windows has actually been a supported platform for Hadoop since 0.1 .  Doug championed supporting windows then and we've continued to do it with varying vigor over time.  To my knowledge we've never made a decision to drop windows support.  The change here is improving our support and dropping the requirement of cigwin.  We had Nutch windows users on the list in 2006 and we've been supporting windows FS requirements since inception.
>>
>> - A little pragmatism will go a long way.  As a community we've got to stay committed to keeping hadoop simple (so it does work on many platforms) and extending it to take advantage of key emerging OS/hardware features, such as containers, new FSs, virtualization, flash ...  We should all plan to let new features & optimizations emerge that don't work everywhere, if they are compelling and central to hadoop's mission of being THE best fabric for storing and processing big data.
>>
>> - A UI project like KDE has to deal with the MANY differences between windows and linux UI APIs.  Hadoop faces no such complex challenge and hence can be maintained from a single codeline IMO.  It is mostly abstracted from the OS APIs via Java and our design choices.  Where it is not we can continue to add plugable abstractions.
>>

Re: [Vote] Merge branch-trunk-win to trunk

[Chris Douglas <cd...@apache.org>]

Konstantin-

There's no debate on the necessity of CI and related infrastructure to
support the platform well. Suresh outlined the support to effect this
here: http://s.apache.org/s1

Is the commitment to establish this infrastructure after the merge
sufficient? -C

On Fri, Mar 1, 2013 at 12:18 PM, Konstantin Shvachko
<sh...@gmail.com> wrote:
> -1
> We should have a CI infrastructure in place before we can commit to
> supporting Windows platform.
>
> Eric is right Win/Cygwin was supported since day one.
> I had a Windows box under my desk running nightly builds back in 2006-07.
> People were irritated but I was filing windows bugs until 0.22 release.
> Times changing and I am glad to see wider support for Win platform.
>
> But in order to make it work you guys need to put the CI process in place
>
> 1. windows jenkins build: could be nightly or PreCommit.
> - Nightly would mean that changes can be committed to trunk based on
> linux PreCommit build. And people will file bugs if the change broke
> Windows nightly build.
> - PreCommit-win build will mean automatic reporting failed tests to
> respective jira blocking commits the same way as it is now with linux
> PreCommit builds.
> We should discuss which way is more efficient for developers.
>
> 2. On-demand-windows Jenkins build.
> I see it as a build to which I can attach my patch and the build will
> run my changes on a dedicated windows box.
> That way people can test their changes without having personal windows nodes.
>
> I think this is the minimal set of requirement for us to be able to
> commit to the new platform.
> Right now I see only one windows related build
> https://builds.apache.org/view/Hadoop/job/Hadoop-1-win/
> Which was failing since Sept 8, 2012 and did not run in the last month.
>
> Thanks,
> --Konst
>
> On Thu, Feb 28, 2013 at 8:47 PM, Eric Baldeschwieler
> <er...@hortonworks.com> wrote:
>> +1 (non-binding)
>>
>> A few of observations:
>>
>> - Windows has actually been a supported platform for Hadoop since 0.1 .  Doug championed supporting windows then and we've continued to do it with varying vigor over time.  To my knowledge we've never made a decision to drop windows support.  The change here is improving our support and dropping the requirement of cigwin.  We had Nutch windows users on the list in 2006 and we've been supporting windows FS requirements since inception.
>>
>> - A little pragmatism will go a long way.  As a community we've got to stay committed to keeping hadoop simple (so it does work on many platforms) and extending it to take advantage of key emerging OS/hardware features, such as containers, new FSs, virtualization, flash ...  We should all plan to let new features & optimizations emerge that don't work everywhere, if they are compelling and central to hadoop's mission of being THE best fabric for storing and processing big data.
>>
>> - A UI project like KDE has to deal with the MANY differences between windows and linux UI APIs.  Hadoop faces no such complex challenge and hence can be maintained from a single codeline IMO.  It is mostly abstracted from the OS APIs via Java and our design choices.  Where it is not we can continue to add plugable abstractions.
>>

Re: [Vote] Merge branch-trunk-win to trunk

[Konstantin Shvachko <sh...@gmail.com>]

-1
We should have a CI infrastructure in place before we can commit to
supporting Windows platform.

Eric is right Win/Cygwin was supported since day one.
I had a Windows box under my desk running nightly builds back in 2006-07.
People were irritated but I was filing windows bugs until 0.22 release.
Times changing and I am glad to see wider support for Win platform.

But in order to make it work you guys need to put the CI process in place

1. windows jenkins build: could be nightly or PreCommit.
- Nightly would mean that changes can be committed to trunk based on
linux PreCommit build. And people will file bugs if the change broke
Windows nightly build.
- PreCommit-win build will mean automatic reporting failed tests to
respective jira blocking commits the same way as it is now with linux
PreCommit builds.
We should discuss which way is more efficient for developers.

2. On-demand-windows Jenkins build.
I see it as a build to which I can attach my patch and the build will
run my changes on a dedicated windows box.
That way people can test their changes without having personal windows nodes.

I think this is the minimal set of requirement for us to be able to
commit to the new platform.
Right now I see only one windows related build
https://builds.apache.org/view/Hadoop/job/Hadoop-1-win/
Which was failing since Sept 8, 2012 and did not run in the last month.

Thanks,
--Konst

On Thu, Feb 28, 2013 at 8:47 PM, Eric Baldeschwieler
<er...@hortonworks.com> wrote:
> +1 (non-binding)
>
> A few of observations:
>
> - Windows has actually been a supported platform for Hadoop since 0.1 .  Doug championed supporting windows then and we've continued to do it with varying vigor over time.  To my knowledge we've never made a decision to drop windows support.  The change here is improving our support and dropping the requirement of cigwin.  We had Nutch windows users on the list in 2006 and we've been supporting windows FS requirements since inception.
>
> - A little pragmatism will go a long way.  As a community we've got to stay committed to keeping hadoop simple (so it does work on many platforms) and extending it to take advantage of key emerging OS/hardware features, such as containers, new FSs, virtualization, flash ...  We should all plan to let new features & optimizations emerge that don't work everywhere, if they are compelling and central to hadoop's mission of being THE best fabric for storing and processing big data.
>
> - A UI project like KDE has to deal with the MANY differences between windows and linux UI APIs.  Hadoop faces no such complex challenge and hence can be maintained from a single codeline IMO.  It is mostly abstracted from the OS APIs via Java and our design choices.  Where it is not we can continue to add plugable abstractions.
>

Re: [Vote] Merge branch-trunk-win to trunk

[Konstantin Shvachko <sh...@gmail.com>]

-1
We should have a CI infrastructure in place before we can commit to
supporting Windows platform.

Eric is right Win/Cygwin was supported since day one.
I had a Windows box under my desk running nightly builds back in 2006-07.
People were irritated but I was filing windows bugs until 0.22 release.
Times changing and I am glad to see wider support for Win platform.

But in order to make it work you guys need to put the CI process in place

1. windows jenkins build: could be nightly or PreCommit.
- Nightly would mean that changes can be committed to trunk based on
linux PreCommit build. And people will file bugs if the change broke
Windows nightly build.
- PreCommit-win build will mean automatic reporting failed tests to
respective jira blocking commits the same way as it is now with linux
PreCommit builds.
We should discuss which way is more efficient for developers.

2. On-demand-windows Jenkins build.
I see it as a build to which I can attach my patch and the build will
run my changes on a dedicated windows box.
That way people can test their changes without having personal windows nodes.

I think this is the minimal set of requirement for us to be able to
commit to the new platform.
Right now I see only one windows related build
https://builds.apache.org/view/Hadoop/job/Hadoop-1-win/
Which was failing since Sept 8, 2012 and did not run in the last month.

Thanks,
--Konst

On Thu, Feb 28, 2013 at 8:47 PM, Eric Baldeschwieler
<er...@hortonworks.com> wrote:
> +1 (non-binding)
>
> A few of observations:
>
> - Windows has actually been a supported platform for Hadoop since 0.1 .  Doug championed supporting windows then and we've continued to do it with varying vigor over time.  To my knowledge we've never made a decision to drop windows support.  The change here is improving our support and dropping the requirement of cigwin.  We had Nutch windows users on the list in 2006 and we've been supporting windows FS requirements since inception.
>
> - A little pragmatism will go a long way.  As a community we've got to stay committed to keeping hadoop simple (so it does work on many platforms) and extending it to take advantage of key emerging OS/hardware features, such as containers, new FSs, virtualization, flash ...  We should all plan to let new features & optimizations emerge that don't work everywhere, if they are compelling and central to hadoop's mission of being THE best fabric for storing and processing big data.
>
> - A UI project like KDE has to deal with the MANY differences between windows and linux UI APIs.  Hadoop faces no such complex challenge and hence can be maintained from a single codeline IMO.  It is mostly abstracted from the OS APIs via Java and our design choices.  Where it is not we can continue to add plugable abstractions.
>

Re: [Vote] Merge branch-trunk-win to trunk

[Konstantin Shvachko <sh...@gmail.com>]

-1
We should have a CI infrastructure in place before we can commit to
supporting Windows platform.

Eric is right Win/Cygwin was supported since day one.
I had a Windows box under my desk running nightly builds back in 2006-07.
People were irritated but I was filing windows bugs until 0.22 release.
Times changing and I am glad to see wider support for Win platform.

But in order to make it work you guys need to put the CI process in place

1. windows jenkins build: could be nightly or PreCommit.
- Nightly would mean that changes can be committed to trunk based on
linux PreCommit build. And people will file bugs if the change broke
Windows nightly build.
- PreCommit-win build will mean automatic reporting failed tests to
respective jira blocking commits the same way as it is now with linux
PreCommit builds.
We should discuss which way is more efficient for developers.

2. On-demand-windows Jenkins build.
I see it as a build to which I can attach my patch and the build will
run my changes on a dedicated windows box.
That way people can test their changes without having personal windows nodes.

I think this is the minimal set of requirement for us to be able to
commit to the new platform.
Right now I see only one windows related build
https://builds.apache.org/view/Hadoop/job/Hadoop-1-win/
Which was failing since Sept 8, 2012 and did not run in the last month.

Thanks,
--Konst

On Thu, Feb 28, 2013 at 8:47 PM, Eric Baldeschwieler
<er...@hortonworks.com> wrote:
> +1 (non-binding)
>
> A few of observations:
>
> - Windows has actually been a supported platform for Hadoop since 0.1 .  Doug championed supporting windows then and we've continued to do it with varying vigor over time.  To my knowledge we've never made a decision to drop windows support.  The change here is improving our support and dropping the requirement of cigwin.  We had Nutch windows users on the list in 2006 and we've been supporting windows FS requirements since inception.
>
> - A little pragmatism will go a long way.  As a community we've got to stay committed to keeping hadoop simple (so it does work on many platforms) and extending it to take advantage of key emerging OS/hardware features, such as containers, new FSs, virtualization, flash ...  We should all plan to let new features & optimizations emerge that don't work everywhere, if they are compelling and central to hadoop's mission of being THE best fabric for storing and processing big data.
>
> - A UI project like KDE has to deal with the MANY differences between windows and linux UI APIs.  Hadoop faces no such complex challenge and hence can be maintained from a single codeline IMO.  It is mostly abstracted from the OS APIs via Java and our design choices.  Where it is not we can continue to add plugable abstractions.
>

Re: [Vote] Merge branch-trunk-win to trunk

[Konstantin Shvachko <sh...@gmail.com>]

-1
We should have a CI infrastructure in place before we can commit to
supporting Windows platform.

Eric is right Win/Cygwin was supported since day one.
I had a Windows box under my desk running nightly builds back in 2006-07.
People were irritated but I was filing windows bugs until 0.22 release.
Times changing and I am glad to see wider support for Win platform.

But in order to make it work you guys need to put the CI process in place

1. windows jenkins build: could be nightly or PreCommit.
- Nightly would mean that changes can be committed to trunk based on
linux PreCommit build. And people will file bugs if the change broke
Windows nightly build.
- PreCommit-win build will mean automatic reporting failed tests to
respective jira blocking commits the same way as it is now with linux
PreCommit builds.
We should discuss which way is more efficient for developers.

2. On-demand-windows Jenkins build.
I see it as a build to which I can attach my patch and the build will
run my changes on a dedicated windows box.
That way people can test their changes without having personal windows nodes.

I think this is the minimal set of requirement for us to be able to
commit to the new platform.
Right now I see only one windows related build
https://builds.apache.org/view/Hadoop/job/Hadoop-1-win/
Which was failing since Sept 8, 2012 and did not run in the last month.

Thanks,
--Konst

On Thu, Feb 28, 2013 at 8:47 PM, Eric Baldeschwieler
<er...@hortonworks.com> wrote:
> +1 (non-binding)
>
> A few of observations:
>
> - Windows has actually been a supported platform for Hadoop since 0.1 .  Doug championed supporting windows then and we've continued to do it with varying vigor over time.  To my knowledge we've never made a decision to drop windows support.  The change here is improving our support and dropping the requirement of cigwin.  We had Nutch windows users on the list in 2006 and we've been supporting windows FS requirements since inception.
>
> - A little pragmatism will go a long way.  As a community we've got to stay committed to keeping hadoop simple (so it does work on many platforms) and extending it to take advantage of key emerging OS/hardware features, such as containers, new FSs, virtualization, flash ...  We should all plan to let new features & optimizations emerge that don't work everywhere, if they are compelling and central to hadoop's mission of being THE best fabric for storing and processing big data.
>
> - A UI project like KDE has to deal with the MANY differences between windows and linux UI APIs.  Hadoop faces no such complex challenge and hence can be maintained from a single codeline IMO.  It is mostly abstracted from the OS APIs via Java and our design choices.  Where it is not we can continue to add plugable abstractions.
>

Re: [Vote] Merge branch-trunk-win to trunk

[Eric Baldeschwieler <er...@hortonworks.com>]

+1 (non-binding)

A few of observations:

- Windows has actually been a supported platform for Hadoop since 0.1 .  Doug championed supporting windows then and we've continued to do it with varying vigor over time.  To my knowledge we've never made a decision to drop windows support.  The change here is improving our support and dropping the requirement of cigwin.  We had Nutch windows users on the list in 2006 and we've been supporting windows FS requirements since inception.

- A little pragmatism will go a long way.  As a community we've got to stay committed to keeping hadoop simple (so it does work on many platforms) and extending it to take advantage of key emerging OS/hardware features, such as containers, new FSs, virtualization, flash ...  We should all plan to let new features & optimizations emerge that don't work everywhere, if they are compelling and central to hadoop's mission of being THE best fabric for storing and processing big data.  

- A UI project like KDE has to deal with the MANY differences between windows and linux UI APIs.  Hadoop faces no such complex challenge and hence can be maintained from a single codeline IMO.  It is mostly abstracted from the OS APIs via Java and our design choices.  Where it is not we can continue to add plugable abstractions.

On Feb 28, 2013, at 6:01 PM, Matt Foley <ma...@apache.org> wrote:

> +1 (binding)
> 
> Apache is supposed to be about the community.  We have here a community of
> developers, who have actively and openly worked to add a major improvement
> to Hadoop: the ability to work cross-platform.  Furthermore, the size of
> the substantive part of the needed patch is only about 1500 lines, much
> smaller than quite a few other additions to Hadoop over the last few
> months.  We should welcome and support this change, and make sure that the
> code stays cross-platform going forward by extending our CI practices,
> especially pre-commit "test-patch", to also include Windows.
> 
> As most of you know, my colleague Giri Kesavan (PMC member) helps maintain
> the Linux CI capability for Hadoop.  I've talked with him, and he and I are
> committing to getting test-patch implemented for Windows, so that along
> with the current automated "+1"s required to commit, we can add two more,
> for javac build in Windows and core unit tests in Windows.
> 
> Members of the team implementing cross-platform compatibility, including
> Microsoft employees, have opened the discussion for providing hardware or
> VM resources to perform this additional CI testing.  I will assist them to
> work with the Apache Infra team and figure out how to make it happen.
> 
> I understand there is some concern about the additional platform test.
> My going-in
> presumption, based on Java's intrinsic, pretty-good, cross-platform
> compatibility, is that patches to Hadoop will by default also have
> cross-platform compatibility, unless they are written in an explicitly
> platform-dependent way.  I also believe that in the vast majority of cases
> the cross-platform compatibility of Java will carry thru to Hadoop patches,
> without additional effort on the developer's part.
> 
> Let's try it, and see what happens.  If we actually find a frequent
> difficulty, we'll change to engineer around it.  But I believe that, in the
> rare cases where a Windows-specific failure occurs, there will be a number
> of people (new, enthusiastic members of the community! :-) willing to help.
> If such help is not forthcoming, then we can discuss work-arounds, but
> like a previous poster, I am confident in the community.
> 
> Regards,
> --Matt
> 
> 
> 
> On Thu, Feb 28, 2013 at 12:21 PM, Chuan Liu <ch...@microsoft.com> wrote:
> 
>> +1 (non-binding)
>>> 
>>> As someone also contributed to porting Hadoop to Windows, I think Java
>>> already provided a very good platform independent platform.
>>> For features that are not available in Java, we will try to provide our
>>> platform independent APIs that abstract OS tasks away.
>>> Most features should have no difficulty running on Windows and Linux by
>>> using Java and those platform independent APIs.
>>> 
>>> For concerns raise on new features that may fail on Windows, I think we
>>> don't need to require passing on Windows a mandate at the moment. We can
>>> simply mark it unavailable to Windows and port it later if the feature is
>>> important.
>>> 
>>> -Chuan
>>> 
>>> -----Original Message-----
>>> From: Chris Nauroth [mailto:cnauroth@hortonworks.com]
>>> Sent: Thursday, February 28, 2013 11:51 AM
>>> To: hdfs-dev@hadoop.apache.org
>>> Cc: mapreduce-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org;
>>> common-dev@hadoop.apache.org
>>> Subject: Re: [Vote] Merge branch-trunk-win to trunk
>>> 
>>>> Is there a jira for resolving the outstanding TODOs in the code base
>>>> (similar to HDFS-2148)?  Looks like this merge doesn't introduce many
>>>> which is great (just did a quick diff and grep).
>>> 
>>> I found 2 remaining TODOs introduced in the current merge patch.  One is
>>> in ContainerLaunch.java.  The container launch script was trying to set a
>>> CLASSPATH that exceeded the Windows maximum command line length.  The fix
>>> was to wrap the long classpath into an intermediate jar containing only a
>>> manifest file with a Class-Path entry.  (See YARN-316.)  Just to be
>>> conservative, we wrapped this logic in an if (Shell.WINDOWS) guard and
>>> marked a TODO to remove it later and use that approach on all platforms
>>> after additional testing.  I've tested this code path successfully on Mac
>>> too, but several people wanted additional testing and performance checks
>>> before removing the if (Shell.WINDOWS) guard.  That work is tracked in an
>>> existing jira: YARN-358.
>>> 
>>> The other TODO is for winutils to print more usage information and
>>> examples.  At this point, I think winutils is printing sufficient
>>> information, and we can just remove the TODO.  I just submitted a new jira
>>> to start that conversation: HADOOP-9348.
>>> 
>>> Thank you,
>>> --Chris
>>> 
>>> 
>>> On Thu, Feb 28, 2013 at 11:29 AM, Robert Evans <ev...@yahoo-inc.com>
>>> wrote:
>>> 
>>>> My initial question was mostly intended to understand the desired new
>>>> classification of Windows after the merge, and how we plan to maintain
>>>> Windows support.  I am happy to hear that hardware for Jenkins will be
>>>> provided.  I am also fine, at least initially, with us trying to treat
>>>> Windows as a first class supported platform.  But I realize that there
>>>> are a lot of people that do not have easy access to Windows for
>>>> development/debugging, myself included. I also don't want to slow down
>>>> the pace of development too much because of this.  It will cause some
>>>> organizations that do not use or support Windows to be more likely to
>>>> run software that has diverged from an official release.  It also has
>>>> the potential to make the patch submission process even more
>>>> difficult, which increases the likelihood of submitters abandoning
>>>> patches.  However, the great thing about being in a community is we can
>>> change if we need to.
>>>> 
>>>> I am +0 for the merge.  I am not a Windows expert so I don't feel
>>>> comfortable giving it a true +1.
>>>> 
>>>> --Bobby
>>>> 
>>>> 
>>>> On 2/28/13 10:45 AM, "Chris Nauroth" <cn...@hortonworks.com> wrote:
>>>> 
>>>>> I'd like to share a few anecdotes about developing cross-platform,
>>>>> hopefully to address some of the concerns about adding overhead to
>>>>> the development process.  By reviewing past cases of cross-platform
>>> Linux vs.
>>>>> Windows bugs, we can get a sense for how the development process
>>>>> could look in the future.
>>>>> 
>>>>> HADOOP-9131: TestLocalFileSystem#testListStatusWithColons cannot run
>>>>> on Windows.  As part of an earlier jira, HADOOP-8962, there was a new
>>>>> test committed on trunk covering the case of a local file system
>>>>> interaction on a file containing a ':'.  On Windows, ':' in a path
>>>>> has special meaning as part of the drive specifier (i.e. C:), so this
>>>>> test cannot pass when running on Windows.  In this kind of case, the
>>>>> cross-platform bug is obvious, and the fix is obvious
>>>>> (assumeTrue(!Shell.WINDOWS)).  Ideally, this would get fixed
>>>>> pre-commit after seeing a -1 from the Windows Jenkins slave.
>>>>> 
>>>>> HDFS-4274: BlockPoolSliceScanner does not close verification log
>>>>> during shutdown.  This caused problems for MiniDFSCluster-based tests
>>>>> running on Windows.  Failure to close the verification log meant that
>>>>> we didn't release file locks, so the tests couldn't delete/recreate
>>>>> working directories during teardown/setup.  Arguably, this was always
>>>>> a bug, and running on Windows just exposed it because of its stricter
>>>>> rules about file locking.  This is a more complex fix, but it doesn't
>>>>> require platform-specific knowledge.  If some future patch
>>>>> accidentally regresses this, then we'll likely see +1 from Linux
>>>>> Jenkins and -1 from Windows Jenkins.  Ideally, it would get fixed
>>>>> pre-commit, because it doesn't require Windows-specific knowledge.
>>>>> There is also the matter of impact.
>>>>> Re-breaking this would re-break many test suites on Windows.
>>>>> 
>>>>> HADOOP-9232: JniBasedUnixGroupsMappingWithFallback fails on Windows
>>>>> with UnsatisfiedLinkError.  This was introduced by HADOOP-8712, which
>>>>> switched to JniBasedUnixGroupsMappingWithFallback as the default
>>>>> hadoop.security.group.mapping, but did not provide a Windows
>>>>> implementation of the JNI function.  In this case, there was a strong
>>>>> desire to get
>>>>> HADOOP-8712 into a release, fixing it on Windows required native
>>>>> Windows API knowledge, and Windows users had a simple workaround
>>>>> available by changing their configs back to
>>>>> ShellBasedUnixGroupsMapping.  I think this is the kind of situation
>>>>> where we could allow HADOOP-8712 to commit despite
>>>>> -1 from Windows Jenkins, with fairly quick follow-up from an engineer
>>>>> with the Windows expertise to fix it.
>>>>> 
>>>>> To summarize, I don't think it needs to differ greatly from our
>>>>> current development process.  We're all responsible for breadth of
>>>>> understanding and maintenance of the whole codebase, but we also rely
>>>>> on specific individuals with deep expertise in particular areas for
>>> certain issues.
>>>>> Sometimes we commit despite a -1 from Jenkins, based on the
>>>>> community's judgment.
>>>>> 
>>>>> Virtualization greatly simplifies cross-platform development.  I use
>>>>> VirtualBox on a Mac host and run VMs for Windows and Ubuntu with a
>>>>> shared drive so that they can all see the same copy of the source
>>>>> code.  There are plenty of variations on this depending on your
>>>>> preference, such as offloading the VMs to a separate server or cloud
>>>>> service to free up local RAM.  I'm planning on submitting
>>>>> BUILDING.txt changes later today that fully describe how to build on
>>>>> Windows.  After some initial setup, it's nearly identical to the mvn
>>>>> commands that you already use today.
>>>>> 
>>>>> Hope this helps,
>>>>> --Chris
>>>>> 
>>>>> 
>>>>> On Thu, Feb 28, 2013 at 3:25 AM, John Gordon
>>>>> <Jo...@microsoft.com>wrote:
>>>>> 
>>>>>> +1 (non-binding)
>>>>>> 
>>>>>> I want to share my vote of confidence in this community.  If
>>>>>> motivated to  do so, this community can keep this project
>>>>>> cross-platform and continue to  rapidly innovate without breaking a
>>>>>> sweat.
>>>>>> 
>>>>>> The day we started working on this, I saw the foundations of
>>>>>> greatness in  the quality and volume of dev tests, the code itself,
>>>>>> and the Apache values  themselves.
>>>>>> 
>>>>>> 1.) Hadoop's unit tests and their frameworks are very well thought
>>>>>> out and  the consideration and energy that went into their design is
>>>>>> worthy of  praise.  The MiniCluster abstractions utilize very few
>>>>>> resources and put  all the processes into one JVM for easy
>>>>>> debugging.  It is very easy to  select specific tests from the full
>>>>>> suite to reproduce an issue reported in  another environment - like
>>>>>> the Jenkins build server or another  contributor's environment.
>>>>>> 2.) This community has done an excellent job of incorporating
>>>>>> well-placed  log messages to make it easy to post mortem
>>>>>> troubleshoot most failures.
>>>>>> The logs are very useful, and it is extremely rare that
>>>>>> troubleshooting a  failure requires debugging a live repro.
>>>>>> 3.) Hadoop is written primarily in Java, a cross-platform language
>>>>>> that  provides its own platform in the form of the JVM to insulate
>>>>>> most of the  code from the specifics of the OS layer.
>>>>>> 4.) CoPDoC - The right priorities, and well stated.
>>>>>> 
>>>>>> 
>>>>>> Thank you,
>>>>>> 
>>>>>> John
>>>>>> 
>>>>>> -----Original Message-----
>>>>>> From: Ivan Mitic [mailto:ivanmi@microsoft.com]
>>>>>> Sent: Wednesday, February 27, 2013 6:32 PM
>>>>>> To: mapreduce-dev@hadoop.apache.org; common-dev@hadoop.apache.org
>>>>>> Cc: yarn-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org
>>>>>> Subject: RE: [Vote] Merge branch-trunk-win to trunk
>>>>>> 
>>>>>> +1 (non-binding)
>>>>>> 
>>>>>> I am really glad to see this happening! As people already
>>>>>> mentioned, this  has been a great engineering effort involving many
>>>>>> people!
>>>>>> 
>>>>>> 
>>>>>> Folks raised some valid concerns below and I thought it would be
>>>>>> good to  share my 2 cents. In my opinion, we don't have to solve all
>>>>>> these problems  right now. As we move forward with two platforms, we
>>>>>> can start addressing  one problem at a time and incrementally
>>>>>> improve. In the first iteration,  maintaining Hadoop on Windows
>>>>>> could be just everyone trying to do their  best effort (make sure
>>>>>> Jenkins build succeeds at least). We already have  people who are
>>>>>> building/running trunk on Windows daily, so they would jump  in and
>>>>>> fix problems as needed (we've been doing this in branch-trunk-win
>>>>>> for a while now). Although I see that the problems could arise with
>>>>>> platform specific features/optimizations, I don't think these are
>>>>>> frequent,  so in most cases everything will just work. Merging the
>>>>>> two branches sooner  rather than later does seems like the right
>>>>>> thing to do if the ultimate  goal is to have Hadoop on both
>>>>>> platforms. Now that the port has completed,  we will have people in
>>>>>> Microsoft (and elsewhere) wanting to contribute
>>>>>> features/improvements to the trunk branch. A separate branch would
>>>>>> just  make things more difficult and confusing for everyone :) Hope
>>>>>> this makes  sense.
>>>>>> 
>>>>>> -----Original Message-----
>>>>>> From: Todd Lipcon [mailto:todd@cloudera.com]
>>>>>> Sent: Wednesday, February 27, 2013 3:43 PM
>>>>>> To: common-dev@hadoop.apache.org
>>>>>> Cc: yarn-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org;
>>>>>> mapreduce-dev@hadoop.apache.org
>>>>>> Subject: Re: [Vote] Merge branch-trunk-win to trunk
>>>>>> 
>>>>>> On Wed, Feb 27, 2013 at 2:54 PM, Suresh Srinivas <
>>>> suresh@hortonworks.com
>>>>>>> wrote:
>>>>>> 
>>>>>>> With that we need to decide how our precommit process looks.
>>>>>>> My inclination is to wait for +1 from precommit builds on both
>>>>>>> the platforms to ensure no issues are introduced.
>>>>>>> Thoughts?
>>>>>>> 
>>>>>>> 2. Feature development impact
>>>>>>> Some questions have been raised about would new features need to
>>>>>>> be supported on both the platforms. Yes. I do not see a reason
>>>>>>> why features cannot work on both the platforms, with the
>>>>>>> exception of platform specific optimizations. This what Java gives
>>> us.
>>>>>>> 
>>>>>>> 
>>>>>> I'm concerned about the above. Personally, I don't have access to
>>>>>> any  Windows boxes with development tools, and I know nothing about
>>>>>> developing  on Windows. The only Windows I run is an 8GB VM with 1
>>>>>> GB RAM allocated,  for powerpoint :)
>>>>>> 
>>>>>> If I submit a patch and it gets -1 "tests failed" on the Windows
>>>>>> slave, how am I supposed to proceed?
>>>>>> 
>>>>>> I think a reasonable compromise would be that the tests should
>>>>>> always
>>>>>> *build* on Windows before commit, and contributors should do their
>>>>>> best to  look at the test logs for any Windows-specific failures.
>>>>>> But, beyond  looking at the logs, a "-1 Tests failed on windows"
>>>>>> should not block a  commit.
>>>>>> 
>>>>>> Those contributors who are interested in Windows being a
>>>>>> first-class platform should be responsible for watching the Windows
>>>>>> builds and debugging/fixing any regressions that might be
>>> Windows-specific.
>>>>>> 
>>>>>> I also think the KDE model that Harsh pointed out is an interesting
>>>>>> one
>>>>>> --
>>>>>> ie the idea that we would not merge windows support to trunk, but
>>>>>> rather  treat is as a "parallel code line" which lives in the ASF
>>>>>> and has its own  builds and releases. The windows team would
>>>>>> periodically merge
>>>>>> trunk->win
>>>>>> to pick up any new changes, and do a separate test/release process.
>>>>>> I'm not  convinced this is the best idea, but worth discussion of
>>>>>> pros and cons.
>>>>>> 
>>>>>> -Todd
>>>>>> 
>>>>>> 
>>>>>>> 
>>>>>>> On Wed, Feb 27, 2013 at 11:56 AM, Eli Collins <el...@cloudera.com>
>>>>>> wrote:
>>>>>>> 
>>>>>>>> Bobby raises some good questions.  A related one, since most
>>>>>>>> current developers won't add Windows support for new features
>>>>>>>> that are platform specific is it assumed that Windows
>>>>>>>> development will either lag or will people actively work on
>>>>>>>> keeping Windows up with the latest?  And vice versa in case
>>>>>>>> Windows support is implemented
>>>>>> first.
>>>>>>>> 
>>>>>>>> Is there a jira for resolving the outstanding TODOs in the code
>>>>>>>> base (similar to HDFS-2148)?  Looks like this merge doesn't
>>>>>>>> introduce many which is great (just did a quick diff and grep).
>>>>>>>> 
>>>>>>>> Thanks,
>>>>>>>> Eli
>>>>>>>> 
>>>>>>>> On Wed, Feb 27, 2013 at 8:17 AM, Robert Evans
>>>>>>>> <ev...@yahoo-inc.com>
>>>>>>> wrote:
>>>>>>>>> After this is merged in is Windows still going to be a second
>>>>>>>>> class citizen but happens to work for more than just
>>>>>>>>> development or is it a fully supported platform where if
>>>>>>>>> something breaks it can block a
>>>>>>>> release?
>>>>>>>>> How do we as a community intend to keep Windows support from
>>>>>> breaking?
>>>>>>>>> We don't have any Jenkins slaves to be able to run nightly
>>>>>>>>> tests to validate everything still compiles/runs.  This is
>>>>>>>>> not a blocker for me because we often rely on individuals and
>>>>>>>>> groups to test Hadoop, but I
>>>>>>> do
>>>>>>>>> think we need to have this discussion before we put it in.
>>>>>>>>> 
>>>>>>>>> --Bobby
>>>>>>>>> 
>>>>>>>>> On 2/26/13 4:55 PM, "Suresh Srinivas"
>>>>>>>>> <su...@hortonworks.com>
>>>>>> wrote:
>>>>>>>>> 
>>>>>>>>>> I had posted heads up about merging branch-trunk-win to trunk
>>>>>>>>>> on Feb
>>>>>>> 8th.
>>>>>>>>>> I
>>>>>>>>>> am happy to announce that we are ready for the merge.
>>>>>>>>>> 
>>>>>>>>>> Here is a brief recap on the highlights of the work done:
>>>>>>>>>> - Command-line scripts for the Hadoop surface area
>>>>>>>>>> - Mapping the HDFS permissions model to Windows
>>>>>>>>>> - Abstracted and reconciled mismatches around differences in
>>>>>>>>>> Path semantics in Java and Windows
>>>>>>>>>> - Native Task Controller for Windows
>>>>>>>>>> - Implementation of a Block Placement Policy to support cloud
>>>>>>>>>> environments, more specifically Azure.
>>>>>>>>>> - Implementation of Hadoop native libraries for Windows
>>>>>>>>>> (compression codecs, native I/O)
>>>>>>>>>> - Several reliability issues, including race-conditions,
>>>>>>>>>> intermittent
>>>>>>>> test
>>>>>>>>>> failures, resource leaks.
>>>>>>>>>> - Several new unit test cases written for the above changes
>>>>>>>>>> 
>>>>>>>>>> Please find the details of the work in
>>>>>>>>>> CHANGES.branch-trunk-win.txt - Common
>>>>>>>>>> changes<http://bit.ly/Xe7Ynv>, HDFS changes<
>>>>>>> http://bit.ly/13QOSo9
>>>>>>>>> ,
>>>>>>>>>> and YARN and MapReduce changes <http://bit.ly/128zzMt>. This
>>>>>>>>>> is the
>>>>>>> work
>>>>>>>>>> ported from branch-1-win to a branch based on trunk.
>>>>>>>>>> 
>>>>>>>>>> For details of the testing done, please see the thread -
>>>>>>>>>> http://bit.ly/WpavJ4. Merge patch for this is available on
>>>>>>> HADOOP-8562<
>>>>>>>>>> https://issues.apache.org/jira/browse/HADOOP-8562>.
>>>>>>>>>> 
>>>>>>>>>> This was a large undertaking that involved developing code,
>>>>>>>>>> testing the entire Hadoop stack, including scale tests. This
>>>>>>>>>> is made possible only with the contribution from many many
>>>>>>>>>> folks in the community. Following
>>>>>>> people
>>>>>>>>>> contributed to this work: Ivan Mitic, Chuan Liu, Ramya Sunil,
>>>>>>>>>> Bikas
>>>>>>> Saha,
>>>>>>>>>> Kanna Karanam, John Gordon, Brandon Li, Chris Nauroth, David
>>>>>>>>>> Lao,
>>>>>>>> Sumadhur
>>>>>>>>>> Reddy Bolli, Arpit Agarwal, Ahmed El Baz, Mike Liddell, Jing
>>>>>>>>>> Zhao,
>>>>>>> Thejas
>>>>>>>>>> Nair, Steve Maine, Ganeshan Iyer, Raja Aluri, Giridharan
>>>>>>>>>> Kesavan, Ramya Bharathi Nimmagadda, Daryn Sharp, Arun Murthy,
>>>>>>>>>> Tsz-Wo Nicholas Sze,
>>>>>>>> Suresh
>>>>>>>>>> Srinivas and Sanjay Radia. There are many others who
>>>>>>>>>> contributed as
>>>>>>> well
>>>>>>>>>> providing feedback and comments on numerous jiras.
>>>>>>>>>> 
>>>>>>>>>> The vote will run for seven days and will end on March 5,
>>>>>>>>>> 6:00PM
>>>>>> PST.
>>>>>>>>>> 
>>>>>>>>>> Regards,
>>>>>>>>>> Suresh
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> On Thu, Feb 7, 2013 at 6:41 PM, Mahadevan Venkatraman
>>>>>>>>>> <ma...@microsoft.com>wrote:
>>>>>>>>>> 
>>>>>>>>>>> It is super exciting to look at the prospect of these
>>>>>>>>>>> changes being merged  to trunk. Having Windows as one of the
>>>>>>>>>>> supported Hadoop platforms is
>>>>>>> a
>>>>>>>>>>> fantastic opportunity both for the Hadoop project and
>>>>>>>>>>> Microsoft customers.
>>>>>>>>>>> 
>>>>>>>>>>> This work began around a year back when a few of us started
>>>>>>>>>>> with a
>>>>>>>> basic
>>>>>>>>>>> port of Hadoop on Windows. Ever since, the Hadoop team in
>>>>>>>>>>> Microsoft
>>>>>>>> have
>>>>>>>>>>> made significant progress in the following areas:
>>>>>>>>>>> (PS: Some of these items are already included in Suresh's
>>>>>>>>>>> email, but including again for completeness)
>>>>>>>>>>> 
>>>>>>>>>>> - Command-line scripts for the Hadoop surface area
>>>>>>>>>>> - Mapping the HDFS permissions model to Windows
>>>>>>>>>>> - Abstracted and reconciled mismatches around differences
>>>>>>>>>>> in Path semantics in Java and Windows
>>>>>>>>>>> - Native Task Controller for Windows
>>>>>>>>>>> - Implementation of a Block Placement Policy to support
>>>>>>>>>>> cloud environments, more specifically Azure.
>>>>>>>>>>> - Implementation of Hadoop native libraries for Windows
>>>>>>>>>>> (compression codecs, native I/O) - Several reliability
>>>>>>>>>>> issues, including race-conditions, intermittent test
>>>>>>>>>>> failures, resource
>>>>>> leaks.
>>>>>>>>>>> - Several new unit test cases written for the above changes
>>>>>>>>>>> 
>>>>>>>>>>> In the process, we have closely engaged with the Apache
>>>>>>>>>>> open source community and have got great support and
>>>>>>>>>>> assistance from the
>>>>>>> community
>>>>>>>>>>> in
>>>>>>>>>>> terms of contributing fixes, code review comments and
>>> commits.
>>>>>>>>>>> 
>>>>>>>>>>> In addition, the Hadoop team at Microsoft has also made
>>>>>>>>>>> good progress
>>>>>>>> in
>>>>>>>>>>> other projects including Hive, Pig, Sqoop, Oozie, HCat and
>>>>>> HBase.
>>>>>>> Many
>>>>>>>>>>> of
>>>>>>>>>>> these changes have already been committed to the respective
>>>>>>>>>>> trunks
>>>>>>> with
>>>>>>>>>>> help from various committers and contributors. It is great
>>>>>>>>>>> to see the commitment of the community to support multiple
>>>>>>>>>>> platforms, and we
>>>>>>> look
>>>>>>>>>>> forward to the day when a developer/customer is able to
>>>>>>>>>>> successfully deploy  a complete solution stack based on
>>>>>>>>>>> Apache Hadoop releases.
>>>>>>>>>>> 
>>>>>>>>>>> Next Steps:
>>>>>>>>>>> 
>>>>>>>>>>> All of the above changes are part of the Windows Azure
>>>>>>>>>>> HDInsight and  HDInsight Server products from Microsoft. We
>>>>>>>>>>> have successfully on-boarded  several internal customers and
>>>>>>>>>>> have been running production workloads
>>>>>>>> on
>>>>>>>>>>> Windows Azure HDInsight. Our vision is to create a big data
>>>>>>>>>>> platform based  on Hadoop, and we are committed to helping
>>>>>>>>>>> make Hadoop a world-class  solution that anyone can use to
>>>>>>>>>>> solve their biggest data challenges.
>>>>>>>>>>> 
>>>>>>>>>>> As an immediate next step, we would like to have a
>>>>>>>>>>> discussion around
>>>>>>>> how
>>>>>>>>>>> we can ensure that the quality of the mainline Hadoop
>>>>>>>>>>> branches on Windows  is maintained. To this end, we would
>>>>>>>>>>> like to get to the state where
>>>>>>> we
>>>>>>>>>>> have
>>>>>>>>>>> pre-checkin validation gates and nightly test runs enabled
>>>>>>>>>>> on
>>>>>>> Windows.
>>>>>>>>>>> If
>>>>>>>>>>> you have any suggestions around this, please do send an
>>> email.
>>>>>>>>>>> We
>>>>>>> are
>>>>>>>>>>> committed to helping sustain the long-term quality of
>>>>>>>>>>> Hadoop on both Linux  and Windows.
>>>>>>>>>>> 
>>>>>>>>>>> We sincerely thank the community for their contribution and
>>>>>>>>>>> support
>>>>>>> so
>>>>>>>>>>> far. And hope to continue having a close engagement in the
>>>>>> future.
>>>>>>>>>>> 
>>>>>>>>>>> -Microsoft HDInsight Team
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> -----Original Message-----
>>>>>>>>>>> From: Suresh Srinivas [mailto:suresh@hortonworks.com]
>>>>>>>>>>> Sent: Thursday, February 7, 2013 5:42 PM
>>>>>>>>>>> To: common-dev@hadoop.apache.org;
>>>>>>>>>>> yarn-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org;
>>>>>>>>>>> mapreduce-dev@hadoop.apache.org
>>>>>>>>>>> Subject: Heads up - merge branch-trunk-win to trunk
>>>>>>>>>>> 
>>>>>>>>>>> The support for Hadoop on Windows was proposed in
>>>>>>>>>>> HADOOP-8079<
>>>>>>>>>>> https://issues.apache.org/jira/browse/HADOOP-8079> almost a
>>>> year
>>>>>>> ago.
>>>>>>>>>>> The
>>>>>>>>>>> goal was to make Hadoop natively integrated, full-featured,
>>>>>>>>>>> and performance  and scalability tuned on Windows Server or
>>>>>>>>>>> Windows Azure.
>>>>>>>>>>> We are happy to announce that a lot of progress has been
>>>>>>>>>>> made in this  regard.
>>>>>>>>>>> 
>>>>>>>>>>> Initial work started in a feature branch, branch-1-win,
>>>>>>>>>>> based on branch-1.
>>>>>>>>>>> The details related to the work done in the branch can be
>>>>>>>>>>> seen in  CHANGES.txt<
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>> 
>>>> http://svn.apache.org/viewvc/hadoop/common/branches/branch-1-win/CHA
>>>>>>>> NGES
>>>>>>> .
>>>>>>>>>>> branch-1-win.txt?view=markup
>>>>>>>>>>>> .
>>>>>>>>>>> This work has been ported to a branch, branch-trunk-win,
>>>>>>>>>>> based on
>>>>>>>> trunk.
>>>>>>>>>>> Merge patch for this is available on
>>>>>>>>>>> HADOOP-8562<https://issues.apache.org/jira/browse/HADOOP-85
>>>>>>>>>>> 62>
>>>>>>>>>>> .
>>>>>>>>>>> 
>>>>>>>>>>> Highlights of the work done so far:
>>>>>>>>>>> 1. Necessary changes in Hadoop to run natively on Windows.
>>>>>>>>>>> These
>>>>>>>> changes
>>>>>>>>>>> handle differences in platforms related to path names,
>>>>>>>>>>> process/task  management etc.
>>>>>>>>>>> 2. Addition of winutils tools for managing file permissions
>>>>>>>>>>> and ownership,  user group mapping, hardlinks, symbolic
>>>>>>>>>>> links, chmod, disk
>>>>>>> utilization,
>>>>>>>>>>> and
>>>>>>>>>>> process/task management.
>>>>>>>>>>> 3. Added cmd scripts equivalent to existing shell scripts
>>>>>>>>>>> hadoop-daemon.sh, start and stop scripts.
>>>>>>>>>>> 4. Addition of block placement policy implemnation to
>>>>>>>>>>> support cloud  enviroment, more specifically Azure.
>>>>>>>>>>> 
>>>>>>>>>>> We are very close to wrapping up the work in
>>>>>>>>>>> branch-trunk-win and getting  ready for a merge. Currently
>>>>>>>>>>> the merge patch is passing close to 100%
>>>>>>>> of
>>>>>>>>>>> unit tests on Linux. Soon I will call for a vote to merge
>>>>>>>>>>> this branch into  trunk.
>>>>>>>>>>> 
>>>>>>>>>>> Next steps:
>>>>>>>>>>> 1. Call for vote to merge branch-trunk-win to trunk, when
>>>>>>>>>>> the work completes and precommit build is clean.
>>>>>>>>>>> 2. Start a discussion on adding Jenkins precommit builds on
>>>>>>>>>>> windows
>>>>>>> and
>>>>>>>>>>> how to integrate that with the existing commit process.
>>>>>>>>>>> 
>>>>>>>>>>> Let me know if you have any questions.
>>>>>>>>>>> 
>>>>>>>>>>> Regards,
>>>>>>>>>>> Suresh
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> --
>>>>>>>>>> http://hortonworks.com/download/
>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> --
>>>>>>> http://hortonworks.com/download/
>>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> --
>>>>>> Todd Lipcon
>>>>>> Software Engineer, Cloudera
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>> 
>>>> 
>>> 
>>> 
>> 

Fwd: [Vote] Merge branch-trunk-win to trunk

[Matt Foley <ma...@apache.org>]

+1 (binding)

Apache is supposed to be about the community.  We have here a community of
developers, who have actively and openly worked to add a major improvement
to Hadoop: the ability to work cross-platform.  Furthermore, the size of
the substantive part of the needed patch is only about 1500 lines, much
smaller than quite a few other additions to Hadoop over the last few
months.  We should welcome and support this change, and make sure that the
code stays cross-platform going forward by extending our CI practices,
especially pre-commit "test-patch", to also include Windows.

As most of you know, my colleague Giri Kesavan (PMC member) helps maintain
the Linux CI capability for Hadoop.  I've talked with him, and he and I are
committing to getting test-patch implemented for Windows, so that along
with the current automated "+1"s required to commit, we can add two more,
for javac build in Windows and core unit tests in Windows.

Members of the team implementing cross-platform compatibility, including
Microsoft employees, have opened the discussion for providing hardware or
VM resources to perform this additional CI testing.  I will assist them to
work with the Apache Infra team and figure out how to make it happen.

I understand there is some concern about the additional platform test.
 My going-in
presumption, based on Java's intrinsic, pretty-good, cross-platform
compatibility, is that patches to Hadoop will by default also have
cross-platform compatibility, unless they are written in an explicitly
platform-dependent way.  I also believe that in the vast majority of cases
the cross-platform compatibility of Java will carry thru to Hadoop patches,
without additional effort on the developer's part.

Let's try it, and see what happens.  If we actually find a frequent
difficulty, we'll change to engineer around it.  But I believe that, in the
rare cases where a Windows-specific failure occurs, there will be a number
of people (new, enthusiastic members of the community! :-) willing to help.
 If such help is not forthcoming, then we can discuss work-arounds, but
like a previous poster, I am confident in the community.

Regards,
--Matt



On Thu, Feb 28, 2013 at 12:21 PM, Chuan Liu <ch...@microsoft.com> wrote:

>  +1 (non-binding)
>>
>> As someone also contributed to porting Hadoop to Windows, I think Java
>> already provided a very good platform independent platform.
>> For features that are not available in Java, we will try to provide our
>> platform independent APIs that abstract OS tasks away.
>> Most features should have no difficulty running on Windows and Linux by
>> using Java and those platform independent APIs.
>>
>> For concerns raise on new features that may fail on Windows, I think we
>> don't need to require passing on Windows a mandate at the moment. We can
>> simply mark it unavailable to Windows and port it later if the feature is
>> important.
>>
>> -Chuan
>>
>> -----Original Message-----
>> From: Chris Nauroth [mailto:cnauroth@hortonworks.com]
>> Sent: Thursday, February 28, 2013 11:51 AM
>> To: hdfs-dev@hadoop.apache.org
>> Cc: mapreduce-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org;
>> common-dev@hadoop.apache.org
>> Subject: Re: [Vote] Merge branch-trunk-win to trunk
>>
>> > Is there a jira for resolving the outstanding TODOs in the code base
>> > (similar to HDFS-2148)?  Looks like this merge doesn't introduce many
>> > which is great (just did a quick diff and grep).
>>
>> I found 2 remaining TODOs introduced in the current merge patch.  One is
>> in ContainerLaunch.java.  The container launch script was trying to set a
>> CLASSPATH that exceeded the Windows maximum command line length.  The fix
>> was to wrap the long classpath into an intermediate jar containing only a
>> manifest file with a Class-Path entry.  (See YARN-316.)  Just to be
>> conservative, we wrapped this logic in an if (Shell.WINDOWS) guard and
>> marked a TODO to remove it later and use that approach on all platforms
>> after additional testing.  I've tested this code path successfully on Mac
>> too, but several people wanted additional testing and performance checks
>> before removing the if (Shell.WINDOWS) guard.  That work is tracked in an
>> existing jira: YARN-358.
>>
>> The other TODO is for winutils to print more usage information and
>> examples.  At this point, I think winutils is printing sufficient
>> information, and we can just remove the TODO.  I just submitted a new jira
>> to start that conversation: HADOOP-9348.
>>
>> Thank you,
>> --Chris
>>
>>
>> On Thu, Feb 28, 2013 at 11:29 AM, Robert Evans <ev...@yahoo-inc.com>
>> wrote:
>>
>> > My initial question was mostly intended to understand the desired new
>> > classification of Windows after the merge, and how we plan to maintain
>> > Windows support.  I am happy to hear that hardware for Jenkins will be
>> > provided.  I am also fine, at least initially, with us trying to treat
>> > Windows as a first class supported platform.  But I realize that there
>> > are a lot of people that do not have easy access to Windows for
>> > development/debugging, myself included. I also don't want to slow down
>> > the pace of development too much because of this.  It will cause some
>> > organizations that do not use or support Windows to be more likely to
>> > run software that has diverged from an official release.  It also has
>> > the potential to make the patch submission process even more
>> > difficult, which increases the likelihood of submitters abandoning
>> > patches.  However, the great thing about being in a community is we can
>> change if we need to.
>> >
>> > I am +0 for the merge.  I am not a Windows expert so I don't feel
>> > comfortable giving it a true +1.
>> >
>> > --Bobby
>> >
>> >
>> > On 2/28/13 10:45 AM, "Chris Nauroth" <cn...@hortonworks.com> wrote:
>> >
>> > >I'd like to share a few anecdotes about developing cross-platform,
>> > >hopefully to address some of the concerns about adding overhead to
>> > >the development process.  By reviewing past cases of cross-platform
>> Linux vs.
>> > >Windows bugs, we can get a sense for how the development process
>> > >could look in the future.
>> > >
>> > >HADOOP-9131: TestLocalFileSystem#testListStatusWithColons cannot run
>> > >on Windows.  As part of an earlier jira, HADOOP-8962, there was a new
>> > >test committed on trunk covering the case of a local file system
>> > >interaction on a file containing a ':'.  On Windows, ':' in a path
>> > >has special meaning as part of the drive specifier (i.e. C:), so this
>> > >test cannot pass when running on Windows.  In this kind of case, the
>> > >cross-platform bug is obvious, and the fix is obvious
>> > >(assumeTrue(!Shell.WINDOWS)).  Ideally, this would get fixed
>> > >pre-commit after seeing a -1 from the Windows Jenkins slave.
>> > >
>> > >HDFS-4274: BlockPoolSliceScanner does not close verification log
>> > >during shutdown.  This caused problems for MiniDFSCluster-based tests
>> > >running on Windows.  Failure to close the verification log meant that
>> > >we didn't release file locks, so the tests couldn't delete/recreate
>> > >working directories during teardown/setup.  Arguably, this was always
>> > >a bug, and running on Windows just exposed it because of its stricter
>> > >rules about file locking.  This is a more complex fix, but it doesn't
>> > >require platform-specific knowledge.  If some future patch
>> > >accidentally regresses this, then we'll likely see +1 from Linux
>> > >Jenkins and -1 from Windows Jenkins.  Ideally, it would get fixed
>> > >pre-commit, because it doesn't require Windows-specific knowledge.
>> > >There is also the matter of impact.
>> > > Re-breaking this would re-break many test suites on Windows.
>> > >
>> > >HADOOP-9232: JniBasedUnixGroupsMappingWithFallback fails on Windows
>> > >with UnsatisfiedLinkError.  This was introduced by HADOOP-8712, which
>> > >switched to JniBasedUnixGroupsMappingWithFallback as the default
>> > >hadoop.security.group.mapping, but did not provide a Windows
>> > >implementation of the JNI function.  In this case, there was a strong
>> > >desire to get
>> > >HADOOP-8712 into a release, fixing it on Windows required native
>> > >Windows API knowledge, and Windows users had a simple workaround
>> > >available by changing their configs back to
>> > >ShellBasedUnixGroupsMapping.  I think this is the kind of situation
>> > >where we could allow HADOOP-8712 to commit despite
>> > >-1 from Windows Jenkins, with fairly quick follow-up from an engineer
>> > >with the Windows expertise to fix it.
>> > >
>> > >To summarize, I don't think it needs to differ greatly from our
>> > >current development process.  We're all responsible for breadth of
>> > >understanding and maintenance of the whole codebase, but we also rely
>> > >on specific individuals with deep expertise in particular areas for
>> certain issues.
>> > > Sometimes we commit despite a -1 from Jenkins, based on the
>> > >community's judgment.
>> > >
>> > >Virtualization greatly simplifies cross-platform development.  I use
>> > >VirtualBox on a Mac host and run VMs for Windows and Ubuntu with a
>> > >shared drive so that they can all see the same copy of the source
>> > >code.  There are plenty of variations on this depending on your
>> > >preference, such as offloading the VMs to a separate server or cloud
>> > >service to free up local RAM.  I'm planning on submitting
>> > >BUILDING.txt changes later today that fully describe how to build on
>> > >Windows.  After some initial setup, it's nearly identical to the mvn
>> > >commands that you already use today.
>> > >
>> > >Hope this helps,
>> > >--Chris
>> > >
>> > >
>> > >On Thu, Feb 28, 2013 at 3:25 AM, John Gordon
>> > ><Jo...@microsoft.com>wrote:
>> > >
>> > >> +1 (non-binding)
>> > >>
>> > >> I want to share my vote of confidence in this community.  If
>> > >>motivated to  do so, this community can keep this project
>> > >>cross-platform and continue to  rapidly innovate without breaking a
>> > >>sweat.
>> > >>
>> > >> The day we started working on this, I saw the foundations of
>> > >>greatness in  the quality and volume of dev tests, the code itself,
>> > >>and the Apache values  themselves.
>> > >>
>> > >> 1.) Hadoop's unit tests and their frameworks are very well thought
>> > >>out and  the consideration and energy that went into their design is
>> > >>worthy of  praise.  The MiniCluster abstractions utilize very few
>> > >>resources and put  all the processes into one JVM for easy
>> > >>debugging.  It is very easy to  select specific tests from the full
>> > >>suite to reproduce an issue reported in  another environment - like
>> > >>the Jenkins build server or another  contributor's environment.
>> > >> 2.) This community has done an excellent job of incorporating
>> > >>well-placed  log messages to make it easy to post mortem
>> > >>troubleshoot most failures.
>> > >>  The logs are very useful, and it is extremely rare that
>> > >>troubleshooting a  failure requires debugging a live repro.
>> > >> 3.) Hadoop is written primarily in Java, a cross-platform language
>> > >>that  provides its own platform in the form of the JVM to insulate
>> > >>most of the  code from the specifics of the OS layer.
>> > >> 4.) CoPDoC - The right priorities, and well stated.
>> > >>
>> > >>
>> > >> Thank you,
>> > >>
>> > >> John
>> > >>
>> > >> -----Original Message-----
>> > >> From: Ivan Mitic [mailto:ivanmi@microsoft.com]
>> > >> Sent: Wednesday, February 27, 2013 6:32 PM
>> > >> To: mapreduce-dev@hadoop.apache.org; common-dev@hadoop.apache.org
>> > >> Cc: yarn-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org
>> > >> Subject: RE: [Vote] Merge branch-trunk-win to trunk
>> > >>
>> > >> +1 (non-binding)
>> > >>
>> > >> I am really glad to see this happening! As people already
>> > >>mentioned, this  has been a great engineering effort involving many
>> > >>people!
>> > >>
>> > >>
>> > >> Folks raised some valid concerns below and I thought it would be
>> > >>good to  share my 2 cents. In my opinion, we don't have to solve all
>> > >>these problems  right now. As we move forward with two platforms, we
>> > >>can start addressing  one problem at a time and incrementally
>> > >>improve. In the first iteration,  maintaining Hadoop on Windows
>> > >>could be just everyone trying to do their  best effort (make sure
>> > >>Jenkins build succeeds at least). We already have  people who are
>> > >>building/running trunk on Windows daily, so they would jump  in and
>> > >>fix problems as needed (we've been doing this in branch-trunk-win
>> > >>for a while now). Although I see that the problems could arise with
>> > >>platform specific features/optimizations, I don't think these are
>> > >>frequent,  so in most cases everything will just work. Merging the
>> > >>two branches sooner  rather than later does seems like the right
>> > >>thing to do if the ultimate  goal is to have Hadoop on both
>> > >>platforms. Now that the port has completed,  we will have people in
>> > >>Microsoft (and elsewhere) wanting to contribute
>> > >>features/improvements to the trunk branch. A separate branch would
>> > >>just  make things more difficult and confusing for everyone :) Hope
>> > >>this makes  sense.
>> > >>
>> > >> -----Original Message-----
>> > >> From: Todd Lipcon [mailto:todd@cloudera.com]
>> > >> Sent: Wednesday, February 27, 2013 3:43 PM
>> > >> To: common-dev@hadoop.apache.org
>> > >> Cc: yarn-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org;
>> > >> mapreduce-dev@hadoop.apache.org
>> > >> Subject: Re: [Vote] Merge branch-trunk-win to trunk
>> > >>
>> > >> On Wed, Feb 27, 2013 at 2:54 PM, Suresh Srinivas <
>> > suresh@hortonworks.com
>> > >> >wrote:
>> > >>
>> > >> > With that we need to decide how our precommit process looks.
>> > >> > My inclination is to wait for +1 from precommit builds on both
>> > >> > the platforms to ensure no issues are introduced.
>> > >> > Thoughts?
>> > >> >
>> > >> > 2. Feature development impact
>> > >> > Some questions have been raised about would new features need to
>> > >> > be supported on both the platforms. Yes. I do not see a reason
>> > >> > why features cannot work on both the platforms, with the
>> > >> > exception of platform specific optimizations. This what Java gives
>> us.
>> > >> >
>> > >> >
>> > >> I'm concerned about the above. Personally, I don't have access to
>> > >>any  Windows boxes with development tools, and I know nothing about
>> > >>developing  on Windows. The only Windows I run is an 8GB VM with 1
>> > >>GB RAM allocated,  for powerpoint :)
>> > >>
>> > >> If I submit a patch and it gets -1 "tests failed" on the Windows
>> > >> slave, how am I supposed to proceed?
>> > >>
>> > >> I think a reasonable compromise would be that the tests should
>> > >>always
>> > >> *build* on Windows before commit, and contributors should do their
>> > >>best to  look at the test logs for any Windows-specific failures.
>> > >>But, beyond  looking at the logs, a "-1 Tests failed on windows"
>> > >>should not block a  commit.
>> > >>
>> > >> Those contributors who are interested in Windows being a
>> > >> first-class platform should be responsible for watching the Windows
>> > >> builds and debugging/fixing any regressions that might be
>> Windows-specific.
>> > >>
>> > >> I also think the KDE model that Harsh pointed out is an interesting
>> > >>one
>> > >>--
>> > >> ie the idea that we would not merge windows support to trunk, but
>> > >>rather  treat is as a "parallel code line" which lives in the ASF
>> > >>and has its own  builds and releases. The windows team would
>> > >>periodically merge
>> > >>trunk->win
>> > >> to pick up any new changes, and do a separate test/release process.
>> > >>I'm not  convinced this is the best idea, but worth discussion of
>> > >>pros and cons.
>> > >>
>> > >> -Todd
>> > >>
>> > >>
>> > >> >
>> > >> > On Wed, Feb 27, 2013 at 11:56 AM, Eli Collins <el...@cloudera.com>
>> > >>wrote:
>> > >> >
>> > >> > > Bobby raises some good questions.  A related one, since most
>> > >> > > current developers won't add Windows support for new features
>> > >> > > that are platform specific is it assumed that Windows
>> > >> > > development will either lag or will people actively work on
>> > >> > > keeping Windows up with the latest?  And vice versa in case
>> > >> > > Windows support is implemented
>> > >>first.
>> > >> > >
>> > >> > > Is there a jira for resolving the outstanding TODOs in the code
>> > >> > > base (similar to HDFS-2148)?  Looks like this merge doesn't
>> > >> > > introduce many which is great (just did a quick diff and grep).
>> > >> > >
>> > >> > > Thanks,
>> > >> > > Eli
>> > >> > >
>> > >> > > On Wed, Feb 27, 2013 at 8:17 AM, Robert Evans
>> > >> > > <ev...@yahoo-inc.com>
>> > >> > wrote:
>> > >> > > > After this is merged in is Windows still going to be a second
>> > >> > > > class citizen but happens to work for more than just
>> > >> > > > development or is it a fully supported platform where if
>> > >> > > > something breaks it can block a
>> > >> > > release?
>> > >> > > >  How do we as a community intend to keep Windows support from
>> > >> breaking?
>> > >> > > > We don't have any Jenkins slaves to be able to run nightly
>> > >> > > > tests to validate everything still compiles/runs.  This is
>> > >> > > > not a blocker for me because we often rely on individuals and
>> > >> > > > groups to test Hadoop, but I
>> > >> > do
>> > >> > > > think we need to have this discussion before we put it in.
>> > >> > > >
>> > >> > > > --Bobby
>> > >> > > >
>> > >> > > > On 2/26/13 4:55 PM, "Suresh Srinivas"
>> > >> > > > <su...@hortonworks.com>
>> > >> wrote:
>> > >> > > >
>> > >> > > >>I had posted heads up about merging branch-trunk-win to trunk
>> > >> > > >>on Feb
>> > >> > 8th.
>> > >> > > >>I
>> > >> > > >>am happy to announce that we are ready for the merge.
>> > >> > > >>
>> > >> > > >>Here is a brief recap on the highlights of the work done:
>> > >> > > >>- Command-line scripts for the Hadoop surface area
>> > >> > > >>- Mapping the HDFS permissions model to Windows
>> > >> > > >>- Abstracted and reconciled mismatches around differences in
>> > >> > > >>Path semantics in Java and Windows
>> > >> > > >>- Native Task Controller for Windows
>> > >> > > >>- Implementation of a Block Placement Policy to support cloud
>> > >> > > >>environments, more specifically Azure.
>> > >> > > >>- Implementation of Hadoop native libraries for Windows
>> > >> > > >>(compression codecs, native I/O)
>> > >> > > >>- Several reliability issues, including race-conditions,
>> > >> > > >>intermittent
>> > >> > > test
>> > >> > > >>failures, resource leaks.
>> > >> > > >>- Several new unit test cases written for the above changes
>> > >> > > >>
>> > >> > > >>Please find the details of the work in
>> > >> > > >>CHANGES.branch-trunk-win.txt - Common
>> > >> > > >>changes<http://bit.ly/Xe7Ynv>, HDFS changes<
>> > >> > http://bit.ly/13QOSo9
>> > >> > > >,
>> > >> > > >>and YARN and MapReduce changes <http://bit.ly/128zzMt>. This
>> > >> > > >>is the
>> > >> > work
>> > >> > > >>ported from branch-1-win to a branch based on trunk.
>> > >> > > >>
>> > >> > > >>For details of the testing done, please see the thread -
>> > >> > > >>http://bit.ly/WpavJ4. Merge patch for this is available on
>> > >> > HADOOP-8562<
>> > >> > > >>https://issues.apache.org/jira/browse/HADOOP-8562>.
>> > >> > > >>
>> > >> > > >>This was a large undertaking that involved developing code,
>> > >> > > >>testing the entire Hadoop stack, including scale tests. This
>> > >> > > >>is made possible only with the contribution from many many
>> > >> > > >>folks in the community. Following
>> > >> > people
>> > >> > > >>contributed to this work: Ivan Mitic, Chuan Liu, Ramya Sunil,
>> > >> > > >>Bikas
>> > >> > Saha,
>> > >> > > >>Kanna Karanam, John Gordon, Brandon Li, Chris Nauroth, David
>> > >> > > >>Lao,
>> > >> > > Sumadhur
>> > >> > > >>Reddy Bolli, Arpit Agarwal, Ahmed El Baz, Mike Liddell, Jing
>> > >> > > >>Zhao,
>> > >> > Thejas
>> > >> > > >>Nair, Steve Maine, Ganeshan Iyer, Raja Aluri, Giridharan
>> > >> > > >>Kesavan, Ramya Bharathi Nimmagadda, Daryn Sharp, Arun Murthy,
>> > >> > > >>Tsz-Wo Nicholas Sze,
>> > >> > > Suresh
>> > >> > > >>Srinivas and Sanjay Radia. There are many others who
>> > >> > > >>contributed as
>> > >> > well
>> > >> > > >>providing feedback and comments on numerous jiras.
>> > >> > > >>
>> > >> > > >>The vote will run for seven days and will end on March 5,
>> > >> > > >>6:00PM
>> > >>PST.
>> > >> > > >>
>> > >> > > >>Regards,
>> > >> > > >>Suresh
>> > >> > > >>
>> > >> > > >>
>> > >> > > >>
>> > >> > > >>
>> > >> > > >>On Thu, Feb 7, 2013 at 6:41 PM, Mahadevan Venkatraman
>> > >> > > >><ma...@microsoft.com>wrote:
>> > >> > > >>
>> > >> > > >>> It is super exciting to look at the prospect of these
>> > >> > > >>>changes being merged  to trunk. Having Windows as one of the
>> > >> > > >>>supported Hadoop platforms is
>> > >> > a
>> > >> > > >>> fantastic opportunity both for the Hadoop project and
>> > >> > > >>>Microsoft customers.
>> > >> > > >>>
>> > >> > > >>> This work began around a year back when a few of us started
>> > >> > > >>> with a
>> > >> > > basic
>> > >> > > >>> port of Hadoop on Windows. Ever since, the Hadoop team in
>> > >> > > >>> Microsoft
>> > >> > > have
>> > >> > > >>> made significant progress in the following areas:
>> > >> > > >>> (PS: Some of these items are already included in Suresh's
>> > >> > > >>> email, but including again for completeness)
>> > >> > > >>>
>> > >> > > >>> - Command-line scripts for the Hadoop surface area
>> > >> > > >>> - Mapping the HDFS permissions model to Windows
>> > >> > > >>> - Abstracted and reconciled mismatches around differences
>> > >> > > >>> in Path semantics in Java and Windows
>> > >> > > >>> - Native Task Controller for Windows
>> > >> > > >>> - Implementation of a Block Placement Policy to support
>> > >> > > >>> cloud environments, more specifically Azure.
>> > >> > > >>> - Implementation of Hadoop native libraries for Windows
>> > >> > > >>> (compression codecs, native I/O) - Several reliability
>> > >> > > >>> issues, including race-conditions, intermittent test
>> > >> > > >>> failures, resource
>> > >> leaks.
>> > >> > > >>> - Several new unit test cases written for the above changes
>> > >> > > >>>
>> > >> > > >>> In the process, we have closely engaged with the Apache
>> > >> > > >>> open source community and have got great support and
>> > >> > > >>> assistance from the
>> > >> > community
>> > >> > > >>>in
>> > >> > > >>> terms of contributing fixes, code review comments and
>> commits.
>> > >> > > >>>
>> > >> > > >>> In addition, the Hadoop team at Microsoft has also made
>> > >> > > >>> good progress
>> > >> > > in
>> > >> > > >>> other projects including Hive, Pig, Sqoop, Oozie, HCat and
>> > >>HBase.
>> > >> > Many
>> > >> > > >>>of
>> > >> > > >>> these changes have already been committed to the respective
>> > >> > > >>>trunks
>> > >> > with
>> > >> > > >>> help from various committers and contributors. It is great
>> > >> > > >>> to see the commitment of the community to support multiple
>> > >> > > >>> platforms, and we
>> > >> > look
>> > >> > > >>> forward to the day when a developer/customer is able to
>> > >> > > >>>successfully deploy  a complete solution stack based on
>> > >> > > >>>Apache Hadoop releases.
>> > >> > > >>>
>> > >> > > >>> Next Steps:
>> > >> > > >>>
>> > >> > > >>> All of the above changes are part of the Windows Azure
>> > >> > > >>>HDInsight and  HDInsight Server products from Microsoft. We
>> > >> > > >>>have successfully on-boarded  several internal customers and
>> > >> > > >>>have been running production workloads
>> > >> > > on
>> > >> > > >>> Windows Azure HDInsight. Our vision is to create a big data
>> > >> > > >>>platform based  on Hadoop, and we are committed to helping
>> > >> > > >>>make Hadoop a world-class  solution that anyone can use to
>> > >> > > >>>solve their biggest data challenges.
>> > >> > > >>>
>> > >> > > >>> As an immediate next step, we would like to have a
>> > >> > > >>> discussion around
>> > >> > > how
>> > >> > > >>> we can ensure that the quality of the mainline Hadoop
>> > >> > > >>>branches on Windows  is maintained. To this end, we would
>> > >> > > >>>like to get to the state where
>> > >> > we
>> > >> > > >>>have
>> > >> > > >>> pre-checkin validation gates and nightly test runs enabled
>> > >> > > >>>on
>> > >> > Windows.
>> > >> > > >>>If
>> > >> > > >>> you have any suggestions around this, please do send an
>> email.
>> > >> > > >>>We
>> > >> > are
>> > >> > > >>> committed to helping sustain the long-term quality of
>> > >> > > >>>Hadoop on both Linux  and Windows.
>> > >> > > >>>
>> > >> > > >>> We sincerely thank the community for their contribution and
>> > >> > > >>> support
>> > >> > so
>> > >> > > >>> far. And hope to continue having a close engagement in the
>> > >>future.
>> > >> > > >>>
>> > >> > > >>> -Microsoft HDInsight Team
>> > >> > > >>>
>> > >> > > >>>
>> > >> > > >>> -----Original Message-----
>> > >> > > >>> From: Suresh Srinivas [mailto:suresh@hortonworks.com]
>> > >> > > >>> Sent: Thursday, February 7, 2013 5:42 PM
>> > >> > > >>> To: common-dev@hadoop.apache.org;
>> > >> > > >>> yarn-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org;
>> > >> > > >>> mapreduce-dev@hadoop.apache.org
>> > >> > > >>> Subject: Heads up - merge branch-trunk-win to trunk
>> > >> > > >>>
>> > >> > > >>> The support for Hadoop on Windows was proposed in
>> > >> > > >>> HADOOP-8079<
>> > >> > > >>> https://issues.apache.org/jira/browse/HADOOP-8079> almost a
>> > year
>> > >> > ago.
>> > >> > > >>>The
>> > >> > > >>> goal was to make Hadoop natively integrated, full-featured,
>> > >> > > >>>and performance  and scalability tuned on Windows Server or
>> > >> > > >>>Windows Azure.
>> > >> > > >>> We are happy to announce that a lot of progress has been
>> > >> > > >>>made in this  regard.
>> > >> > > >>>
>> > >> > > >>> Initial work started in a feature branch, branch-1-win,
>> > >> > > >>>based on branch-1.
>> > >> > > >>> The details related to the work done in the branch can be
>> > >> > > >>>seen in  CHANGES.txt<
>> > >> > > >>>
>> > >> > > >>>
>> > >> > >
>> > http://svn.apache.org/viewvc/hadoop/common/branches/branch-1-win/CHA
>> > >> > > NGES
>> > >> > .
>> > >> > > >>>branch-1-win.txt?view=markup
>> > >> > > >>> >.
>> > >> > > >>> This work has been ported to a branch, branch-trunk-win,
>> > >> > > >>> based on
>> > >> > > trunk.
>> > >> > > >>> Merge patch for this is available on
>> > >> > > >>> HADOOP-8562<https://issues.apache.org/jira/browse/HADOOP-85
>> > >> > > >>> 62>
>> > >> > > >>> .
>> > >> > > >>>
>> > >> > > >>> Highlights of the work done so far:
>> > >> > > >>> 1. Necessary changes in Hadoop to run natively on Windows.
>> > >> > > >>> These
>> > >> > > changes
>> > >> > > >>> handle differences in platforms related to path names,
>> > >> > > >>>process/task  management etc.
>> > >> > > >>> 2. Addition of winutils tools for managing file permissions
>> > >> > > >>>and ownership,  user group mapping, hardlinks, symbolic
>> > >> > > >>>links, chmod, disk
>> > >> > utilization,
>> > >> > > >>>and
>> > >> > > >>> process/task management.
>> > >> > > >>> 3. Added cmd scripts equivalent to existing shell scripts
>> > >> > > >>>hadoop-daemon.sh, start and stop scripts.
>> > >> > > >>> 4. Addition of block placement policy implemnation to
>> > >> > > >>>support cloud  enviroment, more specifically Azure.
>> > >> > > >>>
>> > >> > > >>> We are very close to wrapping up the work in
>> > >> > > >>>branch-trunk-win and getting  ready for a merge. Currently
>> > >> > > >>>the merge patch is passing close to 100%
>> > >> > > of
>> > >> > > >>> unit tests on Linux. Soon I will call for a vote to merge
>> > >> > > >>>this branch into  trunk.
>> > >> > > >>>
>> > >> > > >>> Next steps:
>> > >> > > >>> 1. Call for vote to merge branch-trunk-win to trunk, when
>> > >> > > >>> the work completes and precommit build is clean.
>> > >> > > >>> 2. Start a discussion on adding Jenkins precommit builds on
>> > >> > > >>> windows
>> > >> > and
>> > >> > > >>> how to integrate that with the existing commit process.
>> > >> > > >>>
>> > >> > > >>> Let me know if you have any questions.
>> > >> > > >>>
>> > >> > > >>> Regards,
>> > >> > > >>> Suresh
>> > >> > > >>>
>> > >> > > >>>
>> > >> > > >>
>> > >> > > >>
>> > >> > > >>--
>> > >> > > >>http://hortonworks.com/download/
>> > >> > > >
>> > >> > >
>> > >> >
>> > >> >
>> > >> >
>> > >> > --
>> > >> > http://hortonworks.com/download/
>> > >> >
>> > >>
>> > >>
>> > >>
>> > >> --
>> > >> Todd Lipcon
>> > >> Software Engineer, Cloudera
>> > >>
>> > >>
>> > >>
>> > >>
>> > >>
>> >
>> >
>>
>>
>

Fwd: [Vote] Merge branch-trunk-win to trunk

[Matt Foley <ma...@apache.org>]

+1 (binding)

Apache is supposed to be about the community.  We have here a community of
developers, who have actively and openly worked to add a major improvement
to Hadoop: the ability to work cross-platform.  Furthermore, the size of
the substantive part of the needed patch is only about 1500 lines, much
smaller than quite a few other additions to Hadoop over the last few
months.  We should welcome and support this change, and make sure that the
code stays cross-platform going forward by extending our CI practices,
especially pre-commit "test-patch", to also include Windows.

As most of you know, my colleague Giri Kesavan (PMC member) helps maintain
the Linux CI capability for Hadoop.  I've talked with him, and he and I are
committing to getting test-patch implemented for Windows, so that along
with the current automated "+1"s required to commit, we can add two more,
for javac build in Windows and core unit tests in Windows.

Members of the team implementing cross-platform compatibility, including
Microsoft employees, have opened the discussion for providing hardware or
VM resources to perform this additional CI testing.  I will assist them to
work with the Apache Infra team and figure out how to make it happen.

I understand there is some concern about the additional platform test.
 My going-in
presumption, based on Java's intrinsic, pretty-good, cross-platform
compatibility, is that patches to Hadoop will by default also have
cross-platform compatibility, unless they are written in an explicitly
platform-dependent way.  I also believe that in the vast majority of cases
the cross-platform compatibility of Java will carry thru to Hadoop patches,
without additional effort on the developer's part.

Let's try it, and see what happens.  If we actually find a frequent
difficulty, we'll change to engineer around it.  But I believe that, in the
rare cases where a Windows-specific failure occurs, there will be a number
of people (new, enthusiastic members of the community! :-) willing to help.
 If such help is not forthcoming, then we can discuss work-arounds, but
like a previous poster, I am confident in the community.

Regards,
--Matt



On Thu, Feb 28, 2013 at 12:21 PM, Chuan Liu <ch...@microsoft.com> wrote:

>  +1 (non-binding)
>>
>> As someone also contributed to porting Hadoop to Windows, I think Java
>> already provided a very good platform independent platform.
>> For features that are not available in Java, we will try to provide our
>> platform independent APIs that abstract OS tasks away.
>> Most features should have no difficulty running on Windows and Linux by
>> using Java and those platform independent APIs.
>>
>> For concerns raise on new features that may fail on Windows, I think we
>> don't need to require passing on Windows a mandate at the moment. We can
>> simply mark it unavailable to Windows and port it later if the feature is
>> important.
>>
>> -Chuan
>>
>> -----Original Message-----
>> From: Chris Nauroth [mailto:cnauroth@hortonworks.com]
>> Sent: Thursday, February 28, 2013 11:51 AM
>> To: hdfs-dev@hadoop.apache.org
>> Cc: mapreduce-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org;
>> common-dev@hadoop.apache.org
>> Subject: Re: [Vote] Merge branch-trunk-win to trunk
>>
>> > Is there a jira for resolving the outstanding TODOs in the code base
>> > (similar to HDFS-2148)?  Looks like this merge doesn't introduce many
>> > which is great (just did a quick diff and grep).
>>
>> I found 2 remaining TODOs introduced in the current merge patch.  One is
>> in ContainerLaunch.java.  The container launch script was trying to set a
>> CLASSPATH that exceeded the Windows maximum command line length.  The fix
>> was to wrap the long classpath into an intermediate jar containing only a
>> manifest file with a Class-Path entry.  (See YARN-316.)  Just to be
>> conservative, we wrapped this logic in an if (Shell.WINDOWS) guard and
>> marked a TODO to remove it later and use that approach on all platforms
>> after additional testing.  I've tested this code path successfully on Mac
>> too, but several people wanted additional testing and performance checks
>> before removing the if (Shell.WINDOWS) guard.  That work is tracked in an
>> existing jira: YARN-358.
>>
>> The other TODO is for winutils to print more usage information and
>> examples.  At this point, I think winutils is printing sufficient
>> information, and we can just remove the TODO.  I just submitted a new jira
>> to start that conversation: HADOOP-9348.
>>
>> Thank you,
>> --Chris
>>
>>
>> On Thu, Feb 28, 2013 at 11:29 AM, Robert Evans <ev...@yahoo-inc.com>
>> wrote:
>>
>> > My initial question was mostly intended to understand the desired new
>> > classification of Windows after the merge, and how we plan to maintain
>> > Windows support.  I am happy to hear that hardware for Jenkins will be
>> > provided.  I am also fine, at least initially, with us trying to treat
>> > Windows as a first class supported platform.  But I realize that there
>> > are a lot of people that do not have easy access to Windows for
>> > development/debugging, myself included. I also don't want to slow down
>> > the pace of development too much because of this.  It will cause some
>> > organizations that do not use or support Windows to be more likely to
>> > run software that has diverged from an official release.  It also has
>> > the potential to make the patch submission process even more
>> > difficult, which increases the likelihood of submitters abandoning
>> > patches.  However, the great thing about being in a community is we can
>> change if we need to.
>> >
>> > I am +0 for the merge.  I am not a Windows expert so I don't feel
>> > comfortable giving it a true +1.
>> >
>> > --Bobby
>> >
>> >
>> > On 2/28/13 10:45 AM, "Chris Nauroth" <cn...@hortonworks.com> wrote:
>> >
>> > >I'd like to share a few anecdotes about developing cross-platform,
>> > >hopefully to address some of the concerns about adding overhead to
>> > >the development process.  By reviewing past cases of cross-platform
>> Linux vs.
>> > >Windows bugs, we can get a sense for how the development process
>> > >could look in the future.
>> > >
>> > >HADOOP-9131: TestLocalFileSystem#testListStatusWithColons cannot run
>> > >on Windows.  As part of an earlier jira, HADOOP-8962, there was a new
>> > >test committed on trunk covering the case of a local file system
>> > >interaction on a file containing a ':'.  On Windows, ':' in a path
>> > >has special meaning as part of the drive specifier (i.e. C:), so this
>> > >test cannot pass when running on Windows.  In this kind of case, the
>> > >cross-platform bug is obvious, and the fix is obvious
>> > >(assumeTrue(!Shell.WINDOWS)).  Ideally, this would get fixed
>> > >pre-commit after seeing a -1 from the Windows Jenkins slave.
>> > >
>> > >HDFS-4274: BlockPoolSliceScanner does not close verification log
>> > >during shutdown.  This caused problems for MiniDFSCluster-based tests
>> > >running on Windows.  Failure to close the verification log meant that
>> > >we didn't release file locks, so the tests couldn't delete/recreate
>> > >working directories during teardown/setup.  Arguably, this was always
>> > >a bug, and running on Windows just exposed it because of its stricter
>> > >rules about file locking.  This is a more complex fix, but it doesn't
>> > >require platform-specific knowledge.  If some future patch
>> > >accidentally regresses this, then we'll likely see +1 from Linux
>> > >Jenkins and -1 from Windows Jenkins.  Ideally, it would get fixed
>> > >pre-commit, because it doesn't require Windows-specific knowledge.
>> > >There is also the matter of impact.
>> > > Re-breaking this would re-break many test suites on Windows.
>> > >
>> > >HADOOP-9232: JniBasedUnixGroupsMappingWithFallback fails on Windows
>> > >with UnsatisfiedLinkError.  This was introduced by HADOOP-8712, which
>> > >switched to JniBasedUnixGroupsMappingWithFallback as the default
>> > >hadoop.security.group.mapping, but did not provide a Windows
>> > >implementation of the JNI function.  In this case, there was a strong
>> > >desire to get
>> > >HADOOP-8712 into a release, fixing it on Windows required native
>> > >Windows API knowledge, and Windows users had a simple workaround
>> > >available by changing their configs back to
>> > >ShellBasedUnixGroupsMapping.  I think this is the kind of situation
>> > >where we could allow HADOOP-8712 to commit despite
>> > >-1 from Windows Jenkins, with fairly quick follow-up from an engineer
>> > >with the Windows expertise to fix it.
>> > >
>> > >To summarize, I don't think it needs to differ greatly from our
>> > >current development process.  We're all responsible for breadth of
>> > >understanding and maintenance of the whole codebase, but we also rely
>> > >on specific individuals with deep expertise in particular areas for
>> certain issues.
>> > > Sometimes we commit despite a -1 from Jenkins, based on the
>> > >community's judgment.
>> > >
>> > >Virtualization greatly simplifies cross-platform development.  I use
>> > >VirtualBox on a Mac host and run VMs for Windows and Ubuntu with a
>> > >shared drive so that they can all see the same copy of the source
>> > >code.  There are plenty of variations on this depending on your
>> > >preference, such as offloading the VMs to a separate server or cloud
>> > >service to free up local RAM.  I'm planning on submitting
>> > >BUILDING.txt changes later today that fully describe how to build on
>> > >Windows.  After some initial setup, it's nearly identical to the mvn
>> > >commands that you already use today.
>> > >
>> > >Hope this helps,
>> > >--Chris
>> > >
>> > >
>> > >On Thu, Feb 28, 2013 at 3:25 AM, John Gordon
>> > ><Jo...@microsoft.com>wrote:
>> > >
>> > >> +1 (non-binding)
>> > >>
>> > >> I want to share my vote of confidence in this community.  If
>> > >>motivated to  do so, this community can keep this project
>> > >>cross-platform and continue to  rapidly innovate without breaking a
>> > >>sweat.
>> > >>
>> > >> The day we started working on this, I saw the foundations of
>> > >>greatness in  the quality and volume of dev tests, the code itself,
>> > >>and the Apache values  themselves.
>> > >>
>> > >> 1.) Hadoop's unit tests and their frameworks are very well thought
>> > >>out and  the consideration and energy that went into their design is
>> > >>worthy of  praise.  The MiniCluster abstractions utilize very few
>> > >>resources and put  all the processes into one JVM for easy
>> > >>debugging.  It is very easy to  select specific tests from the full
>> > >>suite to reproduce an issue reported in  another environment - like
>> > >>the Jenkins build server or another  contributor's environment.
>> > >> 2.) This community has done an excellent job of incorporating
>> > >>well-placed  log messages to make it easy to post mortem
>> > >>troubleshoot most failures.
>> > >>  The logs are very useful, and it is extremely rare that
>> > >>troubleshooting a  failure requires debugging a live repro.
>> > >> 3.) Hadoop is written primarily in Java, a cross-platform language
>> > >>that  provides its own platform in the form of the JVM to insulate
>> > >>most of the  code from the specifics of the OS layer.
>> > >> 4.) CoPDoC - The right priorities, and well stated.
>> > >>
>> > >>
>> > >> Thank you,
>> > >>
>> > >> John
>> > >>
>> > >> -----Original Message-----
>> > >> From: Ivan Mitic [mailto:ivanmi@microsoft.com]
>> > >> Sent: Wednesday, February 27, 2013 6:32 PM
>> > >> To: mapreduce-dev@hadoop.apache.org; common-dev@hadoop.apache.org
>> > >> Cc: yarn-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org
>> > >> Subject: RE: [Vote] Merge branch-trunk-win to trunk
>> > >>
>> > >> +1 (non-binding)
>> > >>
>> > >> I am really glad to see this happening! As people already
>> > >>mentioned, this  has been a great engineering effort involving many
>> > >>people!
>> > >>
>> > >>
>> > >> Folks raised some valid concerns below and I thought it would be
>> > >>good to  share my 2 cents. In my opinion, we don't have to solve all
>> > >>these problems  right now. As we move forward with two platforms, we
>> > >>can start addressing  one problem at a time and incrementally
>> > >>improve. In the first iteration,  maintaining Hadoop on Windows
>> > >>could be just everyone trying to do their  best effort (make sure
>> > >>Jenkins build succeeds at least). We already have  people who are
>> > >>building/running trunk on Windows daily, so they would jump  in and
>> > >>fix problems as needed (we've been doing this in branch-trunk-win
>> > >>for a while now). Although I see that the problems could arise with
>> > >>platform specific features/optimizations, I don't think these are
>> > >>frequent,  so in most cases everything will just work. Merging the
>> > >>two branches sooner  rather than later does seems like the right
>> > >>thing to do if the ultimate  goal is to have Hadoop on both
>> > >>platforms. Now that the port has completed,  we will have people in
>> > >>Microsoft (and elsewhere) wanting to contribute
>> > >>features/improvements to the trunk branch. A separate branch would
>> > >>just  make things more difficult and confusing for everyone :) Hope
>> > >>this makes  sense.
>> > >>
>> > >> -----Original Message-----
>> > >> From: Todd Lipcon [mailto:todd@cloudera.com]
>> > >> Sent: Wednesday, February 27, 2013 3:43 PM
>> > >> To: common-dev@hadoop.apache.org
>> > >> Cc: yarn-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org;
>> > >> mapreduce-dev@hadoop.apache.org
>> > >> Subject: Re: [Vote] Merge branch-trunk-win to trunk
>> > >>
>> > >> On Wed, Feb 27, 2013 at 2:54 PM, Suresh Srinivas <
>> > suresh@hortonworks.com
>> > >> >wrote:
>> > >>
>> > >> > With that we need to decide how our precommit process looks.
>> > >> > My inclination is to wait for +1 from precommit builds on both
>> > >> > the platforms to ensure no issues are introduced.
>> > >> > Thoughts?
>> > >> >
>> > >> > 2. Feature development impact
>> > >> > Some questions have been raised about would new features need to
>> > >> > be supported on both the platforms. Yes. I do not see a reason
>> > >> > why features cannot work on both the platforms, with the
>> > >> > exception of platform specific optimizations. This what Java gives
>> us.
>> > >> >
>> > >> >
>> > >> I'm concerned about the above. Personally, I don't have access to
>> > >>any  Windows boxes with development tools, and I know nothing about
>> > >>developing  on Windows. The only Windows I run is an 8GB VM with 1
>> > >>GB RAM allocated,  for powerpoint :)
>> > >>
>> > >> If I submit a patch and it gets -1 "tests failed" on the Windows
>> > >> slave, how am I supposed to proceed?
>> > >>
>> > >> I think a reasonable compromise would be that the tests should
>> > >>always
>> > >> *build* on Windows before commit, and contributors should do their
>> > >>best to  look at the test logs for any Windows-specific failures.
>> > >>But, beyond  looking at the logs, a "-1 Tests failed on windows"
>> > >>should not block a  commit.
>> > >>
>> > >> Those contributors who are interested in Windows being a
>> > >> first-class platform should be responsible for watching the Windows
>> > >> builds and debugging/fixing any regressions that might be
>> Windows-specific.
>> > >>
>> > >> I also think the KDE model that Harsh pointed out is an interesting
>> > >>one
>> > >>--
>> > >> ie the idea that we would not merge windows support to trunk, but
>> > >>rather  treat is as a "parallel code line" which lives in the ASF
>> > >>and has its own  builds and releases. The windows team would
>> > >>periodically merge
>> > >>trunk->win
>> > >> to pick up any new changes, and do a separate test/release process.
>> > >>I'm not  convinced this is the best idea, but worth discussion of
>> > >>pros and cons.
>> > >>
>> > >> -Todd
>> > >>
>> > >>
>> > >> >
>> > >> > On Wed, Feb 27, 2013 at 11:56 AM, Eli Collins <el...@cloudera.com>
>> > >>wrote:
>> > >> >
>> > >> > > Bobby raises some good questions.  A related one, since most
>> > >> > > current developers won't add Windows support for new features
>> > >> > > that are platform specific is it assumed that Windows
>> > >> > > development will either lag or will people actively work on
>> > >> > > keeping Windows up with the latest?  And vice versa in case
>> > >> > > Windows support is implemented
>> > >>first.
>> > >> > >
>> > >> > > Is there a jira for resolving the outstanding TODOs in the code
>> > >> > > base (similar to HDFS-2148)?  Looks like this merge doesn't
>> > >> > > introduce many which is great (just did a quick diff and grep).
>> > >> > >
>> > >> > > Thanks,
>> > >> > > Eli
>> > >> > >
>> > >> > > On Wed, Feb 27, 2013 at 8:17 AM, Robert Evans
>> > >> > > <ev...@yahoo-inc.com>
>> > >> > wrote:
>> > >> > > > After this is merged in is Windows still going to be a second
>> > >> > > > class citizen but happens to work for more than just
>> > >> > > > development or is it a fully supported platform where if
>> > >> > > > something breaks it can block a
>> > >> > > release?
>> > >> > > >  How do we as a community intend to keep Windows support from
>> > >> breaking?
>> > >> > > > We don't have any Jenkins slaves to be able to run nightly
>> > >> > > > tests to validate everything still compiles/runs.  This is
>> > >> > > > not a blocker for me because we often rely on individuals and
>> > >> > > > groups to test Hadoop, but I
>> > >> > do
>> > >> > > > think we need to have this discussion before we put it in.
>> > >> > > >
>> > >> > > > --Bobby
>> > >> > > >
>> > >> > > > On 2/26/13 4:55 PM, "Suresh Srinivas"
>> > >> > > > <su...@hortonworks.com>
>> > >> wrote:
>> > >> > > >
>> > >> > > >>I had posted heads up about merging branch-trunk-win to trunk
>> > >> > > >>on Feb
>> > >> > 8th.
>> > >> > > >>I
>> > >> > > >>am happy to announce that we are ready for the merge.
>> > >> > > >>
>> > >> > > >>Here is a brief recap on the highlights of the work done:
>> > >> > > >>- Command-line scripts for the Hadoop surface area
>> > >> > > >>- Mapping the HDFS permissions model to Windows
>> > >> > > >>- Abstracted and reconciled mismatches around differences in
>> > >> > > >>Path semantics in Java and Windows
>> > >> > > >>- Native Task Controller for Windows
>> > >> > > >>- Implementation of a Block Placement Policy to support cloud
>> > >> > > >>environments, more specifically Azure.
>> > >> > > >>- Implementation of Hadoop native libraries for Windows
>> > >> > > >>(compression codecs, native I/O)
>> > >> > > >>- Several reliability issues, including race-conditions,
>> > >> > > >>intermittent
>> > >> > > test
>> > >> > > >>failures, resource leaks.
>> > >> > > >>- Several new unit test cases written for the above changes
>> > >> > > >>
>> > >> > > >>Please find the details of the work in
>> > >> > > >>CHANGES.branch-trunk-win.txt - Common
>> > >> > > >>changes<http://bit.ly/Xe7Ynv>, HDFS changes<
>> > >> > http://bit.ly/13QOSo9
>> > >> > > >,
>> > >> > > >>and YARN and MapReduce changes <http://bit.ly/128zzMt>. This
>> > >> > > >>is the
>> > >> > work
>> > >> > > >>ported from branch-1-win to a branch based on trunk.
>> > >> > > >>
>> > >> > > >>For details of the testing done, please see the thread -
>> > >> > > >>http://bit.ly/WpavJ4. Merge patch for this is available on
>> > >> > HADOOP-8562<
>> > >> > > >>https://issues.apache.org/jira/browse/HADOOP-8562>.
>> > >> > > >>
>> > >> > > >>This was a large undertaking that involved developing code,
>> > >> > > >>testing the entire Hadoop stack, including scale tests. This
>> > >> > > >>is made possible only with the contribution from many many
>> > >> > > >>folks in the community. Following
>> > >> > people
>> > >> > > >>contributed to this work: Ivan Mitic, Chuan Liu, Ramya Sunil,
>> > >> > > >>Bikas
>> > >> > Saha,
>> > >> > > >>Kanna Karanam, John Gordon, Brandon Li, Chris Nauroth, David
>> > >> > > >>Lao,
>> > >> > > Sumadhur
>> > >> > > >>Reddy Bolli, Arpit Agarwal, Ahmed El Baz, Mike Liddell, Jing
>> > >> > > >>Zhao,
>> > >> > Thejas
>> > >> > > >>Nair, Steve Maine, Ganeshan Iyer, Raja Aluri, Giridharan
>> > >> > > >>Kesavan, Ramya Bharathi Nimmagadda, Daryn Sharp, Arun Murthy,
>> > >> > > >>Tsz-Wo Nicholas Sze,
>> > >> > > Suresh
>> > >> > > >>Srinivas and Sanjay Radia. There are many others who
>> > >> > > >>contributed as
>> > >> > well
>> > >> > > >>providing feedback and comments on numerous jiras.
>> > >> > > >>
>> > >> > > >>The vote will run for seven days and will end on March 5,
>> > >> > > >>6:00PM
>> > >>PST.
>> > >> > > >>
>> > >> > > >>Regards,
>> > >> > > >>Suresh
>> > >> > > >>
>> > >> > > >>
>> > >> > > >>
>> > >> > > >>
>> > >> > > >>On Thu, Feb 7, 2013 at 6:41 PM, Mahadevan Venkatraman
>> > >> > > >><ma...@microsoft.com>wrote:
>> > >> > > >>
>> > >> > > >>> It is super exciting to look at the prospect of these
>> > >> > > >>>changes being merged  to trunk. Having Windows as one of the
>> > >> > > >>>supported Hadoop platforms is
>> > >> > a
>> > >> > > >>> fantastic opportunity both for the Hadoop project and
>> > >> > > >>>Microsoft customers.
>> > >> > > >>>
>> > >> > > >>> This work began around a year back when a few of us started
>> > >> > > >>> with a
>> > >> > > basic
>> > >> > > >>> port of Hadoop on Windows. Ever since, the Hadoop team in
>> > >> > > >>> Microsoft
>> > >> > > have
>> > >> > > >>> made significant progress in the following areas:
>> > >> > > >>> (PS: Some of these items are already included in Suresh's
>> > >> > > >>> email, but including again for completeness)
>> > >> > > >>>
>> > >> > > >>> - Command-line scripts for the Hadoop surface area
>> > >> > > >>> - Mapping the HDFS permissions model to Windows
>> > >> > > >>> - Abstracted and reconciled mismatches around differences
>> > >> > > >>> in Path semantics in Java and Windows
>> > >> > > >>> - Native Task Controller for Windows
>> > >> > > >>> - Implementation of a Block Placement Policy to support
>> > >> > > >>> cloud environments, more specifically Azure.
>> > >> > > >>> - Implementation of Hadoop native libraries for Windows
>> > >> > > >>> (compression codecs, native I/O) - Several reliability
>> > >> > > >>> issues, including race-conditions, intermittent test
>> > >> > > >>> failures, resource
>> > >> leaks.
>> > >> > > >>> - Several new unit test cases written for the above changes
>> > >> > > >>>
>> > >> > > >>> In the process, we have closely engaged with the Apache
>> > >> > > >>> open source community and have got great support and
>> > >> > > >>> assistance from the
>> > >> > community
>> > >> > > >>>in
>> > >> > > >>> terms of contributing fixes, code review comments and
>> commits.
>> > >> > > >>>
>> > >> > > >>> In addition, the Hadoop team at Microsoft has also made
>> > >> > > >>> good progress
>> > >> > > in
>> > >> > > >>> other projects including Hive, Pig, Sqoop, Oozie, HCat and
>> > >>HBase.
>> > >> > Many
>> > >> > > >>>of
>> > >> > > >>> these changes have already been committed to the respective
>> > >> > > >>>trunks
>> > >> > with
>> > >> > > >>> help from various committers and contributors. It is great
>> > >> > > >>> to see the commitment of the community to support multiple
>> > >> > > >>> platforms, and we
>> > >> > look
>> > >> > > >>> forward to the day when a developer/customer is able to
>> > >> > > >>>successfully deploy  a complete solution stack based on
>> > >> > > >>>Apache Hadoop releases.
>> > >> > > >>>
>> > >> > > >>> Next Steps:
>> > >> > > >>>
>> > >> > > >>> All of the above changes are part of the Windows Azure
>> > >> > > >>>HDInsight and  HDInsight Server products from Microsoft. We
>> > >> > > >>>have successfully on-boarded  several internal customers and
>> > >> > > >>>have been running production workloads
>> > >> > > on
>> > >> > > >>> Windows Azure HDInsight. Our vision is to create a big data
>> > >> > > >>>platform based  on Hadoop, and we are committed to helping
>> > >> > > >>>make Hadoop a world-class  solution that anyone can use to
>> > >> > > >>>solve their biggest data challenges.
>> > >> > > >>>
>> > >> > > >>> As an immediate next step, we would like to have a
>> > >> > > >>> discussion around
>> > >> > > how
>> > >> > > >>> we can ensure that the quality of the mainline Hadoop
>> > >> > > >>>branches on Windows  is maintained. To this end, we would
>> > >> > > >>>like to get to the state where
>> > >> > we
>> > >> > > >>>have
>> > >> > > >>> pre-checkin validation gates and nightly test runs enabled
>> > >> > > >>>on
>> > >> > Windows.
>> > >> > > >>>If
>> > >> > > >>> you have any suggestions around this, please do send an
>> email.
>> > >> > > >>>We
>> > >> > are
>> > >> > > >>> committed to helping sustain the long-term quality of
>> > >> > > >>>Hadoop on both Linux  and Windows.
>> > >> > > >>>
>> > >> > > >>> We sincerely thank the community for their contribution and
>> > >> > > >>> support
>> > >> > so
>> > >> > > >>> far. And hope to continue having a close engagement in the
>> > >>future.
>> > >> > > >>>
>> > >> > > >>> -Microsoft HDInsight Team
>> > >> > > >>>
>> > >> > > >>>
>> > >> > > >>> -----Original Message-----
>> > >> > > >>> From: Suresh Srinivas [mailto:suresh@hortonworks.com]
>> > >> > > >>> Sent: Thursday, February 7, 2013 5:42 PM
>> > >> > > >>> To: common-dev@hadoop.apache.org;
>> > >> > > >>> yarn-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org;
>> > >> > > >>> mapreduce-dev@hadoop.apache.org
>> > >> > > >>> Subject: Heads up - merge branch-trunk-win to trunk
>> > >> > > >>>
>> > >> > > >>> The support for Hadoop on Windows was proposed in
>> > >> > > >>> HADOOP-8079<
>> > >> > > >>> https://issues.apache.org/jira/browse/HADOOP-8079> almost a
>> > year
>> > >> > ago.
>> > >> > > >>>The
>> > >> > > >>> goal was to make Hadoop natively integrated, full-featured,
>> > >> > > >>>and performance  and scalability tuned on Windows Server or
>> > >> > > >>>Windows Azure.
>> > >> > > >>> We are happy to announce that a lot of progress has been
>> > >> > > >>>made in this  regard.
>> > >> > > >>>
>> > >> > > >>> Initial work started in a feature branch, branch-1-win,
>> > >> > > >>>based on branch-1.
>> > >> > > >>> The details related to the work done in the branch can be
>> > >> > > >>>seen in  CHANGES.txt<
>> > >> > > >>>
>> > >> > > >>>
>> > >> > >
>> > http://svn.apache.org/viewvc/hadoop/common/branches/branch-1-win/CHA
>> > >> > > NGES
>> > >> > .
>> > >> > > >>>branch-1-win.txt?view=markup
>> > >> > > >>> >.
>> > >> > > >>> This work has been ported to a branch, branch-trunk-win,
>> > >> > > >>> based on
>> > >> > > trunk.
>> > >> > > >>> Merge patch for this is available on
>> > >> > > >>> HADOOP-8562<https://issues.apache.org/jira/browse/HADOOP-85
>> > >> > > >>> 62>
>> > >> > > >>> .
>> > >> > > >>>
>> > >> > > >>> Highlights of the work done so far:
>> > >> > > >>> 1. Necessary changes in Hadoop to run natively on Windows.
>> > >> > > >>> These
>> > >> > > changes
>> > >> > > >>> handle differences in platforms related to path names,
>> > >> > > >>>process/task  management etc.
>> > >> > > >>> 2. Addition of winutils tools for managing file permissions
>> > >> > > >>>and ownership,  user group mapping, hardlinks, symbolic
>> > >> > > >>>links, chmod, disk
>> > >> > utilization,
>> > >> > > >>>and
>> > >> > > >>> process/task management.
>> > >> > > >>> 3. Added cmd scripts equivalent to existing shell scripts
>> > >> > > >>>hadoop-daemon.sh, start and stop scripts.
>> > >> > > >>> 4. Addition of block placement policy implemnation to
>> > >> > > >>>support cloud  enviroment, more specifically Azure.
>> > >> > > >>>
>> > >> > > >>> We are very close to wrapping up the work in
>> > >> > > >>>branch-trunk-win and getting  ready for a merge. Currently
>> > >> > > >>>the merge patch is passing close to 100%
>> > >> > > of
>> > >> > > >>> unit tests on Linux. Soon I will call for a vote to merge
>> > >> > > >>>this branch into  trunk.
>> > >> > > >>>
>> > >> > > >>> Next steps:
>> > >> > > >>> 1. Call for vote to merge branch-trunk-win to trunk, when
>> > >> > > >>> the work completes and precommit build is clean.
>> > >> > > >>> 2. Start a discussion on adding Jenkins precommit builds on
>> > >> > > >>> windows
>> > >> > and
>> > >> > > >>> how to integrate that with the existing commit process.
>> > >> > > >>>
>> > >> > > >>> Let me know if you have any questions.
>> > >> > > >>>
>> > >> > > >>> Regards,
>> > >> > > >>> Suresh
>> > >> > > >>>
>> > >> > > >>>
>> > >> > > >>
>> > >> > > >>
>> > >> > > >>--
>> > >> > > >>http://hortonworks.com/download/
>> > >> > > >
>> > >> > >
>> > >> >
>> > >> >
>> > >> >
>> > >> > --
>> > >> > http://hortonworks.com/download/
>> > >> >
>> > >>
>> > >>
>> > >>
>> > >> --
>> > >> Todd Lipcon
>> > >> Software Engineer, Cloudera
>> > >>
>> > >>
>> > >>
>> > >>
>> > >>
>> >
>> >
>>
>>
>

Fwd: [Vote] Merge branch-trunk-win to trunk

[Matt Foley <ma...@apache.org>]

+1 (binding)

Apache is supposed to be about the community.  We have here a community of
developers, who have actively and openly worked to add a major improvement
to Hadoop: the ability to work cross-platform.  Furthermore, the size of
the substantive part of the needed patch is only about 1500 lines, much
smaller than quite a few other additions to Hadoop over the last few
months.  We should welcome and support this change, and make sure that the
code stays cross-platform going forward by extending our CI practices,
especially pre-commit "test-patch", to also include Windows.

As most of you know, my colleague Giri Kesavan (PMC member) helps maintain
the Linux CI capability for Hadoop.  I've talked with him, and he and I are
committing to getting test-patch implemented for Windows, so that along
with the current automated "+1"s required to commit, we can add two more,
for javac build in Windows and core unit tests in Windows.

Members of the team implementing cross-platform compatibility, including
Microsoft employees, have opened the discussion for providing hardware or
VM resources to perform this additional CI testing.  I will assist them to
work with the Apache Infra team and figure out how to make it happen.

I understand there is some concern about the additional platform test.
 My going-in
presumption, based on Java's intrinsic, pretty-good, cross-platform
compatibility, is that patches to Hadoop will by default also have
cross-platform compatibility, unless they are written in an explicitly
platform-dependent way.  I also believe that in the vast majority of cases
the cross-platform compatibility of Java will carry thru to Hadoop patches,
without additional effort on the developer's part.

Let's try it, and see what happens.  If we actually find a frequent
difficulty, we'll change to engineer around it.  But I believe that, in the
rare cases where a Windows-specific failure occurs, there will be a number
of people (new, enthusiastic members of the community! :-) willing to help.
 If such help is not forthcoming, then we can discuss work-arounds, but
like a previous poster, I am confident in the community.

Regards,
--Matt



On Thu, Feb 28, 2013 at 12:21 PM, Chuan Liu <ch...@microsoft.com> wrote:

>  +1 (non-binding)
>>
>> As someone also contributed to porting Hadoop to Windows, I think Java
>> already provided a very good platform independent platform.
>> For features that are not available in Java, we will try to provide our
>> platform independent APIs that abstract OS tasks away.
>> Most features should have no difficulty running on Windows and Linux by
>> using Java and those platform independent APIs.
>>
>> For concerns raise on new features that may fail on Windows, I think we
>> don't need to require passing on Windows a mandate at the moment. We can
>> simply mark it unavailable to Windows and port it later if the feature is
>> important.
>>
>> -Chuan
>>
>> -----Original Message-----
>> From: Chris Nauroth [mailto:cnauroth@hortonworks.com]
>> Sent: Thursday, February 28, 2013 11:51 AM
>> To: hdfs-dev@hadoop.apache.org
>> Cc: mapreduce-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org;
>> common-dev@hadoop.apache.org
>> Subject: Re: [Vote] Merge branch-trunk-win to trunk
>>
>> > Is there a jira for resolving the outstanding TODOs in the code base
>> > (similar to HDFS-2148)?  Looks like this merge doesn't introduce many
>> > which is great (just did a quick diff and grep).
>>
>> I found 2 remaining TODOs introduced in the current merge patch.  One is
>> in ContainerLaunch.java.  The container launch script was trying to set a
>> CLASSPATH that exceeded the Windows maximum command line length.  The fix
>> was to wrap the long classpath into an intermediate jar containing only a
>> manifest file with a Class-Path entry.  (See YARN-316.)  Just to be
>> conservative, we wrapped this logic in an if (Shell.WINDOWS) guard and
>> marked a TODO to remove it later and use that approach on all platforms
>> after additional testing.  I've tested this code path successfully on Mac
>> too, but several people wanted additional testing and performance checks
>> before removing the if (Shell.WINDOWS) guard.  That work is tracked in an
>> existing jira: YARN-358.
>>
>> The other TODO is for winutils to print more usage information and
>> examples.  At this point, I think winutils is printing sufficient
>> information, and we can just remove the TODO.  I just submitted a new jira
>> to start that conversation: HADOOP-9348.
>>
>> Thank you,
>> --Chris
>>
>>
>> On Thu, Feb 28, 2013 at 11:29 AM, Robert Evans <ev...@yahoo-inc.com>
>> wrote:
>>
>> > My initial question was mostly intended to understand the desired new
>> > classification of Windows after the merge, and how we plan to maintain
>> > Windows support.  I am happy to hear that hardware for Jenkins will be
>> > provided.  I am also fine, at least initially, with us trying to treat
>> > Windows as a first class supported platform.  But I realize that there
>> > are a lot of people that do not have easy access to Windows for
>> > development/debugging, myself included. I also don't want to slow down
>> > the pace of development too much because of this.  It will cause some
>> > organizations that do not use or support Windows to be more likely to
>> > run software that has diverged from an official release.  It also has
>> > the potential to make the patch submission process even more
>> > difficult, which increases the likelihood of submitters abandoning
>> > patches.  However, the great thing about being in a community is we can
>> change if we need to.
>> >
>> > I am +0 for the merge.  I am not a Windows expert so I don't feel
>> > comfortable giving it a true +1.
>> >
>> > --Bobby
>> >
>> >
>> > On 2/28/13 10:45 AM, "Chris Nauroth" <cn...@hortonworks.com> wrote:
>> >
>> > >I'd like to share a few anecdotes about developing cross-platform,
>> > >hopefully to address some of the concerns about adding overhead to
>> > >the development process.  By reviewing past cases of cross-platform
>> Linux vs.
>> > >Windows bugs, we can get a sense for how the development process
>> > >could look in the future.
>> > >
>> > >HADOOP-9131: TestLocalFileSystem#testListStatusWithColons cannot run
>> > >on Windows.  As part of an earlier jira, HADOOP-8962, there was a new
>> > >test committed on trunk covering the case of a local file system
>> > >interaction on a file containing a ':'.  On Windows, ':' in a path
>> > >has special meaning as part of the drive specifier (i.e. C:), so this
>> > >test cannot pass when running on Windows.  In this kind of case, the
>> > >cross-platform bug is obvious, and the fix is obvious
>> > >(assumeTrue(!Shell.WINDOWS)).  Ideally, this would get fixed
>> > >pre-commit after seeing a -1 from the Windows Jenkins slave.
>> > >
>> > >HDFS-4274: BlockPoolSliceScanner does not close verification log
>> > >during shutdown.  This caused problems for MiniDFSCluster-based tests
>> > >running on Windows.  Failure to close the verification log meant that
>> > >we didn't release file locks, so the tests couldn't delete/recreate
>> > >working directories during teardown/setup.  Arguably, this was always
>> > >a bug, and running on Windows just exposed it because of its stricter
>> > >rules about file locking.  This is a more complex fix, but it doesn't
>> > >require platform-specific knowledge.  If some future patch
>> > >accidentally regresses this, then we'll likely see +1 from Linux
>> > >Jenkins and -1 from Windows Jenkins.  Ideally, it would get fixed
>> > >pre-commit, because it doesn't require Windows-specific knowledge.
>> > >There is also the matter of impact.
>> > > Re-breaking this would re-break many test suites on Windows.
>> > >
>> > >HADOOP-9232: JniBasedUnixGroupsMappingWithFallback fails on Windows
>> > >with UnsatisfiedLinkError.  This was introduced by HADOOP-8712, which
>> > >switched to JniBasedUnixGroupsMappingWithFallback as the default
>> > >hadoop.security.group.mapping, but did not provide a Windows
>> > >implementation of the JNI function.  In this case, there was a strong
>> > >desire to get
>> > >HADOOP-8712 into a release, fixing it on Windows required native
>> > >Windows API knowledge, and Windows users had a simple workaround
>> > >available by changing their configs back to
>> > >ShellBasedUnixGroupsMapping.  I think this is the kind of situation
>> > >where we could allow HADOOP-8712 to commit despite
>> > >-1 from Windows Jenkins, with fairly quick follow-up from an engineer
>> > >with the Windows expertise to fix it.
>> > >
>> > >To summarize, I don't think it needs to differ greatly from our
>> > >current development process.  We're all responsible for breadth of
>> > >understanding and maintenance of the whole codebase, but we also rely
>> > >on specific individuals with deep expertise in particular areas for
>> certain issues.
>> > > Sometimes we commit despite a -1 from Jenkins, based on the
>> > >community's judgment.
>> > >
>> > >Virtualization greatly simplifies cross-platform development.  I use
>> > >VirtualBox on a Mac host and run VMs for Windows and Ubuntu with a
>> > >shared drive so that they can all see the same copy of the source
>> > >code.  There are plenty of variations on this depending on your
>> > >preference, such as offloading the VMs to a separate server or cloud
>> > >service to free up local RAM.  I'm planning on submitting
>> > >BUILDING.txt changes later today that fully describe how to build on
>> > >Windows.  After some initial setup, it's nearly identical to the mvn
>> > >commands that you already use today.
>> > >
>> > >Hope this helps,
>> > >--Chris
>> > >
>> > >
>> > >On Thu, Feb 28, 2013 at 3:25 AM, John Gordon
>> > ><Jo...@microsoft.com>wrote:
>> > >
>> > >> +1 (non-binding)
>> > >>
>> > >> I want to share my vote of confidence in this community.  If
>> > >>motivated to  do so, this community can keep this project
>> > >>cross-platform and continue to  rapidly innovate without breaking a
>> > >>sweat.
>> > >>
>> > >> The day we started working on this, I saw the foundations of
>> > >>greatness in  the quality and volume of dev tests, the code itself,
>> > >>and the Apache values  themselves.
>> > >>
>> > >> 1.) Hadoop's unit tests and their frameworks are very well thought
>> > >>out and  the consideration and energy that went into their design is
>> > >>worthy of  praise.  The MiniCluster abstractions utilize very few
>> > >>resources and put  all the processes into one JVM for easy
>> > >>debugging.  It is very easy to  select specific tests from the full
>> > >>suite to reproduce an issue reported in  another environment - like
>> > >>the Jenkins build server or another  contributor's environment.
>> > >> 2.) This community has done an excellent job of incorporating
>> > >>well-placed  log messages to make it easy to post mortem
>> > >>troubleshoot most failures.
>> > >>  The logs are very useful, and it is extremely rare that
>> > >>troubleshooting a  failure requires debugging a live repro.
>> > >> 3.) Hadoop is written primarily in Java, a cross-platform language
>> > >>that  provides its own platform in the form of the JVM to insulate
>> > >>most of the  code from the specifics of the OS layer.
>> > >> 4.) CoPDoC - The right priorities, and well stated.
>> > >>
>> > >>
>> > >> Thank you,
>> > >>
>> > >> John
>> > >>
>> > >> -----Original Message-----
>> > >> From: Ivan Mitic [mailto:ivanmi@microsoft.com]
>> > >> Sent: Wednesday, February 27, 2013 6:32 PM
>> > >> To: mapreduce-dev@hadoop.apache.org; common-dev@hadoop.apache.org
>> > >> Cc: yarn-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org
>> > >> Subject: RE: [Vote] Merge branch-trunk-win to trunk
>> > >>
>> > >> +1 (non-binding)
>> > >>
>> > >> I am really glad to see this happening! As people already
>> > >>mentioned, this  has been a great engineering effort involving many
>> > >>people!
>> > >>
>> > >>
>> > >> Folks raised some valid concerns below and I thought it would be
>> > >>good to  share my 2 cents. In my opinion, we don't have to solve all
>> > >>these problems  right now. As we move forward with two platforms, we
>> > >>can start addressing  one problem at a time and incrementally
>> > >>improve. In the first iteration,  maintaining Hadoop on Windows
>> > >>could be just everyone trying to do their  best effort (make sure
>> > >>Jenkins build succeeds at least). We already have  people who are
>> > >>building/running trunk on Windows daily, so they would jump  in and
>> > >>fix problems as needed (we've been doing this in branch-trunk-win
>> > >>for a while now). Although I see that the problems could arise with
>> > >>platform specific features/optimizations, I don't think these are
>> > >>frequent,  so in most cases everything will just work. Merging the
>> > >>two branches sooner  rather than later does seems like the right
>> > >>thing to do if the ultimate  goal is to have Hadoop on both
>> > >>platforms. Now that the port has completed,  we will have people in
>> > >>Microsoft (and elsewhere) wanting to contribute
>> > >>features/improvements to the trunk branch. A separate branch would
>> > >>just  make things more difficult and confusing for everyone :) Hope
>> > >>this makes  sense.
>> > >>
>> > >> -----Original Message-----
>> > >> From: Todd Lipcon [mailto:todd@cloudera.com]
>> > >> Sent: Wednesday, February 27, 2013 3:43 PM
>> > >> To: common-dev@hadoop.apache.org
>> > >> Cc: yarn-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org;
>> > >> mapreduce-dev@hadoop.apache.org
>> > >> Subject: Re: [Vote] Merge branch-trunk-win to trunk
>> > >>
>> > >> On Wed, Feb 27, 2013 at 2:54 PM, Suresh Srinivas <
>> > suresh@hortonworks.com
>> > >> >wrote:
>> > >>
>> > >> > With that we need to decide how our precommit process looks.
>> > >> > My inclination is to wait for +1 from precommit builds on both
>> > >> > the platforms to ensure no issues are introduced.
>> > >> > Thoughts?
>> > >> >
>> > >> > 2. Feature development impact
>> > >> > Some questions have been raised about would new features need to
>> > >> > be supported on both the platforms. Yes. I do not see a reason
>> > >> > why features cannot work on both the platforms, with the
>> > >> > exception of platform specific optimizations. This what Java gives
>> us.
>> > >> >
>> > >> >
>> > >> I'm concerned about the above. Personally, I don't have access to
>> > >>any  Windows boxes with development tools, and I know nothing about
>> > >>developing  on Windows. The only Windows I run is an 8GB VM with 1
>> > >>GB RAM allocated,  for powerpoint :)
>> > >>
>> > >> If I submit a patch and it gets -1 "tests failed" on the Windows
>> > >> slave, how am I supposed to proceed?
>> > >>
>> > >> I think a reasonable compromise would be that the tests should
>> > >>always
>> > >> *build* on Windows before commit, and contributors should do their
>> > >>best to  look at the test logs for any Windows-specific failures.
>> > >>But, beyond  looking at the logs, a "-1 Tests failed on windows"
>> > >>should not block a  commit.
>> > >>
>> > >> Those contributors who are interested in Windows being a
>> > >> first-class platform should be responsible for watching the Windows
>> > >> builds and debugging/fixing any regressions that might be
>> Windows-specific.
>> > >>
>> > >> I also think the KDE model that Harsh pointed out is an interesting
>> > >>one
>> > >>--
>> > >> ie the idea that we would not merge windows support to trunk, but
>> > >>rather  treat is as a "parallel code line" which lives in the ASF
>> > >>and has its own  builds and releases. The windows team would
>> > >>periodically merge
>> > >>trunk->win
>> > >> to pick up any new changes, and do a separate test/release process.
>> > >>I'm not  convinced this is the best idea, but worth discussion of
>> > >>pros and cons.
>> > >>
>> > >> -Todd
>> > >>
>> > >>
>> > >> >
>> > >> > On Wed, Feb 27, 2013 at 11:56 AM, Eli Collins <el...@cloudera.com>
>> > >>wrote:
>> > >> >
>> > >> > > Bobby raises some good questions.  A related one, since most
>> > >> > > current developers won't add Windows support for new features
>> > >> > > that are platform specific is it assumed that Windows
>> > >> > > development will either lag or will people actively work on
>> > >> > > keeping Windows up with the latest?  And vice versa in case
>> > >> > > Windows support is implemented
>> > >>first.
>> > >> > >
>> > >> > > Is there a jira for resolving the outstanding TODOs in the code
>> > >> > > base (similar to HDFS-2148)?  Looks like this merge doesn't
>> > >> > > introduce many which is great (just did a quick diff and grep).
>> > >> > >
>> > >> > > Thanks,
>> > >> > > Eli
>> > >> > >
>> > >> > > On Wed, Feb 27, 2013 at 8:17 AM, Robert Evans
>> > >> > > <ev...@yahoo-inc.com>
>> > >> > wrote:
>> > >> > > > After this is merged in is Windows still going to be a second
>> > >> > > > class citizen but happens to work for more than just
>> > >> > > > development or is it a fully supported platform where if
>> > >> > > > something breaks it can block a
>> > >> > > release?
>> > >> > > >  How do we as a community intend to keep Windows support from
>> > >> breaking?
>> > >> > > > We don't have any Jenkins slaves to be able to run nightly
>> > >> > > > tests to validate everything still compiles/runs.  This is
>> > >> > > > not a blocker for me because we often rely on individuals and
>> > >> > > > groups to test Hadoop, but I
>> > >> > do
>> > >> > > > think we need to have this discussion before we put it in.
>> > >> > > >
>> > >> > > > --Bobby
>> > >> > > >
>> > >> > > > On 2/26/13 4:55 PM, "Suresh Srinivas"
>> > >> > > > <su...@hortonworks.com>
>> > >> wrote:
>> > >> > > >
>> > >> > > >>I had posted heads up about merging branch-trunk-win to trunk
>> > >> > > >>on Feb
>> > >> > 8th.
>> > >> > > >>I
>> > >> > > >>am happy to announce that we are ready for the merge.
>> > >> > > >>
>> > >> > > >>Here is a brief recap on the highlights of the work done:
>> > >> > > >>- Command-line scripts for the Hadoop surface area
>> > >> > > >>- Mapping the HDFS permissions model to Windows
>> > >> > > >>- Abstracted and reconciled mismatches around differences in
>> > >> > > >>Path semantics in Java and Windows
>> > >> > > >>- Native Task Controller for Windows
>> > >> > > >>- Implementation of a Block Placement Policy to support cloud
>> > >> > > >>environments, more specifically Azure.
>> > >> > > >>- Implementation of Hadoop native libraries for Windows
>> > >> > > >>(compression codecs, native I/O)
>> > >> > > >>- Several reliability issues, including race-conditions,
>> > >> > > >>intermittent
>> > >> > > test
>> > >> > > >>failures, resource leaks.
>> > >> > > >>- Several new unit test cases written for the above changes
>> > >> > > >>
>> > >> > > >>Please find the details of the work in
>> > >> > > >>CHANGES.branch-trunk-win.txt - Common
>> > >> > > >>changes<http://bit.ly/Xe7Ynv>, HDFS changes<
>> > >> > http://bit.ly/13QOSo9
>> > >> > > >,
>> > >> > > >>and YARN and MapReduce changes <http://bit.ly/128zzMt>. This
>> > >> > > >>is the
>> > >> > work
>> > >> > > >>ported from branch-1-win to a branch based on trunk.
>> > >> > > >>
>> > >> > > >>For details of the testing done, please see the thread -
>> > >> > > >>http://bit.ly/WpavJ4. Merge patch for this is available on
>> > >> > HADOOP-8562<
>> > >> > > >>https://issues.apache.org/jira/browse/HADOOP-8562>.
>> > >> > > >>
>> > >> > > >>This was a large undertaking that involved developing code,
>> > >> > > >>testing the entire Hadoop stack, including scale tests. This
>> > >> > > >>is made possible only with the contribution from many many
>> > >> > > >>folks in the community. Following
>> > >> > people
>> > >> > > >>contributed to this work: Ivan Mitic, Chuan Liu, Ramya Sunil,
>> > >> > > >>Bikas
>> > >> > Saha,
>> > >> > > >>Kanna Karanam, John Gordon, Brandon Li, Chris Nauroth, David
>> > >> > > >>Lao,
>> > >> > > Sumadhur
>> > >> > > >>Reddy Bolli, Arpit Agarwal, Ahmed El Baz, Mike Liddell, Jing
>> > >> > > >>Zhao,
>> > >> > Thejas
>> > >> > > >>Nair, Steve Maine, Ganeshan Iyer, Raja Aluri, Giridharan
>> > >> > > >>Kesavan, Ramya Bharathi Nimmagadda, Daryn Sharp, Arun Murthy,
>> > >> > > >>Tsz-Wo Nicholas Sze,
>> > >> > > Suresh
>> > >> > > >>Srinivas and Sanjay Radia. There are many others who
>> > >> > > >>contributed as
>> > >> > well
>> > >> > > >>providing feedback and comments on numerous jiras.
>> > >> > > >>
>> > >> > > >>The vote will run for seven days and will end on March 5,
>> > >> > > >>6:00PM
>> > >>PST.
>> > >> > > >>
>> > >> > > >>Regards,
>> > >> > > >>Suresh
>> > >> > > >>
>> > >> > > >>
>> > >> > > >>
>> > >> > > >>
>> > >> > > >>On Thu, Feb 7, 2013 at 6:41 PM, Mahadevan Venkatraman
>> > >> > > >><ma...@microsoft.com>wrote:
>> > >> > > >>
>> > >> > > >>> It is super exciting to look at the prospect of these
>> > >> > > >>>changes being merged  to trunk. Having Windows as one of the
>> > >> > > >>>supported Hadoop platforms is
>> > >> > a
>> > >> > > >>> fantastic opportunity both for the Hadoop project and
>> > >> > > >>>Microsoft customers.
>> > >> > > >>>
>> > >> > > >>> This work began around a year back when a few of us started
>> > >> > > >>> with a
>> > >> > > basic
>> > >> > > >>> port of Hadoop on Windows. Ever since, the Hadoop team in
>> > >> > > >>> Microsoft
>> > >> > > have
>> > >> > > >>> made significant progress in the following areas:
>> > >> > > >>> (PS: Some of these items are already included in Suresh's
>> > >> > > >>> email, but including again for completeness)
>> > >> > > >>>
>> > >> > > >>> - Command-line scripts for the Hadoop surface area
>> > >> > > >>> - Mapping the HDFS permissions model to Windows
>> > >> > > >>> - Abstracted and reconciled mismatches around differences
>> > >> > > >>> in Path semantics in Java and Windows
>> > >> > > >>> - Native Task Controller for Windows
>> > >> > > >>> - Implementation of a Block Placement Policy to support
>> > >> > > >>> cloud environments, more specifically Azure.
>> > >> > > >>> - Implementation of Hadoop native libraries for Windows
>> > >> > > >>> (compression codecs, native I/O) - Several reliability
>> > >> > > >>> issues, including race-conditions, intermittent test
>> > >> > > >>> failures, resource
>> > >> leaks.
>> > >> > > >>> - Several new unit test cases written for the above changes
>> > >> > > >>>
>> > >> > > >>> In the process, we have closely engaged with the Apache
>> > >> > > >>> open source community and have got great support and
>> > >> > > >>> assistance from the
>> > >> > community
>> > >> > > >>>in
>> > >> > > >>> terms of contributing fixes, code review comments and
>> commits.
>> > >> > > >>>
>> > >> > > >>> In addition, the Hadoop team at Microsoft has also made
>> > >> > > >>> good progress
>> > >> > > in
>> > >> > > >>> other projects including Hive, Pig, Sqoop, Oozie, HCat and
>> > >>HBase.
>> > >> > Many
>> > >> > > >>>of
>> > >> > > >>> these changes have already been committed to the respective
>> > >> > > >>>trunks
>> > >> > with
>> > >> > > >>> help from various committers and contributors. It is great
>> > >> > > >>> to see the commitment of the community to support multiple
>> > >> > > >>> platforms, and we
>> > >> > look
>> > >> > > >>> forward to the day when a developer/customer is able to
>> > >> > > >>>successfully deploy  a complete solution stack based on
>> > >> > > >>>Apache Hadoop releases.
>> > >> > > >>>
>> > >> > > >>> Next Steps:
>> > >> > > >>>
>> > >> > > >>> All of the above changes are part of the Windows Azure
>> > >> > > >>>HDInsight and  HDInsight Server products from Microsoft. We
>> > >> > > >>>have successfully on-boarded  several internal customers and
>> > >> > > >>>have been running production workloads
>> > >> > > on
>> > >> > > >>> Windows Azure HDInsight. Our vision is to create a big data
>> > >> > > >>>platform based  on Hadoop, and we are committed to helping
>> > >> > > >>>make Hadoop a world-class  solution that anyone can use to
>> > >> > > >>>solve their biggest data challenges.
>> > >> > > >>>
>> > >> > > >>> As an immediate next step, we would like to have a
>> > >> > > >>> discussion around
>> > >> > > how
>> > >> > > >>> we can ensure that the quality of the mainline Hadoop
>> > >> > > >>>branches on Windows  is maintained. To this end, we would
>> > >> > > >>>like to get to the state where
>> > >> > we
>> > >> > > >>>have
>> > >> > > >>> pre-checkin validation gates and nightly test runs enabled
>> > >> > > >>>on
>> > >> > Windows.
>> > >> > > >>>If
>> > >> > > >>> you have any suggestions around this, please do send an
>> email.
>> > >> > > >>>We
>> > >> > are
>> > >> > > >>> committed to helping sustain the long-term quality of
>> > >> > > >>>Hadoop on both Linux  and Windows.
>> > >> > > >>>
>> > >> > > >>> We sincerely thank the community for their contribution and
>> > >> > > >>> support
>> > >> > so
>> > >> > > >>> far. And hope to continue having a close engagement in the
>> > >>future.
>> > >> > > >>>
>> > >> > > >>> -Microsoft HDInsight Team
>> > >> > > >>>
>> > >> > > >>>
>> > >> > > >>> -----Original Message-----
>> > >> > > >>> From: Suresh Srinivas [mailto:suresh@hortonworks.com]
>> > >> > > >>> Sent: Thursday, February 7, 2013 5:42 PM
>> > >> > > >>> To: common-dev@hadoop.apache.org;
>> > >> > > >>> yarn-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org;
>> > >> > > >>> mapreduce-dev@hadoop.apache.org
>> > >> > > >>> Subject: Heads up - merge branch-trunk-win to trunk
>> > >> > > >>>
>> > >> > > >>> The support for Hadoop on Windows was proposed in
>> > >> > > >>> HADOOP-8079<
>> > >> > > >>> https://issues.apache.org/jira/browse/HADOOP-8079> almost a
>> > year
>> > >> > ago.
>> > >> > > >>>The
>> > >> > > >>> goal was to make Hadoop natively integrated, full-featured,
>> > >> > > >>>and performance  and scalability tuned on Windows Server or
>> > >> > > >>>Windows Azure.
>> > >> > > >>> We are happy to announce that a lot of progress has been
>> > >> > > >>>made in this  regard.
>> > >> > > >>>
>> > >> > > >>> Initial work started in a feature branch, branch-1-win,
>> > >> > > >>>based on branch-1.
>> > >> > > >>> The details related to the work done in the branch can be
>> > >> > > >>>seen in  CHANGES.txt<
>> > >> > > >>>
>> > >> > > >>>
>> > >> > >
>> > http://svn.apache.org/viewvc/hadoop/common/branches/branch-1-win/CHA
>> > >> > > NGES
>> > >> > .
>> > >> > > >>>branch-1-win.txt?view=markup
>> > >> > > >>> >.
>> > >> > > >>> This work has been ported to a branch, branch-trunk-win,
>> > >> > > >>> based on
>> > >> > > trunk.
>> > >> > > >>> Merge patch for this is available on
>> > >> > > >>> HADOOP-8562<https://issues.apache.org/jira/browse/HADOOP-85
>> > >> > > >>> 62>
>> > >> > > >>> .
>> > >> > > >>>
>> > >> > > >>> Highlights of the work done so far:
>> > >> > > >>> 1. Necessary changes in Hadoop to run natively on Windows.
>> > >> > > >>> These
>> > >> > > changes
>> > >> > > >>> handle differences in platforms related to path names,
>> > >> > > >>>process/task  management etc.
>> > >> > > >>> 2. Addition of winutils tools for managing file permissions
>> > >> > > >>>and ownership,  user group mapping, hardlinks, symbolic
>> > >> > > >>>links, chmod, disk
>> > >> > utilization,
>> > >> > > >>>and
>> > >> > > >>> process/task management.
>> > >> > > >>> 3. Added cmd scripts equivalent to existing shell scripts
>> > >> > > >>>hadoop-daemon.sh, start and stop scripts.
>> > >> > > >>> 4. Addition of block placement policy implemnation to
>> > >> > > >>>support cloud  enviroment, more specifically Azure.
>> > >> > > >>>
>> > >> > > >>> We are very close to wrapping up the work in
>> > >> > > >>>branch-trunk-win and getting  ready for a merge. Currently
>> > >> > > >>>the merge patch is passing close to 100%
>> > >> > > of
>> > >> > > >>> unit tests on Linux. Soon I will call for a vote to merge
>> > >> > > >>>this branch into  trunk.
>> > >> > > >>>
>> > >> > > >>> Next steps:
>> > >> > > >>> 1. Call for vote to merge branch-trunk-win to trunk, when
>> > >> > > >>> the work completes and precommit build is clean.
>> > >> > > >>> 2. Start a discussion on adding Jenkins precommit builds on
>> > >> > > >>> windows
>> > >> > and
>> > >> > > >>> how to integrate that with the existing commit process.
>> > >> > > >>>
>> > >> > > >>> Let me know if you have any questions.
>> > >> > > >>>
>> > >> > > >>> Regards,
>> > >> > > >>> Suresh
>> > >> > > >>>
>> > >> > > >>>
>> > >> > > >>
>> > >> > > >>
>> > >> > > >>--
>> > >> > > >>http://hortonworks.com/download/
>> > >> > > >
>> > >> > >
>> > >> >
>> > >> >
>> > >> >
>> > >> > --
>> > >> > http://hortonworks.com/download/
>> > >> >
>> > >>
>> > >>
>> > >>
>> > >> --
>> > >> Todd Lipcon
>> > >> Software Engineer, Cloudera
>> > >>
>> > >>
>> > >>
>> > >>
>> > >>
>> >
>> >
>>
>>
>

RE: [Vote] Merge branch-trunk-win to trunk

[Chuan Liu <ch...@microsoft.com>]

+1 (non-binding)

As someone also contributed to porting Hadoop to Windows, I think Java already provided a very good platform independent platform.
For features that are not available in Java, we will try to provide our platform independent APIs that abstract OS tasks away.
Most features should have no difficulty running on Windows and Linux by using Java and those platform independent APIs.

For concerns raise on new features that may fail on Windows, I think we don't need to require passing on Windows a mandate at the moment. We can simply mark it unavailable to Windows and port it later if the feature is important.

-Chuan

-----Original Message-----
From: Chris Nauroth [mailto:cnauroth@hortonworks.com] 
Sent: Thursday, February 28, 2013 11:51 AM
To: hdfs-dev@hadoop.apache.org
Cc: mapreduce-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org; common-dev@hadoop.apache.org
Subject: Re: [Vote] Merge branch-trunk-win to trunk

> Is there a jira for resolving the outstanding TODOs in the code base 
> (similar to HDFS-2148)?  Looks like this merge doesn't introduce many 
> which is great (just did a quick diff and grep).

I found 2 remaining TODOs introduced in the current merge patch.  One is in ContainerLaunch.java.  The container launch script was trying to set a CLASSPATH that exceeded the Windows maximum command line length.  The fix was to wrap the long classpath into an intermediate jar containing only a manifest file with a Class-Path entry.  (See YARN-316.)  Just to be conservative, we wrapped this logic in an if (Shell.WINDOWS) guard and marked a TODO to remove it later and use that approach on all platforms after additional testing.  I've tested this code path successfully on Mac too, but several people wanted additional testing and performance checks before removing the if (Shell.WINDOWS) guard.  That work is tracked in an existing jira: YARN-358.

The other TODO is for winutils to print more usage information and examples.  At this point, I think winutils is printing sufficient information, and we can just remove the TODO.  I just submitted a new jira to start that conversation: HADOOP-9348.

Thank you,
--Chris


On Thu, Feb 28, 2013 at 11:29 AM, Robert Evans <ev...@yahoo-inc.com> wrote:

> My initial question was mostly intended to understand the desired new 
> classification of Windows after the merge, and how we plan to maintain 
> Windows support.  I am happy to hear that hardware for Jenkins will be 
> provided.  I am also fine, at least initially, with us trying to treat 
> Windows as a first class supported platform.  But I realize that there 
> are a lot of people that do not have easy access to Windows for 
> development/debugging, myself included. I also don't want to slow down 
> the pace of development too much because of this.  It will cause some 
> organizations that do not use or support Windows to be more likely to 
> run software that has diverged from an official release.  It also has 
> the potential to make the patch submission process even more 
> difficult, which increases the likelihood of submitters abandoning 
> patches.  However, the great thing about being in a community is we can change if we need to.
>
> I am +0 for the merge.  I am not a Windows expert so I don't feel 
> comfortable giving it a true +1.
>
> --Bobby
>
>
> On 2/28/13 10:45 AM, "Chris Nauroth" <cn...@hortonworks.com> wrote:
>
> >I'd like to share a few anecdotes about developing cross-platform, 
> >hopefully to address some of the concerns about adding overhead to 
> >the development process.  By reviewing past cases of cross-platform Linux vs.
> >Windows bugs, we can get a sense for how the development process 
> >could look in the future.
> >
> >HADOOP-9131: TestLocalFileSystem#testListStatusWithColons cannot run 
> >on Windows.  As part of an earlier jira, HADOOP-8962, there was a new 
> >test committed on trunk covering the case of a local file system 
> >interaction on a file containing a ':'.  On Windows, ':' in a path 
> >has special meaning as part of the drive specifier (i.e. C:), so this 
> >test cannot pass when running on Windows.  In this kind of case, the 
> >cross-platform bug is obvious, and the fix is obvious 
> >(assumeTrue(!Shell.WINDOWS)).  Ideally, this would get fixed 
> >pre-commit after seeing a -1 from the Windows Jenkins slave.
> >
> >HDFS-4274: BlockPoolSliceScanner does not close verification log 
> >during shutdown.  This caused problems for MiniDFSCluster-based tests 
> >running on Windows.  Failure to close the verification log meant that 
> >we didn't release file locks, so the tests couldn't delete/recreate 
> >working directories during teardown/setup.  Arguably, this was always 
> >a bug, and running on Windows just exposed it because of its stricter 
> >rules about file locking.  This is a more complex fix, but it doesn't 
> >require platform-specific knowledge.  If some future patch 
> >accidentally regresses this, then we'll likely see +1 from Linux 
> >Jenkins and -1 from Windows Jenkins.  Ideally, it would get fixed 
> >pre-commit, because it doesn't require Windows-specific knowledge.  
> >There is also the matter of impact.
> > Re-breaking this would re-break many test suites on Windows.
> >
> >HADOOP-9232: JniBasedUnixGroupsMappingWithFallback fails on Windows 
> >with UnsatisfiedLinkError.  This was introduced by HADOOP-8712, which 
> >switched to JniBasedUnixGroupsMappingWithFallback as the default 
> >hadoop.security.group.mapping, but did not provide a Windows 
> >implementation of the JNI function.  In this case, there was a strong 
> >desire to get
> >HADOOP-8712 into a release, fixing it on Windows required native 
> >Windows API knowledge, and Windows users had a simple workaround 
> >available by changing their configs back to 
> >ShellBasedUnixGroupsMapping.  I think this is the kind of situation 
> >where we could allow HADOOP-8712 to commit despite
> >-1 from Windows Jenkins, with fairly quick follow-up from an engineer 
> >with the Windows expertise to fix it.
> >
> >To summarize, I don't think it needs to differ greatly from our 
> >current development process.  We're all responsible for breadth of 
> >understanding and maintenance of the whole codebase, but we also rely 
> >on specific individuals with deep expertise in particular areas for certain issues.
> > Sometimes we commit despite a -1 from Jenkins, based on the 
> >community's judgment.
> >
> >Virtualization greatly simplifies cross-platform development.  I use 
> >VirtualBox on a Mac host and run VMs for Windows and Ubuntu with a 
> >shared drive so that they can all see the same copy of the source 
> >code.  There are plenty of variations on this depending on your 
> >preference, such as offloading the VMs to a separate server or cloud 
> >service to free up local RAM.  I'm planning on submitting 
> >BUILDING.txt changes later today that fully describe how to build on 
> >Windows.  After some initial setup, it's nearly identical to the mvn 
> >commands that you already use today.
> >
> >Hope this helps,
> >--Chris
> >
> >
> >On Thu, Feb 28, 2013 at 3:25 AM, John Gordon
> ><Jo...@microsoft.com>wrote:
> >
> >> +1 (non-binding)
> >>
> >> I want to share my vote of confidence in this community.  If 
> >>motivated to  do so, this community can keep this project 
> >>cross-platform and continue to  rapidly innovate without breaking a 
> >>sweat.
> >>
> >> The day we started working on this, I saw the foundations of 
> >>greatness in  the quality and volume of dev tests, the code itself, 
> >>and the Apache values  themselves.
> >>
> >> 1.) Hadoop's unit tests and their frameworks are very well thought 
> >>out and  the consideration and energy that went into their design is 
> >>worthy of  praise.  The MiniCluster abstractions utilize very few 
> >>resources and put  all the processes into one JVM for easy 
> >>debugging.  It is very easy to  select specific tests from the full 
> >>suite to reproduce an issue reported in  another environment - like 
> >>the Jenkins build server or another  contributor's environment.
> >> 2.) This community has done an excellent job of incorporating 
> >>well-placed  log messages to make it easy to post mortem 
> >>troubleshoot most failures.
> >>  The logs are very useful, and it is extremely rare that 
> >>troubleshooting a  failure requires debugging a live repro.
> >> 3.) Hadoop is written primarily in Java, a cross-platform language 
> >>that  provides its own platform in the form of the JVM to insulate 
> >>most of the  code from the specifics of the OS layer.
> >> 4.) CoPDoC - The right priorities, and well stated.
> >>
> >>
> >> Thank you,
> >>
> >> John
> >>
> >> -----Original Message-----
> >> From: Ivan Mitic [mailto:ivanmi@microsoft.com]
> >> Sent: Wednesday, February 27, 2013 6:32 PM
> >> To: mapreduce-dev@hadoop.apache.org; common-dev@hadoop.apache.org
> >> Cc: yarn-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org
> >> Subject: RE: [Vote] Merge branch-trunk-win to trunk
> >>
> >> +1 (non-binding)
> >>
> >> I am really glad to see this happening! As people already 
> >>mentioned, this  has been a great engineering effort involving many 
> >>people!
> >>
> >>
> >> Folks raised some valid concerns below and I thought it would be 
> >>good to  share my 2 cents. In my opinion, we don't have to solve all 
> >>these problems  right now. As we move forward with two platforms, we 
> >>can start addressing  one problem at a time and incrementally 
> >>improve. In the first iteration,  maintaining Hadoop on Windows 
> >>could be just everyone trying to do their  best effort (make sure 
> >>Jenkins build succeeds at least). We already have  people who are 
> >>building/running trunk on Windows daily, so they would jump  in and 
> >>fix problems as needed (we've been doing this in branch-trunk-win  
> >>for a while now). Although I see that the problems could arise with  
> >>platform specific features/optimizations, I don't think these are 
> >>frequent,  so in most cases everything will just work. Merging the 
> >>two branches sooner  rather than later does seems like the right 
> >>thing to do if the ultimate  goal is to have Hadoop on both 
> >>platforms. Now that the port has completed,  we will have people in 
> >>Microsoft (and elsewhere) wanting to contribute  
> >>features/improvements to the trunk branch. A separate branch would 
> >>just  make things more difficult and confusing for everyone :) Hope 
> >>this makes  sense.
> >>
> >> -----Original Message-----
> >> From: Todd Lipcon [mailto:todd@cloudera.com]
> >> Sent: Wednesday, February 27, 2013 3:43 PM
> >> To: common-dev@hadoop.apache.org
> >> Cc: yarn-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org; 
> >> mapreduce-dev@hadoop.apache.org
> >> Subject: Re: [Vote] Merge branch-trunk-win to trunk
> >>
> >> On Wed, Feb 27, 2013 at 2:54 PM, Suresh Srinivas <
> suresh@hortonworks.com
> >> >wrote:
> >>
> >> > With that we need to decide how our precommit process looks.
> >> > My inclination is to wait for +1 from precommit builds on both 
> >> > the platforms to ensure no issues are introduced.
> >> > Thoughts?
> >> >
> >> > 2. Feature development impact
> >> > Some questions have been raised about would new features need to 
> >> > be supported on both the platforms. Yes. I do not see a reason 
> >> > why features cannot work on both the platforms, with the 
> >> > exception of platform specific optimizations. This what Java gives us.
> >> >
> >> >
> >> I'm concerned about the above. Personally, I don't have access to 
> >>any  Windows boxes with development tools, and I know nothing about 
> >>developing  on Windows. The only Windows I run is an 8GB VM with 1 
> >>GB RAM allocated,  for powerpoint :)
> >>
> >> If I submit a patch and it gets -1 "tests failed" on the Windows 
> >> slave, how am I supposed to proceed?
> >>
> >> I think a reasonable compromise would be that the tests should 
> >>always
> >> *build* on Windows before commit, and contributors should do their 
> >>best to  look at the test logs for any Windows-specific failures. 
> >>But, beyond  looking at the logs, a "-1 Tests failed on windows" 
> >>should not block a  commit.
> >>
> >> Those contributors who are interested in Windows being a 
> >> first-class platform should be responsible for watching the Windows 
> >> builds and debugging/fixing any regressions that might be Windows-specific.
> >>
> >> I also think the KDE model that Harsh pointed out is an interesting 
> >>one
> >>--
> >> ie the idea that we would not merge windows support to trunk, but 
> >>rather  treat is as a "parallel code line" which lives in the ASF 
> >>and has its own  builds and releases. The windows team would 
> >>periodically merge
> >>trunk->win
> >> to pick up any new changes, and do a separate test/release process. 
> >>I'm not  convinced this is the best idea, but worth discussion of 
> >>pros and cons.
> >>
> >> -Todd
> >>
> >>
> >> >
> >> > On Wed, Feb 27, 2013 at 11:56 AM, Eli Collins <el...@cloudera.com>
> >>wrote:
> >> >
> >> > > Bobby raises some good questions.  A related one, since most 
> >> > > current developers won't add Windows support for new features 
> >> > > that are platform specific is it assumed that Windows 
> >> > > development will either lag or will people actively work on 
> >> > > keeping Windows up with the latest?  And vice versa in case 
> >> > > Windows support is implemented
> >>first.
> >> > >
> >> > > Is there a jira for resolving the outstanding TODOs in the code 
> >> > > base (similar to HDFS-2148)?  Looks like this merge doesn't 
> >> > > introduce many which is great (just did a quick diff and grep).
> >> > >
> >> > > Thanks,
> >> > > Eli
> >> > >
> >> > > On Wed, Feb 27, 2013 at 8:17 AM, Robert Evans 
> >> > > <ev...@yahoo-inc.com>
> >> > wrote:
> >> > > > After this is merged in is Windows still going to be a second 
> >> > > > class citizen but happens to work for more than just 
> >> > > > development or is it a fully supported platform where if 
> >> > > > something breaks it can block a
> >> > > release?
> >> > > >  How do we as a community intend to keep Windows support from
> >> breaking?
> >> > > > We don't have any Jenkins slaves to be able to run nightly 
> >> > > > tests to validate everything still compiles/runs.  This is 
> >> > > > not a blocker for me because we often rely on individuals and 
> >> > > > groups to test Hadoop, but I
> >> > do
> >> > > > think we need to have this discussion before we put it in.
> >> > > >
> >> > > > --Bobby
> >> > > >
> >> > > > On 2/26/13 4:55 PM, "Suresh Srinivas" 
> >> > > > <su...@hortonworks.com>
> >> wrote:
> >> > > >
> >> > > >>I had posted heads up about merging branch-trunk-win to trunk 
> >> > > >>on Feb
> >> > 8th.
> >> > > >>I
> >> > > >>am happy to announce that we are ready for the merge.
> >> > > >>
> >> > > >>Here is a brief recap on the highlights of the work done:
> >> > > >>- Command-line scripts for the Hadoop surface area
> >> > > >>- Mapping the HDFS permissions model to Windows
> >> > > >>- Abstracted and reconciled mismatches around differences in 
> >> > > >>Path semantics in Java and Windows
> >> > > >>- Native Task Controller for Windows
> >> > > >>- Implementation of a Block Placement Policy to support cloud 
> >> > > >>environments, more specifically Azure.
> >> > > >>- Implementation of Hadoop native libraries for Windows 
> >> > > >>(compression codecs, native I/O)
> >> > > >>- Several reliability issues, including race-conditions, 
> >> > > >>intermittent
> >> > > test
> >> > > >>failures, resource leaks.
> >> > > >>- Several new unit test cases written for the above changes
> >> > > >>
> >> > > >>Please find the details of the work in 
> >> > > >>CHANGES.branch-trunk-win.txt - Common 
> >> > > >>changes<http://bit.ly/Xe7Ynv>, HDFS changes<
> >> > http://bit.ly/13QOSo9
> >> > > >,
> >> > > >>and YARN and MapReduce changes <http://bit.ly/128zzMt>. This 
> >> > > >>is the
> >> > work
> >> > > >>ported from branch-1-win to a branch based on trunk.
> >> > > >>
> >> > > >>For details of the testing done, please see the thread - 
> >> > > >>http://bit.ly/WpavJ4. Merge patch for this is available on
> >> > HADOOP-8562<
> >> > > >>https://issues.apache.org/jira/browse/HADOOP-8562>.
> >> > > >>
> >> > > >>This was a large undertaking that involved developing code, 
> >> > > >>testing the entire Hadoop stack, including scale tests. This 
> >> > > >>is made possible only with the contribution from many many 
> >> > > >>folks in the community. Following
> >> > people
> >> > > >>contributed to this work: Ivan Mitic, Chuan Liu, Ramya Sunil, 
> >> > > >>Bikas
> >> > Saha,
> >> > > >>Kanna Karanam, John Gordon, Brandon Li, Chris Nauroth, David 
> >> > > >>Lao,
> >> > > Sumadhur
> >> > > >>Reddy Bolli, Arpit Agarwal, Ahmed El Baz, Mike Liddell, Jing 
> >> > > >>Zhao,
> >> > Thejas
> >> > > >>Nair, Steve Maine, Ganeshan Iyer, Raja Aluri, Giridharan 
> >> > > >>Kesavan, Ramya Bharathi Nimmagadda, Daryn Sharp, Arun Murthy, 
> >> > > >>Tsz-Wo Nicholas Sze,
> >> > > Suresh
> >> > > >>Srinivas and Sanjay Radia. There are many others who 
> >> > > >>contributed as
> >> > well
> >> > > >>providing feedback and comments on numerous jiras.
> >> > > >>
> >> > > >>The vote will run for seven days and will end on March 5, 
> >> > > >>6:00PM
> >>PST.
> >> > > >>
> >> > > >>Regards,
> >> > > >>Suresh
> >> > > >>
> >> > > >>
> >> > > >>
> >> > > >>
> >> > > >>On Thu, Feb 7, 2013 at 6:41 PM, Mahadevan Venkatraman
> >> > > >><ma...@microsoft.com>wrote:
> >> > > >>
> >> > > >>> It is super exciting to look at the prospect of these 
> >> > > >>>changes being merged  to trunk. Having Windows as one of the 
> >> > > >>>supported Hadoop platforms is
> >> > a
> >> > > >>> fantastic opportunity both for the Hadoop project and 
> >> > > >>>Microsoft customers.
> >> > > >>>
> >> > > >>> This work began around a year back when a few of us started 
> >> > > >>> with a
> >> > > basic
> >> > > >>> port of Hadoop on Windows. Ever since, the Hadoop team in 
> >> > > >>> Microsoft
> >> > > have
> >> > > >>> made significant progress in the following areas:
> >> > > >>> (PS: Some of these items are already included in Suresh's 
> >> > > >>> email, but including again for completeness)
> >> > > >>>
> >> > > >>> - Command-line scripts for the Hadoop surface area
> >> > > >>> - Mapping the HDFS permissions model to Windows
> >> > > >>> - Abstracted and reconciled mismatches around differences 
> >> > > >>> in Path semantics in Java and Windows
> >> > > >>> - Native Task Controller for Windows
> >> > > >>> - Implementation of a Block Placement Policy to support 
> >> > > >>> cloud environments, more specifically Azure.
> >> > > >>> - Implementation of Hadoop native libraries for Windows 
> >> > > >>> (compression codecs, native I/O) - Several reliability 
> >> > > >>> issues, including race-conditions, intermittent test 
> >> > > >>> failures, resource
> >> leaks.
> >> > > >>> - Several new unit test cases written for the above changes
> >> > > >>>
> >> > > >>> In the process, we have closely engaged with the Apache 
> >> > > >>> open source community and have got great support and 
> >> > > >>> assistance from the
> >> > community
> >> > > >>>in
> >> > > >>> terms of contributing fixes, code review comments and commits.
> >> > > >>>
> >> > > >>> In addition, the Hadoop team at Microsoft has also made 
> >> > > >>> good progress
> >> > > in
> >> > > >>> other projects including Hive, Pig, Sqoop, Oozie, HCat and
> >>HBase.
> >> > Many
> >> > > >>>of
> >> > > >>> these changes have already been committed to the respective 
> >> > > >>>trunks
> >> > with
> >> > > >>> help from various committers and contributors. It is great 
> >> > > >>> to see the commitment of the community to support multiple 
> >> > > >>> platforms, and we
> >> > look
> >> > > >>> forward to the day when a developer/customer is able to 
> >> > > >>>successfully deploy  a complete solution stack based on 
> >> > > >>>Apache Hadoop releases.
> >> > > >>>
> >> > > >>> Next Steps:
> >> > > >>>
> >> > > >>> All of the above changes are part of the Windows Azure 
> >> > > >>>HDInsight and  HDInsight Server products from Microsoft. We 
> >> > > >>>have successfully on-boarded  several internal customers and 
> >> > > >>>have been running production workloads
> >> > > on
> >> > > >>> Windows Azure HDInsight. Our vision is to create a big data 
> >> > > >>>platform based  on Hadoop, and we are committed to helping 
> >> > > >>>make Hadoop a world-class  solution that anyone can use to 
> >> > > >>>solve their biggest data challenges.
> >> > > >>>
> >> > > >>> As an immediate next step, we would like to have a 
> >> > > >>> discussion around
> >> > > how
> >> > > >>> we can ensure that the quality of the mainline Hadoop 
> >> > > >>>branches on Windows  is maintained. To this end, we would 
> >> > > >>>like to get to the state where
> >> > we
> >> > > >>>have
> >> > > >>> pre-checkin validation gates and nightly test runs enabled 
> >> > > >>>on
> >> > Windows.
> >> > > >>>If
> >> > > >>> you have any suggestions around this, please do send an email.
> >> > > >>>We
> >> > are
> >> > > >>> committed to helping sustain the long-term quality of 
> >> > > >>>Hadoop on both Linux  and Windows.
> >> > > >>>
> >> > > >>> We sincerely thank the community for their contribution and 
> >> > > >>> support
> >> > so
> >> > > >>> far. And hope to continue having a close engagement in the
> >>future.
> >> > > >>>
> >> > > >>> -Microsoft HDInsight Team
> >> > > >>>
> >> > > >>>
> >> > > >>> -----Original Message-----
> >> > > >>> From: Suresh Srinivas [mailto:suresh@hortonworks.com]
> >> > > >>> Sent: Thursday, February 7, 2013 5:42 PM
> >> > > >>> To: common-dev@hadoop.apache.org; 
> >> > > >>> yarn-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org; 
> >> > > >>> mapreduce-dev@hadoop.apache.org
> >> > > >>> Subject: Heads up - merge branch-trunk-win to trunk
> >> > > >>>
> >> > > >>> The support for Hadoop on Windows was proposed in 
> >> > > >>> HADOOP-8079< 
> >> > > >>> https://issues.apache.org/jira/browse/HADOOP-8079> almost a
> year
> >> > ago.
> >> > > >>>The
> >> > > >>> goal was to make Hadoop natively integrated, full-featured, 
> >> > > >>>and performance  and scalability tuned on Windows Server or 
> >> > > >>>Windows Azure.
> >> > > >>> We are happy to announce that a lot of progress has been 
> >> > > >>>made in this  regard.
> >> > > >>>
> >> > > >>> Initial work started in a feature branch, branch-1-win, 
> >> > > >>>based on branch-1.
> >> > > >>> The details related to the work done in the branch can be 
> >> > > >>>seen in  CHANGES.txt<
> >> > > >>>
> >> > > >>>
> >> > >
> http://svn.apache.org/viewvc/hadoop/common/branches/branch-1-win/CHA
> >> > > NGES
> >> > .
> >> > > >>>branch-1-win.txt?view=markup
> >> > > >>> >.
> >> > > >>> This work has been ported to a branch, branch-trunk-win, 
> >> > > >>> based on
> >> > > trunk.
> >> > > >>> Merge patch for this is available on 
> >> > > >>> HADOOP-8562<https://issues.apache.org/jira/browse/HADOOP-85
> >> > > >>> 62>
> >> > > >>> .
> >> > > >>>
> >> > > >>> Highlights of the work done so far:
> >> > > >>> 1. Necessary changes in Hadoop to run natively on Windows. 
> >> > > >>> These
> >> > > changes
> >> > > >>> handle differences in platforms related to path names, 
> >> > > >>>process/task  management etc.
> >> > > >>> 2. Addition of winutils tools for managing file permissions 
> >> > > >>>and ownership,  user group mapping, hardlinks, symbolic 
> >> > > >>>links, chmod, disk
> >> > utilization,
> >> > > >>>and
> >> > > >>> process/task management.
> >> > > >>> 3. Added cmd scripts equivalent to existing shell scripts 
> >> > > >>>hadoop-daemon.sh, start and stop scripts.
> >> > > >>> 4. Addition of block placement policy implemnation to 
> >> > > >>>support cloud  enviroment, more specifically Azure.
> >> > > >>>
> >> > > >>> We are very close to wrapping up the work in 
> >> > > >>>branch-trunk-win and getting  ready for a merge. Currently 
> >> > > >>>the merge patch is passing close to 100%
> >> > > of
> >> > > >>> unit tests on Linux. Soon I will call for a vote to merge 
> >> > > >>>this branch into  trunk.
> >> > > >>>
> >> > > >>> Next steps:
> >> > > >>> 1. Call for vote to merge branch-trunk-win to trunk, when 
> >> > > >>> the work completes and precommit build is clean.
> >> > > >>> 2. Start a discussion on adding Jenkins precommit builds on 
> >> > > >>> windows
> >> > and
> >> > > >>> how to integrate that with the existing commit process.
> >> > > >>>
> >> > > >>> Let me know if you have any questions.
> >> > > >>>
> >> > > >>> Regards,
> >> > > >>> Suresh
> >> > > >>>
> >> > > >>>
> >> > > >>
> >> > > >>
> >> > > >>--
> >> > > >>http://hortonworks.com/download/
> >> > > >
> >> > >
> >> >
> >> >
> >> >
> >> > --
> >> > http://hortonworks.com/download/
> >> >
> >>
> >>
> >>
> >> --
> >> Todd Lipcon
> >> Software Engineer, Cloudera
> >>
> >>
> >>
> >>
> >>
>
>

Re: [Vote] Merge branch-trunk-win to trunk

[Chris Nauroth <cn...@hortonworks.com>]

> Is there a jira for resolving the outstanding TODOs in the code base
> (similar to HDFS-2148)?  Looks like this merge doesn't introduce many
> which is great (just did a quick diff and grep).

I found 2 remaining TODOs introduced in the current merge patch.  One is in
ContainerLaunch.java.  The container launch script was trying to set a
CLASSPATH that exceeded the Windows maximum command line length.  The fix
was to wrap the long classpath into an intermediate jar containing only a
manifest file with a Class-Path entry.  (See YARN-316.)  Just to be
conservative, we wrapped this logic in an if (Shell.WINDOWS) guard and
marked a TODO to remove it later and use that approach on all platforms
after additional testing.  I've tested this code path successfully on Mac
too, but several people wanted additional testing and performance checks
before removing the if (Shell.WINDOWS) guard.  That work is tracked in an
existing jira: YARN-358.

The other TODO is for winutils to print more usage information and
examples.  At this point, I think winutils is printing sufficient
information, and we can just remove the TODO.  I just submitted a new jira
to start that conversation: HADOOP-9348.

Thank you,
--Chris


On Thu, Feb 28, 2013 at 11:29 AM, Robert Evans <ev...@yahoo-inc.com> wrote:

> My initial question was mostly intended to understand the desired new
> classification of Windows after the merge, and how we plan to maintain
> Windows support.  I am happy to hear that hardware for Jenkins will be
> provided.  I am also fine, at least initially, with us trying to treat
> Windows as a first class supported platform.  But I realize that there are
> a lot of people that do not have easy access to Windows for
> development/debugging, myself included. I also don't want to slow down the
> pace of development too much because of this.  It will cause some
> organizations that do not use or support Windows to be more likely to run
> software that has diverged from an official release.  It also has the
> potential to make the patch submission process even more difficult, which
> increases the likelihood of submitters abandoning patches.  However, the
> great thing about being in a community is we can change if we need to.
>
> I am +0 for the merge.  I am not a Windows expert so I don't feel
> comfortable giving it a true +1.
>
> --Bobby
>
>
> On 2/28/13 10:45 AM, "Chris Nauroth" <cn...@hortonworks.com> wrote:
>
> >I'd like to share a few anecdotes about developing cross-platform,
> >hopefully to address some of the concerns about adding overhead to the
> >development process.  By reviewing past cases of cross-platform Linux vs.
> >Windows bugs, we can get a sense for how the development process could
> >look
> >in the future.
> >
> >HADOOP-9131: TestLocalFileSystem#testListStatusWithColons cannot run on
> >Windows.  As part of an earlier jira, HADOOP-8962, there was a new test
> >committed on trunk covering the case of a local file system interaction on
> >a file containing a ':'.  On Windows, ':' in a path has special meaning as
> >part of the drive specifier (i.e. C:), so this test cannot pass when
> >running on Windows.  In this kind of case, the cross-platform bug is
> >obvious, and the fix is obvious (assumeTrue(!Shell.WINDOWS)).  Ideally,
> >this would get fixed pre-commit after seeing a -1 from the Windows Jenkins
> >slave.
> >
> >HDFS-4274: BlockPoolSliceScanner does not close verification log during
> >shutdown.  This caused problems for MiniDFSCluster-based tests running on
> >Windows.  Failure to close the verification log meant that we didn't
> >release file locks, so the tests couldn't delete/recreate working
> >directories during teardown/setup.  Arguably, this was always a bug, and
> >running on Windows just exposed it because of its stricter rules about
> >file
> >locking.  This is a more complex fix, but it doesn't require
> >platform-specific knowledge.  If some future patch accidentally regresses
> >this, then we'll likely see +1 from Linux Jenkins and -1 from Windows
> >Jenkins.  Ideally, it would get fixed pre-commit, because it doesn't
> >require Windows-specific knowledge.  There is also the matter of impact.
> > Re-breaking this would re-break many test suites on Windows.
> >
> >HADOOP-9232: JniBasedUnixGroupsMappingWithFallback fails on Windows with
> >UnsatisfiedLinkError.  This was introduced by HADOOP-8712, which switched
> >to JniBasedUnixGroupsMappingWithFallback as the default
> >hadoop.security.group.mapping, but did not provide a Windows
> >implementation
> >of the JNI function.  In this case, there was a strong desire to get
> >HADOOP-8712 into a release, fixing it on Windows required native Windows
> >API knowledge, and Windows users had a simple workaround available by
> >changing their configs back to ShellBasedUnixGroupsMapping.  I think this
> >is the kind of situation where we could allow HADOOP-8712 to commit
> >despite
> >-1 from Windows Jenkins, with fairly quick follow-up from an engineer with
> >the Windows expertise to fix it.
> >
> >To summarize, I don't think it needs to differ greatly from our current
> >development process.  We're all responsible for breadth of understanding
> >and maintenance of the whole codebase, but we also rely on specific
> >individuals with deep expertise in particular areas for certain issues.
> > Sometimes we commit despite a -1 from Jenkins, based on the community's
> >judgment.
> >
> >Virtualization greatly simplifies cross-platform development.  I use
> >VirtualBox on a Mac host and run VMs for Windows and Ubuntu with a shared
> >drive so that they can all see the same copy of the source code.  There
> >are
> >plenty of variations on this depending on your preference, such as
> >offloading the VMs to a separate server or cloud service to free up local
> >RAM.  I'm planning on submitting BUILDING.txt changes later today that
> >fully describe how to build on Windows.  After some initial setup, it's
> >nearly identical to the mvn commands that you already use today.
> >
> >Hope this helps,
> >--Chris
> >
> >
> >On Thu, Feb 28, 2013 at 3:25 AM, John Gordon
> ><Jo...@microsoft.com>wrote:
> >
> >> +1 (non-binding)
> >>
> >> I want to share my vote of confidence in this community.  If motivated
> >>to
> >> do so, this community can keep this project cross-platform and continue
> >>to
> >> rapidly innovate without breaking a sweat.
> >>
> >> The day we started working on this, I saw the foundations of greatness
> >>in
> >> the quality and volume of dev tests, the code itself, and the Apache
> >>values
> >> themselves.
> >>
> >> 1.) Hadoop's unit tests and their frameworks are very well thought out
> >>and
> >> the consideration and energy that went into their design is worthy of
> >> praise.  The MiniCluster abstractions utilize very few resources and put
> >> all the processes into one JVM for easy debugging.  It is very easy to
> >> select specific tests from the full suite to reproduce an issue
> >>reported in
> >> another environment - like the Jenkins build server or another
> >> contributor's environment.
> >> 2.) This community has done an excellent job of incorporating
> >>well-placed
> >> log messages to make it easy to post mortem troubleshoot most failures.
> >>  The logs are very useful, and it is extremely rare that
> >>troubleshooting a
> >> failure requires debugging a live repro.
> >> 3.) Hadoop is written primarily in Java, a cross-platform language that
> >> provides its own platform in the form of the JVM to insulate most of the
> >> code from the specifics of the OS layer.
> >> 4.) CoPDoC - The right priorities, and well stated.
> >>
> >>
> >> Thank you,
> >>
> >> John
> >>
> >> -----Original Message-----
> >> From: Ivan Mitic [mailto:ivanmi@microsoft.com]
> >> Sent: Wednesday, February 27, 2013 6:32 PM
> >> To: mapreduce-dev@hadoop.apache.org; common-dev@hadoop.apache.org
> >> Cc: yarn-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org
> >> Subject: RE: [Vote] Merge branch-trunk-win to trunk
> >>
> >> +1 (non-binding)
> >>
> >> I am really glad to see this happening! As people already mentioned,
> >>this
> >> has been a great engineering effort involving many people!
> >>
> >>
> >> Folks raised some valid concerns below and I thought it would be good to
> >> share my 2 cents. In my opinion, we don't have to solve all these
> >>problems
> >> right now. As we move forward with two platforms, we can start
> >>addressing
> >> one problem at a time and incrementally improve. In the first iteration,
> >> maintaining Hadoop on Windows could be just everyone trying to do their
> >> best effort (make sure Jenkins build succeeds at least). We already have
> >> people who are building/running trunk on Windows daily, so they would
> >>jump
> >> in and fix problems as needed (we've been doing this in branch-trunk-win
> >> for a while now). Although I see that the problems could arise with
> >> platform specific features/optimizations, I don't think these are
> >>frequent,
> >> so in most cases everything will just work. Merging the two branches
> >>sooner
> >> rather than later does seems like the right thing to do if the ultimate
> >> goal is to have Hadoop on both platforms. Now that the port has
> >>completed,
> >> we will have people in Microsoft (and elsewhere) wanting to contribute
> >> features/improvements to the trunk branch. A separate branch would just
> >> make things more difficult and confusing for everyone :) Hope this makes
> >> sense.
> >>
> >> -----Original Message-----
> >> From: Todd Lipcon [mailto:todd@cloudera.com]
> >> Sent: Wednesday, February 27, 2013 3:43 PM
> >> To: common-dev@hadoop.apache.org
> >> Cc: yarn-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org;
> >> mapreduce-dev@hadoop.apache.org
> >> Subject: Re: [Vote] Merge branch-trunk-win to trunk
> >>
> >> On Wed, Feb 27, 2013 at 2:54 PM, Suresh Srinivas <
> suresh@hortonworks.com
> >> >wrote:
> >>
> >> > With that we need to decide how our precommit process looks.
> >> > My inclination is to wait for +1 from precommit builds on both the
> >> > platforms to ensure no issues are introduced.
> >> > Thoughts?
> >> >
> >> > 2. Feature development impact
> >> > Some questions have been raised about would new features need to be
> >> > supported on both the platforms. Yes. I do not see a reason why
> >> > features cannot work on both the platforms, with the exception of
> >> > platform specific optimizations. This what Java gives us.
> >> >
> >> >
> >> I'm concerned about the above. Personally, I don't have access to any
> >> Windows boxes with development tools, and I know nothing about
> >>developing
> >> on Windows. The only Windows I run is an 8GB VM with 1 GB RAM allocated,
> >> for powerpoint :)
> >>
> >> If I submit a patch and it gets -1 "tests failed" on the Windows slave,
> >> how am I supposed to proceed?
> >>
> >> I think a reasonable compromise would be that the tests should always
> >> *build* on Windows before commit, and contributors should do their best
> >>to
> >> look at the test logs for any Windows-specific failures. But, beyond
> >> looking at the logs, a "-1 Tests failed on windows" should not block a
> >> commit.
> >>
> >> Those contributors who are interested in Windows being a first-class
> >> platform should be responsible for watching the Windows builds and
> >> debugging/fixing any regressions that might be Windows-specific.
> >>
> >> I also think the KDE model that Harsh pointed out is an interesting one
> >>--
> >> ie the idea that we would not merge windows support to trunk, but rather
> >> treat is as a "parallel code line" which lives in the ASF and has its
> >>own
> >> builds and releases. The windows team would periodically merge
> >>trunk->win
> >> to pick up any new changes, and do a separate test/release process. I'm
> >>not
> >> convinced this is the best idea, but worth discussion of pros and cons.
> >>
> >> -Todd
> >>
> >>
> >> >
> >> > On Wed, Feb 27, 2013 at 11:56 AM, Eli Collins <el...@cloudera.com>
> >>wrote:
> >> >
> >> > > Bobby raises some good questions.  A related one, since most current
> >> > > developers won't add Windows support for new features that are
> >> > > platform specific is it assumed that Windows development will either
> >> > > lag or will people actively work on keeping Windows up with the
> >> > > latest?  And vice versa in case Windows support is implemented
> >>first.
> >> > >
> >> > > Is there a jira for resolving the outstanding TODOs in the code base
> >> > > (similar to HDFS-2148)?  Looks like this merge doesn't introduce
> >> > > many which is great (just did a quick diff and grep).
> >> > >
> >> > > Thanks,
> >> > > Eli
> >> > >
> >> > > On Wed, Feb 27, 2013 at 8:17 AM, Robert Evans <ev...@yahoo-inc.com>
> >> > wrote:
> >> > > > After this is merged in is Windows still going to be a second
> >> > > > class citizen but happens to work for more than just development
> >> > > > or is it a fully supported platform where if something breaks it
> >> > > > can block a
> >> > > release?
> >> > > >  How do we as a community intend to keep Windows support from
> >> breaking?
> >> > > > We don't have any Jenkins slaves to be able to run nightly tests
> >> > > > to validate everything still compiles/runs.  This is not a blocker
> >> > > > for me because we often rely on individuals and groups to test
> >> > > > Hadoop, but I
> >> > do
> >> > > > think we need to have this discussion before we put it in.
> >> > > >
> >> > > > --Bobby
> >> > > >
> >> > > > On 2/26/13 4:55 PM, "Suresh Srinivas" <su...@hortonworks.com>
> >> wrote:
> >> > > >
> >> > > >>I had posted heads up about merging branch-trunk-win to trunk on
> >> > > >>Feb
> >> > 8th.
> >> > > >>I
> >> > > >>am happy to announce that we are ready for the merge.
> >> > > >>
> >> > > >>Here is a brief recap on the highlights of the work done:
> >> > > >>- Command-line scripts for the Hadoop surface area
> >> > > >>- Mapping the HDFS permissions model to Windows
> >> > > >>- Abstracted and reconciled mismatches around differences in Path
> >> > > >>semantics in Java and Windows
> >> > > >>- Native Task Controller for Windows
> >> > > >>- Implementation of a Block Placement Policy to support cloud
> >> > > >>environments, more specifically Azure.
> >> > > >>- Implementation of Hadoop native libraries for Windows
> >> > > >>(compression codecs, native I/O)
> >> > > >>- Several reliability issues, including race-conditions,
> >> > > >>intermittent
> >> > > test
> >> > > >>failures, resource leaks.
> >> > > >>- Several new unit test cases written for the above changes
> >> > > >>
> >> > > >>Please find the details of the work in
> >> > > >>CHANGES.branch-trunk-win.txt - Common
> >> > > >>changes<http://bit.ly/Xe7Ynv>, HDFS changes<
> >> > http://bit.ly/13QOSo9
> >> > > >,
> >> > > >>and YARN and MapReduce changes <http://bit.ly/128zzMt>. This is
> >> > > >>the
> >> > work
> >> > > >>ported from branch-1-win to a branch based on trunk.
> >> > > >>
> >> > > >>For details of the testing done, please see the thread -
> >> > > >>http://bit.ly/WpavJ4. Merge patch for this is available on
> >> > HADOOP-8562<
> >> > > >>https://issues.apache.org/jira/browse/HADOOP-8562>.
> >> > > >>
> >> > > >>This was a large undertaking that involved developing code,
> >> > > >>testing the entire Hadoop stack, including scale tests. This is
> >> > > >>made possible only with the contribution from many many folks in
> >> > > >>the community. Following
> >> > people
> >> > > >>contributed to this work: Ivan Mitic, Chuan Liu, Ramya Sunil,
> >> > > >>Bikas
> >> > Saha,
> >> > > >>Kanna Karanam, John Gordon, Brandon Li, Chris Nauroth, David Lao,
> >> > > Sumadhur
> >> > > >>Reddy Bolli, Arpit Agarwal, Ahmed El Baz, Mike Liddell, Jing Zhao,
> >> > Thejas
> >> > > >>Nair, Steve Maine, Ganeshan Iyer, Raja Aluri, Giridharan Kesavan,
> >> > > >>Ramya Bharathi Nimmagadda, Daryn Sharp, Arun Murthy, Tsz-Wo
> >> > > >>Nicholas Sze,
> >> > > Suresh
> >> > > >>Srinivas and Sanjay Radia. There are many others who contributed
> >> > > >>as
> >> > well
> >> > > >>providing feedback and comments on numerous jiras.
> >> > > >>
> >> > > >>The vote will run for seven days and will end on March 5, 6:00PM
> >>PST.
> >> > > >>
> >> > > >>Regards,
> >> > > >>Suresh
> >> > > >>
> >> > > >>
> >> > > >>
> >> > > >>
> >> > > >>On Thu, Feb 7, 2013 at 6:41 PM, Mahadevan Venkatraman
> >> > > >><ma...@microsoft.com>wrote:
> >> > > >>
> >> > > >>> It is super exciting to look at the prospect of these changes
> >> > > >>>being merged  to trunk. Having Windows as one of the supported
> >> > > >>>Hadoop platforms is
> >> > a
> >> > > >>> fantastic opportunity both for the Hadoop project and Microsoft
> >> > > >>>customers.
> >> > > >>>
> >> > > >>> This work began around a year back when a few of us started with
> >> > > >>> a
> >> > > basic
> >> > > >>> port of Hadoop on Windows. Ever since, the Hadoop team in
> >> > > >>> Microsoft
> >> > > have
> >> > > >>> made significant progress in the following areas:
> >> > > >>> (PS: Some of these items are already included in Suresh's email,
> >> > > >>> but including again for completeness)
> >> > > >>>
> >> > > >>> - Command-line scripts for the Hadoop surface area
> >> > > >>> - Mapping the HDFS permissions model to Windows
> >> > > >>> - Abstracted and reconciled mismatches around differences in
> >> > > >>> Path semantics in Java and Windows
> >> > > >>> - Native Task Controller for Windows
> >> > > >>> - Implementation of a Block Placement Policy to support cloud
> >> > > >>> environments, more specifically Azure.
> >> > > >>> - Implementation of Hadoop native libraries for Windows
> >> > > >>> (compression codecs, native I/O) - Several reliability issues,
> >> > > >>> including race-conditions, intermittent test failures, resource
> >> leaks.
> >> > > >>> - Several new unit test cases written for the above changes
> >> > > >>>
> >> > > >>> In the process, we have closely engaged with the Apache open
> >> > > >>> source community and have got great support and assistance from
> >> > > >>> the
> >> > community
> >> > > >>>in
> >> > > >>> terms of contributing fixes, code review comments and commits.
> >> > > >>>
> >> > > >>> In addition, the Hadoop team at Microsoft has also made good
> >> > > >>> progress
> >> > > in
> >> > > >>> other projects including Hive, Pig, Sqoop, Oozie, HCat and
> >>HBase.
> >> > Many
> >> > > >>>of
> >> > > >>> these changes have already been committed to the respective
> >> > > >>>trunks
> >> > with
> >> > > >>> help from various committers and contributors. It is great to
> >> > > >>> see the commitment of the community to support multiple
> >> > > >>> platforms, and we
> >> > look
> >> > > >>> forward to the day when a developer/customer is able to
> >> > > >>>successfully deploy  a complete solution stack based on Apache
> >> > > >>>Hadoop releases.
> >> > > >>>
> >> > > >>> Next Steps:
> >> > > >>>
> >> > > >>> All of the above changes are part of the Windows Azure HDInsight
> >> > > >>>and  HDInsight Server products from Microsoft. We have
> >> > > >>>successfully on-boarded  several internal customers and have been
> >> > > >>>running production workloads
> >> > > on
> >> > > >>> Windows Azure HDInsight. Our vision is to create a big data
> >> > > >>>platform based  on Hadoop, and we are committed to helping make
> >> > > >>>Hadoop a world-class  solution that anyone can use to solve their
> >> > > >>>biggest data challenges.
> >> > > >>>
> >> > > >>> As an immediate next step, we would like to have a discussion
> >> > > >>> around
> >> > > how
> >> > > >>> we can ensure that the quality of the mainline Hadoop branches
> >> > > >>>on Windows  is maintained. To this end, we would like to get to
> >> > > >>>the state where
> >> > we
> >> > > >>>have
> >> > > >>> pre-checkin validation gates and nightly test runs enabled on
> >> > Windows.
> >> > > >>>If
> >> > > >>> you have any suggestions around this, please do send an email.
> >> > > >>>We
> >> > are
> >> > > >>> committed to helping sustain the long-term quality of Hadoop on
> >> > > >>>both Linux  and Windows.
> >> > > >>>
> >> > > >>> We sincerely thank the community for their contribution and
> >> > > >>> support
> >> > so
> >> > > >>> far. And hope to continue having a close engagement in the
> >>future.
> >> > > >>>
> >> > > >>> -Microsoft HDInsight Team
> >> > > >>>
> >> > > >>>
> >> > > >>> -----Original Message-----
> >> > > >>> From: Suresh Srinivas [mailto:suresh@hortonworks.com]
> >> > > >>> Sent: Thursday, February 7, 2013 5:42 PM
> >> > > >>> To: common-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org;
> >> > > >>> hdfs-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
> >> > > >>> Subject: Heads up - merge branch-trunk-win to trunk
> >> > > >>>
> >> > > >>> The support for Hadoop on Windows was proposed in HADOOP-8079<
> >> > > >>> https://issues.apache.org/jira/browse/HADOOP-8079> almost a
> year
> >> > ago.
> >> > > >>>The
> >> > > >>> goal was to make Hadoop natively integrated, full-featured, and
> >> > > >>>performance  and scalability tuned on Windows Server or Windows
> >> > > >>>Azure.
> >> > > >>> We are happy to announce that a lot of progress has been made in
> >> > > >>>this  regard.
> >> > > >>>
> >> > > >>> Initial work started in a feature branch, branch-1-win, based on
> >> > > >>>branch-1.
> >> > > >>> The details related to the work done in the branch can be seen
> >> > > >>>in  CHANGES.txt<
> >> > > >>>
> >> > > >>>
> >> > >
> http://svn.apache.org/viewvc/hadoop/common/branches/branch-1-win/CHA
> >> > > NGES
> >> > .
> >> > > >>>branch-1-win.txt?view=markup
> >> > > >>> >.
> >> > > >>> This work has been ported to a branch, branch-trunk-win, based
> >> > > >>> on
> >> > > trunk.
> >> > > >>> Merge patch for this is available on
> >> > > >>> HADOOP-8562<https://issues.apache.org/jira/browse/HADOOP-8562>
> >> > > >>> .
> >> > > >>>
> >> > > >>> Highlights of the work done so far:
> >> > > >>> 1. Necessary changes in Hadoop to run natively on Windows. These
> >> > > changes
> >> > > >>> handle differences in platforms related to path names,
> >> > > >>>process/task  management etc.
> >> > > >>> 2. Addition of winutils tools for managing file permissions and
> >> > > >>>ownership,  user group mapping, hardlinks, symbolic links, chmod,
> >> > > >>>disk
> >> > utilization,
> >> > > >>>and
> >> > > >>> process/task management.
> >> > > >>> 3. Added cmd scripts equivalent to existing shell scripts
> >> > > >>>hadoop-daemon.sh, start and stop scripts.
> >> > > >>> 4. Addition of block placement policy implemnation to support
> >> > > >>>cloud  enviroment, more specifically Azure.
> >> > > >>>
> >> > > >>> We are very close to wrapping up the work in branch-trunk-win
> >> > > >>>and getting  ready for a merge. Currently the merge patch is
> >> > > >>>passing close to 100%
> >> > > of
> >> > > >>> unit tests on Linux. Soon I will call for a vote to merge this
> >> > > >>>branch into  trunk.
> >> > > >>>
> >> > > >>> Next steps:
> >> > > >>> 1. Call for vote to merge branch-trunk-win to trunk, when the
> >> > > >>> work completes and precommit build is clean.
> >> > > >>> 2. Start a discussion on adding Jenkins precommit builds on
> >> > > >>> windows
> >> > and
> >> > > >>> how to integrate that with the existing commit process.
> >> > > >>>
> >> > > >>> Let me know if you have any questions.
> >> > > >>>
> >> > > >>> Regards,
> >> > > >>> Suresh
> >> > > >>>
> >> > > >>>
> >> > > >>
> >> > > >>
> >> > > >>--
> >> > > >>http://hortonworks.com/download/
> >> > > >
> >> > >
> >> >
> >> >
> >> >
> >> > --
> >> > http://hortonworks.com/download/
> >> >
> >>
> >>
> >>
> >> --
> >> Todd Lipcon
> >> Software Engineer, Cloudera
> >>
> >>
> >>
> >>
> >>
>
>

Re: [Vote] Merge branch-trunk-win to trunk

[Chris Nauroth <cn...@hortonworks.com>]

> Is there a jira for resolving the outstanding TODOs in the code base
> (similar to HDFS-2148)?  Looks like this merge doesn't introduce many
> which is great (just did a quick diff and grep).

I found 2 remaining TODOs introduced in the current merge patch.  One is in
ContainerLaunch.java.  The container launch script was trying to set a
CLASSPATH that exceeded the Windows maximum command line length.  The fix
was to wrap the long classpath into an intermediate jar containing only a
manifest file with a Class-Path entry.  (See YARN-316.)  Just to be
conservative, we wrapped this logic in an if (Shell.WINDOWS) guard and
marked a TODO to remove it later and use that approach on all platforms
after additional testing.  I've tested this code path successfully on Mac
too, but several people wanted additional testing and performance checks
before removing the if (Shell.WINDOWS) guard.  That work is tracked in an
existing jira: YARN-358.

The other TODO is for winutils to print more usage information and
examples.  At this point, I think winutils is printing sufficient
information, and we can just remove the TODO.  I just submitted a new jira
to start that conversation: HADOOP-9348.

Thank you,
--Chris


On Thu, Feb 28, 2013 at 11:29 AM, Robert Evans <ev...@yahoo-inc.com> wrote:

> My initial question was mostly intended to understand the desired new
> classification of Windows after the merge, and how we plan to maintain
> Windows support.  I am happy to hear that hardware for Jenkins will be
> provided.  I am also fine, at least initially, with us trying to treat
> Windows as a first class supported platform.  But I realize that there are
> a lot of people that do not have easy access to Windows for
> development/debugging, myself included. I also don't want to slow down the
> pace of development too much because of this.  It will cause some
> organizations that do not use or support Windows to be more likely to run
> software that has diverged from an official release.  It also has the
> potential to make the patch submission process even more difficult, which
> increases the likelihood of submitters abandoning patches.  However, the
> great thing about being in a community is we can change if we need to.
>
> I am +0 for the merge.  I am not a Windows expert so I don't feel
> comfortable giving it a true +1.
>
> --Bobby
>
>
> On 2/28/13 10:45 AM, "Chris Nauroth" <cn...@hortonworks.com> wrote:
>
> >I'd like to share a few anecdotes about developing cross-platform,
> >hopefully to address some of the concerns about adding overhead to the
> >development process.  By reviewing past cases of cross-platform Linux vs.
> >Windows bugs, we can get a sense for how the development process could
> >look
> >in the future.
> >
> >HADOOP-9131: TestLocalFileSystem#testListStatusWithColons cannot run on
> >Windows.  As part of an earlier jira, HADOOP-8962, there was a new test
> >committed on trunk covering the case of a local file system interaction on
> >a file containing a ':'.  On Windows, ':' in a path has special meaning as
> >part of the drive specifier (i.e. C:), so this test cannot pass when
> >running on Windows.  In this kind of case, the cross-platform bug is
> >obvious, and the fix is obvious (assumeTrue(!Shell.WINDOWS)).  Ideally,
> >this would get fixed pre-commit after seeing a -1 from the Windows Jenkins
> >slave.
> >
> >HDFS-4274: BlockPoolSliceScanner does not close verification log during
> >shutdown.  This caused problems for MiniDFSCluster-based tests running on
> >Windows.  Failure to close the verification log meant that we didn't
> >release file locks, so the tests couldn't delete/recreate working
> >directories during teardown/setup.  Arguably, this was always a bug, and
> >running on Windows just exposed it because of its stricter rules about
> >file
> >locking.  This is a more complex fix, but it doesn't require
> >platform-specific knowledge.  If some future patch accidentally regresses
> >this, then we'll likely see +1 from Linux Jenkins and -1 from Windows
> >Jenkins.  Ideally, it would get fixed pre-commit, because it doesn't
> >require Windows-specific knowledge.  There is also the matter of impact.
> > Re-breaking this would re-break many test suites on Windows.
> >
> >HADOOP-9232: JniBasedUnixGroupsMappingWithFallback fails on Windows with
> >UnsatisfiedLinkError.  This was introduced by HADOOP-8712, which switched
> >to JniBasedUnixGroupsMappingWithFallback as the default
> >hadoop.security.group.mapping, but did not provide a Windows
> >implementation
> >of the JNI function.  In this case, there was a strong desire to get
> >HADOOP-8712 into a release, fixing it on Windows required native Windows
> >API knowledge, and Windows users had a simple workaround available by
> >changing their configs back to ShellBasedUnixGroupsMapping.  I think this
> >is the kind of situation where we could allow HADOOP-8712 to commit
> >despite
> >-1 from Windows Jenkins, with fairly quick follow-up from an engineer with
> >the Windows expertise to fix it.
> >
> >To summarize, I don't think it needs to differ greatly from our current
> >development process.  We're all responsible for breadth of understanding
> >and maintenance of the whole codebase, but we also rely on specific
> >individuals with deep expertise in particular areas for certain issues.
> > Sometimes we commit despite a -1 from Jenkins, based on the community's
> >judgment.
> >
> >Virtualization greatly simplifies cross-platform development.  I use
> >VirtualBox on a Mac host and run VMs for Windows and Ubuntu with a shared
> >drive so that they can all see the same copy of the source code.  There
> >are
> >plenty of variations on this depending on your preference, such as
> >offloading the VMs to a separate server or cloud service to free up local
> >RAM.  I'm planning on submitting BUILDING.txt changes later today that
> >fully describe how to build on Windows.  After some initial setup, it's
> >nearly identical to the mvn commands that you already use today.
> >
> >Hope this helps,
> >--Chris
> >
> >
> >On Thu, Feb 28, 2013 at 3:25 AM, John Gordon
> ><Jo...@microsoft.com>wrote:
> >
> >> +1 (non-binding)
> >>
> >> I want to share my vote of confidence in this community.  If motivated
> >>to
> >> do so, this community can keep this project cross-platform and continue
> >>to
> >> rapidly innovate without breaking a sweat.
> >>
> >> The day we started working on this, I saw the foundations of greatness
> >>in
> >> the quality and volume of dev tests, the code itself, and the Apache
> >>values
> >> themselves.
> >>
> >> 1.) Hadoop's unit tests and their frameworks are very well thought out
> >>and
> >> the consideration and energy that went into their design is worthy of
> >> praise.  The MiniCluster abstractions utilize very few resources and put
> >> all the processes into one JVM for easy debugging.  It is very easy to
> >> select specific tests from the full suite to reproduce an issue
> >>reported in
> >> another environment - like the Jenkins build server or another
> >> contributor's environment.
> >> 2.) This community has done an excellent job of incorporating
> >>well-placed
> >> log messages to make it easy to post mortem troubleshoot most failures.
> >>  The logs are very useful, and it is extremely rare that
> >>troubleshooting a
> >> failure requires debugging a live repro.
> >> 3.) Hadoop is written primarily in Java, a cross-platform language that
> >> provides its own platform in the form of the JVM to insulate most of the
> >> code from the specifics of the OS layer.
> >> 4.) CoPDoC - The right priorities, and well stated.
> >>
> >>
> >> Thank you,
> >>
> >> John
> >>
> >> -----Original Message-----
> >> From: Ivan Mitic [mailto:ivanmi@microsoft.com]
> >> Sent: Wednesday, February 27, 2013 6:32 PM
> >> To: mapreduce-dev@hadoop.apache.org; common-dev@hadoop.apache.org
> >> Cc: yarn-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org
> >> Subject: RE: [Vote] Merge branch-trunk-win to trunk
> >>
> >> +1 (non-binding)
> >>
> >> I am really glad to see this happening! As people already mentioned,
> >>this
> >> has been a great engineering effort involving many people!
> >>
> >>
> >> Folks raised some valid concerns below and I thought it would be good to
> >> share my 2 cents. In my opinion, we don't have to solve all these
> >>problems
> >> right now. As we move forward with two platforms, we can start
> >>addressing
> >> one problem at a time and incrementally improve. In the first iteration,
> >> maintaining Hadoop on Windows could be just everyone trying to do their
> >> best effort (make sure Jenkins build succeeds at least). We already have
> >> people who are building/running trunk on Windows daily, so they would
> >>jump
> >> in and fix problems as needed (we've been doing this in branch-trunk-win
> >> for a while now). Although I see that the problems could arise with
> >> platform specific features/optimizations, I don't think these are
> >>frequent,
> >> so in most cases everything will just work. Merging the two branches
> >>sooner
> >> rather than later does seems like the right thing to do if the ultimate
> >> goal is to have Hadoop on both platforms. Now that the port has
> >>completed,
> >> we will have people in Microsoft (and elsewhere) wanting to contribute
> >> features/improvements to the trunk branch. A separate branch would just
> >> make things more difficult and confusing for everyone :) Hope this makes
> >> sense.
> >>
> >> -----Original Message-----
> >> From: Todd Lipcon [mailto:todd@cloudera.com]
> >> Sent: Wednesday, February 27, 2013 3:43 PM
> >> To: common-dev@hadoop.apache.org
> >> Cc: yarn-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org;
> >> mapreduce-dev@hadoop.apache.org
> >> Subject: Re: [Vote] Merge branch-trunk-win to trunk
> >>
> >> On Wed, Feb 27, 2013 at 2:54 PM, Suresh Srinivas <
> suresh@hortonworks.com
> >> >wrote:
> >>
> >> > With that we need to decide how our precommit process looks.
> >> > My inclination is to wait for +1 from precommit builds on both the
> >> > platforms to ensure no issues are introduced.
> >> > Thoughts?
> >> >
> >> > 2. Feature development impact
> >> > Some questions have been raised about would new features need to be
> >> > supported on both the platforms. Yes. I do not see a reason why
> >> > features cannot work on both the platforms, with the exception of
> >> > platform specific optimizations. This what Java gives us.
> >> >
> >> >
> >> I'm concerned about the above. Personally, I don't have access to any
> >> Windows boxes with development tools, and I know nothing about
> >>developing
> >> on Windows. The only Windows I run is an 8GB VM with 1 GB RAM allocated,
> >> for powerpoint :)
> >>
> >> If I submit a patch and it gets -1 "tests failed" on the Windows slave,
> >> how am I supposed to proceed?
> >>
> >> I think a reasonable compromise would be that the tests should always
> >> *build* on Windows before commit, and contributors should do their best
> >>to
> >> look at the test logs for any Windows-specific failures. But, beyond
> >> looking at the logs, a "-1 Tests failed on windows" should not block a
> >> commit.
> >>
> >> Those contributors who are interested in Windows being a first-class
> >> platform should be responsible for watching the Windows builds and
> >> debugging/fixing any regressions that might be Windows-specific.
> >>
> >> I also think the KDE model that Harsh pointed out is an interesting one
> >>--
> >> ie the idea that we would not merge windows support to trunk, but rather
> >> treat is as a "parallel code line" which lives in the ASF and has its
> >>own
> >> builds and releases. The windows team would periodically merge
> >>trunk->win
> >> to pick up any new changes, and do a separate test/release process. I'm
> >>not
> >> convinced this is the best idea, but worth discussion of pros and cons.
> >>
> >> -Todd
> >>
> >>
> >> >
> >> > On Wed, Feb 27, 2013 at 11:56 AM, Eli Collins <el...@cloudera.com>
> >>wrote:
> >> >
> >> > > Bobby raises some good questions.  A related one, since most current
> >> > > developers won't add Windows support for new features that are
> >> > > platform specific is it assumed that Windows development will either
> >> > > lag or will people actively work on keeping Windows up with the
> >> > > latest?  And vice versa in case Windows support is implemented
> >>first.
> >> > >
> >> > > Is there a jira for resolving the outstanding TODOs in the code base
> >> > > (similar to HDFS-2148)?  Looks like this merge doesn't introduce
> >> > > many which is great (just did a quick diff and grep).
> >> > >
> >> > > Thanks,
> >> > > Eli
> >> > >
> >> > > On Wed, Feb 27, 2013 at 8:17 AM, Robert Evans <ev...@yahoo-inc.com>
> >> > wrote:
> >> > > > After this is merged in is Windows still going to be a second
> >> > > > class citizen but happens to work for more than just development
> >> > > > or is it a fully supported platform where if something breaks it
> >> > > > can block a
> >> > > release?
> >> > > >  How do we as a community intend to keep Windows support from
> >> breaking?
> >> > > > We don't have any Jenkins slaves to be able to run nightly tests
> >> > > > to validate everything still compiles/runs.  This is not a blocker
> >> > > > for me because we often rely on individuals and groups to test
> >> > > > Hadoop, but I
> >> > do
> >> > > > think we need to have this discussion before we put it in.
> >> > > >
> >> > > > --Bobby
> >> > > >
> >> > > > On 2/26/13 4:55 PM, "Suresh Srinivas" <su...@hortonworks.com>
> >> wrote:
> >> > > >
> >> > > >>I had posted heads up about merging branch-trunk-win to trunk on
> >> > > >>Feb
> >> > 8th.
> >> > > >>I
> >> > > >>am happy to announce that we are ready for the merge.
> >> > > >>
> >> > > >>Here is a brief recap on the highlights of the work done:
> >> > > >>- Command-line scripts for the Hadoop surface area
> >> > > >>- Mapping the HDFS permissions model to Windows
> >> > > >>- Abstracted and reconciled mismatches around differences in Path
> >> > > >>semantics in Java and Windows
> >> > > >>- Native Task Controller for Windows
> >> > > >>- Implementation of a Block Placement Policy to support cloud
> >> > > >>environments, more specifically Azure.
> >> > > >>- Implementation of Hadoop native libraries for Windows
> >> > > >>(compression codecs, native I/O)
> >> > > >>- Several reliability issues, including race-conditions,
> >> > > >>intermittent
> >> > > test
> >> > > >>failures, resource leaks.
> >> > > >>- Several new unit test cases written for the above changes
> >> > > >>
> >> > > >>Please find the details of the work in
> >> > > >>CHANGES.branch-trunk-win.txt - Common
> >> > > >>changes<http://bit.ly/Xe7Ynv>, HDFS changes<
> >> > http://bit.ly/13QOSo9
> >> > > >,
> >> > > >>and YARN and MapReduce changes <http://bit.ly/128zzMt>. This is
> >> > > >>the
> >> > work
> >> > > >>ported from branch-1-win to a branch based on trunk.
> >> > > >>
> >> > > >>For details of the testing done, please see the thread -
> >> > > >>http://bit.ly/WpavJ4. Merge patch for this is available on
> >> > HADOOP-8562<
> >> > > >>https://issues.apache.org/jira/browse/HADOOP-8562>.
> >> > > >>
> >> > > >>This was a large undertaking that involved developing code,
> >> > > >>testing the entire Hadoop stack, including scale tests. This is
> >> > > >>made possible only with the contribution from many many folks in
> >> > > >>the community. Following
> >> > people
> >> > > >>contributed to this work: Ivan Mitic, Chuan Liu, Ramya Sunil,
> >> > > >>Bikas
> >> > Saha,
> >> > > >>Kanna Karanam, John Gordon, Brandon Li, Chris Nauroth, David Lao,
> >> > > Sumadhur
> >> > > >>Reddy Bolli, Arpit Agarwal, Ahmed El Baz, Mike Liddell, Jing Zhao,
> >> > Thejas
> >> > > >>Nair, Steve Maine, Ganeshan Iyer, Raja Aluri, Giridharan Kesavan,
> >> > > >>Ramya Bharathi Nimmagadda, Daryn Sharp, Arun Murthy, Tsz-Wo
> >> > > >>Nicholas Sze,
> >> > > Suresh
> >> > > >>Srinivas and Sanjay Radia. There are many others who contributed
> >> > > >>as
> >> > well
> >> > > >>providing feedback and comments on numerous jiras.
> >> > > >>
> >> > > >>The vote will run for seven days and will end on March 5, 6:00PM
> >>PST.
> >> > > >>
> >> > > >>Regards,
> >> > > >>Suresh
> >> > > >>
> >> > > >>
> >> > > >>
> >> > > >>
> >> > > >>On Thu, Feb 7, 2013 at 6:41 PM, Mahadevan Venkatraman
> >> > > >><ma...@microsoft.com>wrote:
> >> > > >>
> >> > > >>> It is super exciting to look at the prospect of these changes
> >> > > >>>being merged  to trunk. Having Windows as one of the supported
> >> > > >>>Hadoop platforms is
> >> > a
> >> > > >>> fantastic opportunity both for the Hadoop project and Microsoft
> >> > > >>>customers.
> >> > > >>>
> >> > > >>> This work began around a year back when a few of us started with
> >> > > >>> a
> >> > > basic
> >> > > >>> port of Hadoop on Windows. Ever since, the Hadoop team in
> >> > > >>> Microsoft
> >> > > have
> >> > > >>> made significant progress in the following areas:
> >> > > >>> (PS: Some of these items are already included in Suresh's email,
> >> > > >>> but including again for completeness)
> >> > > >>>
> >> > > >>> - Command-line scripts for the Hadoop surface area
> >> > > >>> - Mapping the HDFS permissions model to Windows
> >> > > >>> - Abstracted and reconciled mismatches around differences in
> >> > > >>> Path semantics in Java and Windows
> >> > > >>> - Native Task Controller for Windows
> >> > > >>> - Implementation of a Block Placement Policy to support cloud
> >> > > >>> environments, more specifically Azure.
> >> > > >>> - Implementation of Hadoop native libraries for Windows
> >> > > >>> (compression codecs, native I/O) - Several reliability issues,
> >> > > >>> including race-conditions, intermittent test failures, resource
> >> leaks.
> >> > > >>> - Several new unit test cases written for the above changes
> >> > > >>>
> >> > > >>> In the process, we have closely engaged with the Apache open
> >> > > >>> source community and have got great support and assistance from
> >> > > >>> the
> >> > community
> >> > > >>>in
> >> > > >>> terms of contributing fixes, code review comments and commits.
> >> > > >>>
> >> > > >>> In addition, the Hadoop team at Microsoft has also made good
> >> > > >>> progress
> >> > > in
> >> > > >>> other projects including Hive, Pig, Sqoop, Oozie, HCat and
> >>HBase.
> >> > Many
> >> > > >>>of
> >> > > >>> these changes have already been committed to the respective
> >> > > >>>trunks
> >> > with
> >> > > >>> help from various committers and contributors. It is great to
> >> > > >>> see the commitment of the community to support multiple
> >> > > >>> platforms, and we
> >> > look
> >> > > >>> forward to the day when a developer/customer is able to
> >> > > >>>successfully deploy  a complete solution stack based on Apache
> >> > > >>>Hadoop releases.
> >> > > >>>
> >> > > >>> Next Steps:
> >> > > >>>
> >> > > >>> All of the above changes are part of the Windows Azure HDInsight
> >> > > >>>and  HDInsight Server products from Microsoft. We have
> >> > > >>>successfully on-boarded  several internal customers and have been
> >> > > >>>running production workloads
> >> > > on
> >> > > >>> Windows Azure HDInsight. Our vision is to create a big data
> >> > > >>>platform based  on Hadoop, and we are committed to helping make
> >> > > >>>Hadoop a world-class  solution that anyone can use to solve their
> >> > > >>>biggest data challenges.
> >> > > >>>
> >> > > >>> As an immediate next step, we would like to have a discussion
> >> > > >>> around
> >> > > how
> >> > > >>> we can ensure that the quality of the mainline Hadoop branches
> >> > > >>>on Windows  is maintained. To this end, we would like to get to
> >> > > >>>the state where
> >> > we
> >> > > >>>have
> >> > > >>> pre-checkin validation gates and nightly test runs enabled on
> >> > Windows.
> >> > > >>>If
> >> > > >>> you have any suggestions around this, please do send an email.
> >> > > >>>We
> >> > are
> >> > > >>> committed to helping sustain the long-term quality of Hadoop on
> >> > > >>>both Linux  and Windows.
> >> > > >>>
> >> > > >>> We sincerely thank the community for their contribution and
> >> > > >>> support
> >> > so
> >> > > >>> far. And hope to continue having a close engagement in the
> >>future.
> >> > > >>>
> >> > > >>> -Microsoft HDInsight Team
> >> > > >>>
> >> > > >>>
> >> > > >>> -----Original Message-----
> >> > > >>> From: Suresh Srinivas [mailto:suresh@hortonworks.com]
> >> > > >>> Sent: Thursday, February 7, 2013 5:42 PM
> >> > > >>> To: common-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org;
> >> > > >>> hdfs-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
> >> > > >>> Subject: Heads up - merge branch-trunk-win to trunk
> >> > > >>>
> >> > > >>> The support for Hadoop on Windows was proposed in HADOOP-8079<
> >> > > >>> https://issues.apache.org/jira/browse/HADOOP-8079> almost a
> year
> >> > ago.
> >> > > >>>The
> >> > > >>> goal was to make Hadoop natively integrated, full-featured, and
> >> > > >>>performance  and scalability tuned on Windows Server or Windows
> >> > > >>>Azure.
> >> > > >>> We are happy to announce that a lot of progress has been made in
> >> > > >>>this  regard.
> >> > > >>>
> >> > > >>> Initial work started in a feature branch, branch-1-win, based on
> >> > > >>>branch-1.
> >> > > >>> The details related to the work done in the branch can be seen
> >> > > >>>in  CHANGES.txt<
> >> > > >>>
> >> > > >>>
> >> > >
> http://svn.apache.org/viewvc/hadoop/common/branches/branch-1-win/CHA
> >> > > NGES
> >> > .
> >> > > >>>branch-1-win.txt?view=markup
> >> > > >>> >.
> >> > > >>> This work has been ported to a branch, branch-trunk-win, based
> >> > > >>> on
> >> > > trunk.
> >> > > >>> Merge patch for this is available on
> >> > > >>> HADOOP-8562<https://issues.apache.org/jira/browse/HADOOP-8562>
> >> > > >>> .
> >> > > >>>
> >> > > >>> Highlights of the work done so far:
> >> > > >>> 1. Necessary changes in Hadoop to run natively on Windows. These
> >> > > changes
> >> > > >>> handle differences in platforms related to path names,
> >> > > >>>process/task  management etc.
> >> > > >>> 2. Addition of winutils tools for managing file permissions and
> >> > > >>>ownership,  user group mapping, hardlinks, symbolic links, chmod,
> >> > > >>>disk
> >> > utilization,
> >> > > >>>and
> >> > > >>> process/task management.
> >> > > >>> 3. Added cmd scripts equivalent to existing shell scripts
> >> > > >>>hadoop-daemon.sh, start and stop scripts.
> >> > > >>> 4. Addition of block placement policy implemnation to support
> >> > > >>>cloud  enviroment, more specifically Azure.
> >> > > >>>
> >> > > >>> We are very close to wrapping up the work in branch-trunk-win
> >> > > >>>and getting  ready for a merge. Currently the merge patch is
> >> > > >>>passing close to 100%
> >> > > of
> >> > > >>> unit tests on Linux. Soon I will call for a vote to merge this
> >> > > >>>branch into  trunk.
> >> > > >>>
> >> > > >>> Next steps:
> >> > > >>> 1. Call for vote to merge branch-trunk-win to trunk, when the
> >> > > >>> work completes and precommit build is clean.
> >> > > >>> 2. Start a discussion on adding Jenkins precommit builds on
> >> > > >>> windows
> >> > and
> >> > > >>> how to integrate that with the existing commit process.
> >> > > >>>
> >> > > >>> Let me know if you have any questions.
> >> > > >>>
> >> > > >>> Regards,
> >> > > >>> Suresh
> >> > > >>>
> >> > > >>>
> >> > > >>
> >> > > >>
> >> > > >>--
> >> > > >>http://hortonworks.com/download/
> >> > > >
> >> > >
> >> >
> >> >
> >> >
> >> > --
> >> > http://hortonworks.com/download/
> >> >
> >>
> >>
> >>
> >> --
> >> Todd Lipcon
> >> Software Engineer, Cloudera
> >>
> >>
> >>
> >>
> >>
>
>

Re: [Vote] Merge branch-trunk-win to trunk

[Chris Nauroth <cn...@hortonworks.com>]

> Is there a jira for resolving the outstanding TODOs in the code base
> (similar to HDFS-2148)?  Looks like this merge doesn't introduce many
> which is great (just did a quick diff and grep).

I found 2 remaining TODOs introduced in the current merge patch.  One is in
ContainerLaunch.java.  The container launch script was trying to set a
CLASSPATH that exceeded the Windows maximum command line length.  The fix
was to wrap the long classpath into an intermediate jar containing only a
manifest file with a Class-Path entry.  (See YARN-316.)  Just to be
conservative, we wrapped this logic in an if (Shell.WINDOWS) guard and
marked a TODO to remove it later and use that approach on all platforms
after additional testing.  I've tested this code path successfully on Mac
too, but several people wanted additional testing and performance checks
before removing the if (Shell.WINDOWS) guard.  That work is tracked in an
existing jira: YARN-358.

The other TODO is for winutils to print more usage information and
examples.  At this point, I think winutils is printing sufficient
information, and we can just remove the TODO.  I just submitted a new jira
to start that conversation: HADOOP-9348.

Thank you,
--Chris


On Thu, Feb 28, 2013 at 11:29 AM, Robert Evans <ev...@yahoo-inc.com> wrote:

> My initial question was mostly intended to understand the desired new
> classification of Windows after the merge, and how we plan to maintain
> Windows support.  I am happy to hear that hardware for Jenkins will be
> provided.  I am also fine, at least initially, with us trying to treat
> Windows as a first class supported platform.  But I realize that there are
> a lot of people that do not have easy access to Windows for
> development/debugging, myself included. I also don't want to slow down the
> pace of development too much because of this.  It will cause some
> organizations that do not use or support Windows to be more likely to run
> software that has diverged from an official release.  It also has the
> potential to make the patch submission process even more difficult, which
> increases the likelihood of submitters abandoning patches.  However, the
> great thing about being in a community is we can change if we need to.
>
> I am +0 for the merge.  I am not a Windows expert so I don't feel
> comfortable giving it a true +1.
>
> --Bobby
>
>
> On 2/28/13 10:45 AM, "Chris Nauroth" <cn...@hortonworks.com> wrote:
>
> >I'd like to share a few anecdotes about developing cross-platform,
> >hopefully to address some of the concerns about adding overhead to the
> >development process.  By reviewing past cases of cross-platform Linux vs.
> >Windows bugs, we can get a sense for how the development process could
> >look
> >in the future.
> >
> >HADOOP-9131: TestLocalFileSystem#testListStatusWithColons cannot run on
> >Windows.  As part of an earlier jira, HADOOP-8962, there was a new test
> >committed on trunk covering the case of a local file system interaction on
> >a file containing a ':'.  On Windows, ':' in a path has special meaning as
> >part of the drive specifier (i.e. C:), so this test cannot pass when
> >running on Windows.  In this kind of case, the cross-platform bug is
> >obvious, and the fix is obvious (assumeTrue(!Shell.WINDOWS)).  Ideally,
> >this would get fixed pre-commit after seeing a -1 from the Windows Jenkins
> >slave.
> >
> >HDFS-4274: BlockPoolSliceScanner does not close verification log during
> >shutdown.  This caused problems for MiniDFSCluster-based tests running on
> >Windows.  Failure to close the verification log meant that we didn't
> >release file locks, so the tests couldn't delete/recreate working
> >directories during teardown/setup.  Arguably, this was always a bug, and
> >running on Windows just exposed it because of its stricter rules about
> >file
> >locking.  This is a more complex fix, but it doesn't require
> >platform-specific knowledge.  If some future patch accidentally regresses
> >this, then we'll likely see +1 from Linux Jenkins and -1 from Windows
> >Jenkins.  Ideally, it would get fixed pre-commit, because it doesn't
> >require Windows-specific knowledge.  There is also the matter of impact.
> > Re-breaking this would re-break many test suites on Windows.
> >
> >HADOOP-9232: JniBasedUnixGroupsMappingWithFallback fails on Windows with
> >UnsatisfiedLinkError.  This was introduced by HADOOP-8712, which switched
> >to JniBasedUnixGroupsMappingWithFallback as the default
> >hadoop.security.group.mapping, but did not provide a Windows
> >implementation
> >of the JNI function.  In this case, there was a strong desire to get
> >HADOOP-8712 into a release, fixing it on Windows required native Windows
> >API knowledge, and Windows users had a simple workaround available by
> >changing their configs back to ShellBasedUnixGroupsMapping.  I think this
> >is the kind of situation where we could allow HADOOP-8712 to commit
> >despite
> >-1 from Windows Jenkins, with fairly quick follow-up from an engineer with
> >the Windows expertise to fix it.
> >
> >To summarize, I don't think it needs to differ greatly from our current
> >development process.  We're all responsible for breadth of understanding
> >and maintenance of the whole codebase, but we also rely on specific
> >individuals with deep expertise in particular areas for certain issues.
> > Sometimes we commit despite a -1 from Jenkins, based on the community's
> >judgment.
> >
> >Virtualization greatly simplifies cross-platform development.  I use
> >VirtualBox on a Mac host and run VMs for Windows and Ubuntu with a shared
> >drive so that they can all see the same copy of the source code.  There
> >are
> >plenty of variations on this depending on your preference, such as
> >offloading the VMs to a separate server or cloud service to free up local
> >RAM.  I'm planning on submitting BUILDING.txt changes later today that
> >fully describe how to build on Windows.  After some initial setup, it's
> >nearly identical to the mvn commands that you already use today.
> >
> >Hope this helps,
> >--Chris
> >
> >
> >On Thu, Feb 28, 2013 at 3:25 AM, John Gordon
> ><Jo...@microsoft.com>wrote:
> >
> >> +1 (non-binding)
> >>
> >> I want to share my vote of confidence in this community.  If motivated
> >>to
> >> do so, this community can keep this project cross-platform and continue
> >>to
> >> rapidly innovate without breaking a sweat.
> >>
> >> The day we started working on this, I saw the foundations of greatness
> >>in
> >> the quality and volume of dev tests, the code itself, and the Apache
> >>values
> >> themselves.
> >>
> >> 1.) Hadoop's unit tests and their frameworks are very well thought out
> >>and
> >> the consideration and energy that went into their design is worthy of
> >> praise.  The MiniCluster abstractions utilize very few resources and put
> >> all the processes into one JVM for easy debugging.  It is very easy to
> >> select specific tests from the full suite to reproduce an issue
> >>reported in
> >> another environment - like the Jenkins build server or another
> >> contributor's environment.
> >> 2.) This community has done an excellent job of incorporating
> >>well-placed
> >> log messages to make it easy to post mortem troubleshoot most failures.
> >>  The logs are very useful, and it is extremely rare that
> >>troubleshooting a
> >> failure requires debugging a live repro.
> >> 3.) Hadoop is written primarily in Java, a cross-platform language that
> >> provides its own platform in the form of the JVM to insulate most of the
> >> code from the specifics of the OS layer.
> >> 4.) CoPDoC - The right priorities, and well stated.
> >>
> >>
> >> Thank you,
> >>
> >> John
> >>
> >> -----Original Message-----
> >> From: Ivan Mitic [mailto:ivanmi@microsoft.com]
> >> Sent: Wednesday, February 27, 2013 6:32 PM
> >> To: mapreduce-dev@hadoop.apache.org; common-dev@hadoop.apache.org
> >> Cc: yarn-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org
> >> Subject: RE: [Vote] Merge branch-trunk-win to trunk
> >>
> >> +1 (non-binding)
> >>
> >> I am really glad to see this happening! As people already mentioned,
> >>this
> >> has been a great engineering effort involving many people!
> >>
> >>
> >> Folks raised some valid concerns below and I thought it would be good to
> >> share my 2 cents. In my opinion, we don't have to solve all these
> >>problems
> >> right now. As we move forward with two platforms, we can start
> >>addressing
> >> one problem at a time and incrementally improve. In the first iteration,
> >> maintaining Hadoop on Windows could be just everyone trying to do their
> >> best effort (make sure Jenkins build succeeds at least). We already have
> >> people who are building/running trunk on Windows daily, so they would
> >>jump
> >> in and fix problems as needed (we've been doing this in branch-trunk-win
> >> for a while now). Although I see that the problems could arise with
> >> platform specific features/optimizations, I don't think these are
> >>frequent,
> >> so in most cases everything will just work. Merging the two branches
> >>sooner
> >> rather than later does seems like the right thing to do if the ultimate
> >> goal is to have Hadoop on both platforms. Now that the port has
> >>completed,
> >> we will have people in Microsoft (and elsewhere) wanting to contribute
> >> features/improvements to the trunk branch. A separate branch would just
> >> make things more difficult and confusing for everyone :) Hope this makes
> >> sense.
> >>
> >> -----Original Message-----
> >> From: Todd Lipcon [mailto:todd@cloudera.com]
> >> Sent: Wednesday, February 27, 2013 3:43 PM
> >> To: common-dev@hadoop.apache.org
> >> Cc: yarn-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org;
> >> mapreduce-dev@hadoop.apache.org
> >> Subject: Re: [Vote] Merge branch-trunk-win to trunk
> >>
> >> On Wed, Feb 27, 2013 at 2:54 PM, Suresh Srinivas <
> suresh@hortonworks.com
> >> >wrote:
> >>
> >> > With that we need to decide how our precommit process looks.
> >> > My inclination is to wait for +1 from precommit builds on both the
> >> > platforms to ensure no issues are introduced.
> >> > Thoughts?
> >> >
> >> > 2. Feature development impact
> >> > Some questions have been raised about would new features need to be
> >> > supported on both the platforms. Yes. I do not see a reason why
> >> > features cannot work on both the platforms, with the exception of
> >> > platform specific optimizations. This what Java gives us.
> >> >
> >> >
> >> I'm concerned about the above. Personally, I don't have access to any
> >> Windows boxes with development tools, and I know nothing about
> >>developing
> >> on Windows. The only Windows I run is an 8GB VM with 1 GB RAM allocated,
> >> for powerpoint :)
> >>
> >> If I submit a patch and it gets -1 "tests failed" on the Windows slave,
> >> how am I supposed to proceed?
> >>
> >> I think a reasonable compromise would be that the tests should always
> >> *build* on Windows before commit, and contributors should do their best
> >>to
> >> look at the test logs for any Windows-specific failures. But, beyond
> >> looking at the logs, a "-1 Tests failed on windows" should not block a
> >> commit.
> >>
> >> Those contributors who are interested in Windows being a first-class
> >> platform should be responsible for watching the Windows builds and
> >> debugging/fixing any regressions that might be Windows-specific.
> >>
> >> I also think the KDE model that Harsh pointed out is an interesting one
> >>--
> >> ie the idea that we would not merge windows support to trunk, but rather
> >> treat is as a "parallel code line" which lives in the ASF and has its
> >>own
> >> builds and releases. The windows team would periodically merge
> >>trunk->win
> >> to pick up any new changes, and do a separate test/release process. I'm
> >>not
> >> convinced this is the best idea, but worth discussion of pros and cons.
> >>
> >> -Todd
> >>
> >>
> >> >
> >> > On Wed, Feb 27, 2013 at 11:56 AM, Eli Collins <el...@cloudera.com>
> >>wrote:
> >> >
> >> > > Bobby raises some good questions.  A related one, since most current
> >> > > developers won't add Windows support for new features that are
> >> > > platform specific is it assumed that Windows development will either
> >> > > lag or will people actively work on keeping Windows up with the
> >> > > latest?  And vice versa in case Windows support is implemented
> >>first.
> >> > >
> >> > > Is there a jira for resolving the outstanding TODOs in the code base
> >> > > (similar to HDFS-2148)?  Looks like this merge doesn't introduce
> >> > > many which is great (just did a quick diff and grep).
> >> > >
> >> > > Thanks,
> >> > > Eli
> >> > >
> >> > > On Wed, Feb 27, 2013 at 8:17 AM, Robert Evans <ev...@yahoo-inc.com>
> >> > wrote:
> >> > > > After this is merged in is Windows still going to be a second
> >> > > > class citizen but happens to work for more than just development
> >> > > > or is it a fully supported platform where if something breaks it
> >> > > > can block a
> >> > > release?
> >> > > >  How do we as a community intend to keep Windows support from
> >> breaking?
> >> > > > We don't have any Jenkins slaves to be able to run nightly tests
> >> > > > to validate everything still compiles/runs.  This is not a blocker
> >> > > > for me because we often rely on individuals and groups to test
> >> > > > Hadoop, but I
> >> > do
> >> > > > think we need to have this discussion before we put it in.
> >> > > >
> >> > > > --Bobby
> >> > > >
> >> > > > On 2/26/13 4:55 PM, "Suresh Srinivas" <su...@hortonworks.com>
> >> wrote:
> >> > > >
> >> > > >>I had posted heads up about merging branch-trunk-win to trunk on
> >> > > >>Feb
> >> > 8th.
> >> > > >>I
> >> > > >>am happy to announce that we are ready for the merge.
> >> > > >>
> >> > > >>Here is a brief recap on the highlights of the work done:
> >> > > >>- Command-line scripts for the Hadoop surface area
> >> > > >>- Mapping the HDFS permissions model to Windows
> >> > > >>- Abstracted and reconciled mismatches around differences in Path
> >> > > >>semantics in Java and Windows
> >> > > >>- Native Task Controller for Windows
> >> > > >>- Implementation of a Block Placement Policy to support cloud
> >> > > >>environments, more specifically Azure.
> >> > > >>- Implementation of Hadoop native libraries for Windows
> >> > > >>(compression codecs, native I/O)
> >> > > >>- Several reliability issues, including race-conditions,
> >> > > >>intermittent
> >> > > test
> >> > > >>failures, resource leaks.
> >> > > >>- Several new unit test cases written for the above changes
> >> > > >>
> >> > > >>Please find the details of the work in
> >> > > >>CHANGES.branch-trunk-win.txt - Common
> >> > > >>changes<http://bit.ly/Xe7Ynv>, HDFS changes<
> >> > http://bit.ly/13QOSo9
> >> > > >,
> >> > > >>and YARN and MapReduce changes <http://bit.ly/128zzMt>. This is
> >> > > >>the
> >> > work
> >> > > >>ported from branch-1-win to a branch based on trunk.
> >> > > >>
> >> > > >>For details of the testing done, please see the thread -
> >> > > >>http://bit.ly/WpavJ4. Merge patch for this is available on
> >> > HADOOP-8562<
> >> > > >>https://issues.apache.org/jira/browse/HADOOP-8562>.
> >> > > >>
> >> > > >>This was a large undertaking that involved developing code,
> >> > > >>testing the entire Hadoop stack, including scale tests. This is
> >> > > >>made possible only with the contribution from many many folks in
> >> > > >>the community. Following
> >> > people
> >> > > >>contributed to this work: Ivan Mitic, Chuan Liu, Ramya Sunil,
> >> > > >>Bikas
> >> > Saha,
> >> > > >>Kanna Karanam, John Gordon, Brandon Li, Chris Nauroth, David Lao,
> >> > > Sumadhur
> >> > > >>Reddy Bolli, Arpit Agarwal, Ahmed El Baz, Mike Liddell, Jing Zhao,
> >> > Thejas
> >> > > >>Nair, Steve Maine, Ganeshan Iyer, Raja Aluri, Giridharan Kesavan,
> >> > > >>Ramya Bharathi Nimmagadda, Daryn Sharp, Arun Murthy, Tsz-Wo
> >> > > >>Nicholas Sze,
> >> > > Suresh
> >> > > >>Srinivas and Sanjay Radia. There are many others who contributed
> >> > > >>as
> >> > well
> >> > > >>providing feedback and comments on numerous jiras.
> >> > > >>
> >> > > >>The vote will run for seven days and will end on March 5, 6:00PM
> >>PST.
> >> > > >>
> >> > > >>Regards,
> >> > > >>Suresh
> >> > > >>
> >> > > >>
> >> > > >>
> >> > > >>
> >> > > >>On Thu, Feb 7, 2013 at 6:41 PM, Mahadevan Venkatraman
> >> > > >><ma...@microsoft.com>wrote:
> >> > > >>
> >> > > >>> It is super exciting to look at the prospect of these changes
> >> > > >>>being merged  to trunk. Having Windows as one of the supported
> >> > > >>>Hadoop platforms is
> >> > a
> >> > > >>> fantastic opportunity both for the Hadoop project and Microsoft
> >> > > >>>customers.
> >> > > >>>
> >> > > >>> This work began around a year back when a few of us started with
> >> > > >>> a
> >> > > basic
> >> > > >>> port of Hadoop on Windows. Ever since, the Hadoop team in
> >> > > >>> Microsoft
> >> > > have
> >> > > >>> made significant progress in the following areas:
> >> > > >>> (PS: Some of these items are already included in Suresh's email,
> >> > > >>> but including again for completeness)
> >> > > >>>
> >> > > >>> - Command-line scripts for the Hadoop surface area
> >> > > >>> - Mapping the HDFS permissions model to Windows
> >> > > >>> - Abstracted and reconciled mismatches around differences in
> >> > > >>> Path semantics in Java and Windows
> >> > > >>> - Native Task Controller for Windows
> >> > > >>> - Implementation of a Block Placement Policy to support cloud
> >> > > >>> environments, more specifically Azure.
> >> > > >>> - Implementation of Hadoop native libraries for Windows
> >> > > >>> (compression codecs, native I/O) - Several reliability issues,
> >> > > >>> including race-conditions, intermittent test failures, resource
> >> leaks.
> >> > > >>> - Several new unit test cases written for the above changes
> >> > > >>>
> >> > > >>> In the process, we have closely engaged with the Apache open
> >> > > >>> source community and have got great support and assistance from
> >> > > >>> the
> >> > community
> >> > > >>>in
> >> > > >>> terms of contributing fixes, code review comments and commits.
> >> > > >>>
> >> > > >>> In addition, the Hadoop team at Microsoft has also made good
> >> > > >>> progress
> >> > > in
> >> > > >>> other projects including Hive, Pig, Sqoop, Oozie, HCat and
> >>HBase.
> >> > Many
> >> > > >>>of
> >> > > >>> these changes have already been committed to the respective
> >> > > >>>trunks
> >> > with
> >> > > >>> help from various committers and contributors. It is great to
> >> > > >>> see the commitment of the community to support multiple
> >> > > >>> platforms, and we
> >> > look
> >> > > >>> forward to the day when a developer/customer is able to
> >> > > >>>successfully deploy  a complete solution stack based on Apache
> >> > > >>>Hadoop releases.
> >> > > >>>
> >> > > >>> Next Steps:
> >> > > >>>
> >> > > >>> All of the above changes are part of the Windows Azure HDInsight
> >> > > >>>and  HDInsight Server products from Microsoft. We have
> >> > > >>>successfully on-boarded  several internal customers and have been
> >> > > >>>running production workloads
> >> > > on
> >> > > >>> Windows Azure HDInsight. Our vision is to create a big data
> >> > > >>>platform based  on Hadoop, and we are committed to helping make
> >> > > >>>Hadoop a world-class  solution that anyone can use to solve their
> >> > > >>>biggest data challenges.
> >> > > >>>
> >> > > >>> As an immediate next step, we would like to have a discussion
> >> > > >>> around
> >> > > how
> >> > > >>> we can ensure that the quality of the mainline Hadoop branches
> >> > > >>>on Windows  is maintained. To this end, we would like to get to
> >> > > >>>the state where
> >> > we
> >> > > >>>have
> >> > > >>> pre-checkin validation gates and nightly test runs enabled on
> >> > Windows.
> >> > > >>>If
> >> > > >>> you have any suggestions around this, please do send an email.
> >> > > >>>We
> >> > are
> >> > > >>> committed to helping sustain the long-term quality of Hadoop on
> >> > > >>>both Linux  and Windows.
> >> > > >>>
> >> > > >>> We sincerely thank the community for their contribution and
> >> > > >>> support
> >> > so
> >> > > >>> far. And hope to continue having a close engagement in the
> >>future.
> >> > > >>>
> >> > > >>> -Microsoft HDInsight Team
> >> > > >>>
> >> > > >>>
> >> > > >>> -----Original Message-----
> >> > > >>> From: Suresh Srinivas [mailto:suresh@hortonworks.com]
> >> > > >>> Sent: Thursday, February 7, 2013 5:42 PM
> >> > > >>> To: common-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org;
> >> > > >>> hdfs-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
> >> > > >>> Subject: Heads up - merge branch-trunk-win to trunk
> >> > > >>>
> >> > > >>> The support for Hadoop on Windows was proposed in HADOOP-8079<
> >> > > >>> https://issues.apache.org/jira/browse/HADOOP-8079> almost a
> year
> >> > ago.
> >> > > >>>The
> >> > > >>> goal was to make Hadoop natively integrated, full-featured, and
> >> > > >>>performance  and scalability tuned on Windows Server or Windows
> >> > > >>>Azure.
> >> > > >>> We are happy to announce that a lot of progress has been made in
> >> > > >>>this  regard.
> >> > > >>>
> >> > > >>> Initial work started in a feature branch, branch-1-win, based on
> >> > > >>>branch-1.
> >> > > >>> The details related to the work done in the branch can be seen
> >> > > >>>in  CHANGES.txt<
> >> > > >>>
> >> > > >>>
> >> > >
> http://svn.apache.org/viewvc/hadoop/common/branches/branch-1-win/CHA
> >> > > NGES
> >> > .
> >> > > >>>branch-1-win.txt?view=markup
> >> > > >>> >.
> >> > > >>> This work has been ported to a branch, branch-trunk-win, based
> >> > > >>> on
> >> > > trunk.
> >> > > >>> Merge patch for this is available on
> >> > > >>> HADOOP-8562<https://issues.apache.org/jira/browse/HADOOP-8562>
> >> > > >>> .
> >> > > >>>
> >> > > >>> Highlights of the work done so far:
> >> > > >>> 1. Necessary changes in Hadoop to run natively on Windows. These
> >> > > changes
> >> > > >>> handle differences in platforms related to path names,
> >> > > >>>process/task  management etc.
> >> > > >>> 2. Addition of winutils tools for managing file permissions and
> >> > > >>>ownership,  user group mapping, hardlinks, symbolic links, chmod,
> >> > > >>>disk
> >> > utilization,
> >> > > >>>and
> >> > > >>> process/task management.
> >> > > >>> 3. Added cmd scripts equivalent to existing shell scripts
> >> > > >>>hadoop-daemon.sh, start and stop scripts.
> >> > > >>> 4. Addition of block placement policy implemnation to support
> >> > > >>>cloud  enviroment, more specifically Azure.
> >> > > >>>
> >> > > >>> We are very close to wrapping up the work in branch-trunk-win
> >> > > >>>and getting  ready for a merge. Currently the merge patch is
> >> > > >>>passing close to 100%
> >> > > of
> >> > > >>> unit tests on Linux. Soon I will call for a vote to merge this
> >> > > >>>branch into  trunk.
> >> > > >>>
> >> > > >>> Next steps:
> >> > > >>> 1. Call for vote to merge branch-trunk-win to trunk, when the
> >> > > >>> work completes and precommit build is clean.
> >> > > >>> 2. Start a discussion on adding Jenkins precommit builds on
> >> > > >>> windows
> >> > and
> >> > > >>> how to integrate that with the existing commit process.
> >> > > >>>
> >> > > >>> Let me know if you have any questions.
> >> > > >>>
> >> > > >>> Regards,
> >> > > >>> Suresh
> >> > > >>>
> >> > > >>>
> >> > > >>
> >> > > >>
> >> > > >>--
> >> > > >>http://hortonworks.com/download/
> >> > > >
> >> > >
> >> >
> >> >
> >> >
> >> > --
> >> > http://hortonworks.com/download/
> >> >
> >>
> >>
> >>
> >> --
> >> Todd Lipcon
> >> Software Engineer, Cloudera
> >>
> >>
> >>
> >>
> >>
>
>

Re: [Vote] Merge branch-trunk-win to trunk

[Chris Nauroth <cn...@hortonworks.com>]

> Is there a jira for resolving the outstanding TODOs in the code base
> (similar to HDFS-2148)?  Looks like this merge doesn't introduce many
> which is great (just did a quick diff and grep).

I found 2 remaining TODOs introduced in the current merge patch.  One is in
ContainerLaunch.java.  The container launch script was trying to set a
CLASSPATH that exceeded the Windows maximum command line length.  The fix
was to wrap the long classpath into an intermediate jar containing only a
manifest file with a Class-Path entry.  (See YARN-316.)  Just to be
conservative, we wrapped this logic in an if (Shell.WINDOWS) guard and
marked a TODO to remove it later and use that approach on all platforms
after additional testing.  I've tested this code path successfully on Mac
too, but several people wanted additional testing and performance checks
before removing the if (Shell.WINDOWS) guard.  That work is tracked in an
existing jira: YARN-358.

The other TODO is for winutils to print more usage information and
examples.  At this point, I think winutils is printing sufficient
information, and we can just remove the TODO.  I just submitted a new jira
to start that conversation: HADOOP-9348.

Thank you,
--Chris


On Thu, Feb 28, 2013 at 11:29 AM, Robert Evans <ev...@yahoo-inc.com> wrote:

> My initial question was mostly intended to understand the desired new
> classification of Windows after the merge, and how we plan to maintain
> Windows support.  I am happy to hear that hardware for Jenkins will be
> provided.  I am also fine, at least initially, with us trying to treat
> Windows as a first class supported platform.  But I realize that there are
> a lot of people that do not have easy access to Windows for
> development/debugging, myself included. I also don't want to slow down the
> pace of development too much because of this.  It will cause some
> organizations that do not use or support Windows to be more likely to run
> software that has diverged from an official release.  It also has the
> potential to make the patch submission process even more difficult, which
> increases the likelihood of submitters abandoning patches.  However, the
> great thing about being in a community is we can change if we need to.
>
> I am +0 for the merge.  I am not a Windows expert so I don't feel
> comfortable giving it a true +1.
>
> --Bobby
>
>
> On 2/28/13 10:45 AM, "Chris Nauroth" <cn...@hortonworks.com> wrote:
>
> >I'd like to share a few anecdotes about developing cross-platform,
> >hopefully to address some of the concerns about adding overhead to the
> >development process.  By reviewing past cases of cross-platform Linux vs.
> >Windows bugs, we can get a sense for how the development process could
> >look
> >in the future.
> >
> >HADOOP-9131: TestLocalFileSystem#testListStatusWithColons cannot run on
> >Windows.  As part of an earlier jira, HADOOP-8962, there was a new test
> >committed on trunk covering the case of a local file system interaction on
> >a file containing a ':'.  On Windows, ':' in a path has special meaning as
> >part of the drive specifier (i.e. C:), so this test cannot pass when
> >running on Windows.  In this kind of case, the cross-platform bug is
> >obvious, and the fix is obvious (assumeTrue(!Shell.WINDOWS)).  Ideally,
> >this would get fixed pre-commit after seeing a -1 from the Windows Jenkins
> >slave.
> >
> >HDFS-4274: BlockPoolSliceScanner does not close verification log during
> >shutdown.  This caused problems for MiniDFSCluster-based tests running on
> >Windows.  Failure to close the verification log meant that we didn't
> >release file locks, so the tests couldn't delete/recreate working
> >directories during teardown/setup.  Arguably, this was always a bug, and
> >running on Windows just exposed it because of its stricter rules about
> >file
> >locking.  This is a more complex fix, but it doesn't require
> >platform-specific knowledge.  If some future patch accidentally regresses
> >this, then we'll likely see +1 from Linux Jenkins and -1 from Windows
> >Jenkins.  Ideally, it would get fixed pre-commit, because it doesn't
> >require Windows-specific knowledge.  There is also the matter of impact.
> > Re-breaking this would re-break many test suites on Windows.
> >
> >HADOOP-9232: JniBasedUnixGroupsMappingWithFallback fails on Windows with
> >UnsatisfiedLinkError.  This was introduced by HADOOP-8712, which switched
> >to JniBasedUnixGroupsMappingWithFallback as the default
> >hadoop.security.group.mapping, but did not provide a Windows
> >implementation
> >of the JNI function.  In this case, there was a strong desire to get
> >HADOOP-8712 into a release, fixing it on Windows required native Windows
> >API knowledge, and Windows users had a simple workaround available by
> >changing their configs back to ShellBasedUnixGroupsMapping.  I think this
> >is the kind of situation where we could allow HADOOP-8712 to commit
> >despite
> >-1 from Windows Jenkins, with fairly quick follow-up from an engineer with
> >the Windows expertise to fix it.
> >
> >To summarize, I don't think it needs to differ greatly from our current
> >development process.  We're all responsible for breadth of understanding
> >and maintenance of the whole codebase, but we also rely on specific
> >individuals with deep expertise in particular areas for certain issues.
> > Sometimes we commit despite a -1 from Jenkins, based on the community's
> >judgment.
> >
> >Virtualization greatly simplifies cross-platform development.  I use
> >VirtualBox on a Mac host and run VMs for Windows and Ubuntu with a shared
> >drive so that they can all see the same copy of the source code.  There
> >are
> >plenty of variations on this depending on your preference, such as
> >offloading the VMs to a separate server or cloud service to free up local
> >RAM.  I'm planning on submitting BUILDING.txt changes later today that
> >fully describe how to build on Windows.  After some initial setup, it's
> >nearly identical to the mvn commands that you already use today.
> >
> >Hope this helps,
> >--Chris
> >
> >
> >On Thu, Feb 28, 2013 at 3:25 AM, John Gordon
> ><Jo...@microsoft.com>wrote:
> >
> >> +1 (non-binding)
> >>
> >> I want to share my vote of confidence in this community.  If motivated
> >>to
> >> do so, this community can keep this project cross-platform and continue
> >>to
> >> rapidly innovate without breaking a sweat.
> >>
> >> The day we started working on this, I saw the foundations of greatness
> >>in
> >> the quality and volume of dev tests, the code itself, and the Apache
> >>values
> >> themselves.
> >>
> >> 1.) Hadoop's unit tests and their frameworks are very well thought out
> >>and
> >> the consideration and energy that went into their design is worthy of
> >> praise.  The MiniCluster abstractions utilize very few resources and put
> >> all the processes into one JVM for easy debugging.  It is very easy to
> >> select specific tests from the full suite to reproduce an issue
> >>reported in
> >> another environment - like the Jenkins build server or another
> >> contributor's environment.
> >> 2.) This community has done an excellent job of incorporating
> >>well-placed
> >> log messages to make it easy to post mortem troubleshoot most failures.
> >>  The logs are very useful, and it is extremely rare that
> >>troubleshooting a
> >> failure requires debugging a live repro.
> >> 3.) Hadoop is written primarily in Java, a cross-platform language that
> >> provides its own platform in the form of the JVM to insulate most of the
> >> code from the specifics of the OS layer.
> >> 4.) CoPDoC - The right priorities, and well stated.
> >>
> >>
> >> Thank you,
> >>
> >> John
> >>
> >> -----Original Message-----
> >> From: Ivan Mitic [mailto:ivanmi@microsoft.com]
> >> Sent: Wednesday, February 27, 2013 6:32 PM
> >> To: mapreduce-dev@hadoop.apache.org; common-dev@hadoop.apache.org
> >> Cc: yarn-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org
> >> Subject: RE: [Vote] Merge branch-trunk-win to trunk
> >>
> >> +1 (non-binding)
> >>
> >> I am really glad to see this happening! As people already mentioned,
> >>this
> >> has been a great engineering effort involving many people!
> >>
> >>
> >> Folks raised some valid concerns below and I thought it would be good to
> >> share my 2 cents. In my opinion, we don't have to solve all these
> >>problems
> >> right now. As we move forward with two platforms, we can start
> >>addressing
> >> one problem at a time and incrementally improve. In the first iteration,
> >> maintaining Hadoop on Windows could be just everyone trying to do their
> >> best effort (make sure Jenkins build succeeds at least). We already have
> >> people who are building/running trunk on Windows daily, so they would
> >>jump
> >> in and fix problems as needed (we've been doing this in branch-trunk-win
> >> for a while now). Although I see that the problems could arise with
> >> platform specific features/optimizations, I don't think these are
> >>frequent,
> >> so in most cases everything will just work. Merging the two branches
> >>sooner
> >> rather than later does seems like the right thing to do if the ultimate
> >> goal is to have Hadoop on both platforms. Now that the port has
> >>completed,
> >> we will have people in Microsoft (and elsewhere) wanting to contribute
> >> features/improvements to the trunk branch. A separate branch would just
> >> make things more difficult and confusing for everyone :) Hope this makes
> >> sense.
> >>
> >> -----Original Message-----
> >> From: Todd Lipcon [mailto:todd@cloudera.com]
> >> Sent: Wednesday, February 27, 2013 3:43 PM
> >> To: common-dev@hadoop.apache.org
> >> Cc: yarn-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org;
> >> mapreduce-dev@hadoop.apache.org
> >> Subject: Re: [Vote] Merge branch-trunk-win to trunk
> >>
> >> On Wed, Feb 27, 2013 at 2:54 PM, Suresh Srinivas <
> suresh@hortonworks.com
> >> >wrote:
> >>
> >> > With that we need to decide how our precommit process looks.
> >> > My inclination is to wait for +1 from precommit builds on both the
> >> > platforms to ensure no issues are introduced.
> >> > Thoughts?
> >> >
> >> > 2. Feature development impact
> >> > Some questions have been raised about would new features need to be
> >> > supported on both the platforms. Yes. I do not see a reason why
> >> > features cannot work on both the platforms, with the exception of
> >> > platform specific optimizations. This what Java gives us.
> >> >
> >> >
> >> I'm concerned about the above. Personally, I don't have access to any
> >> Windows boxes with development tools, and I know nothing about
> >>developing
> >> on Windows. The only Windows I run is an 8GB VM with 1 GB RAM allocated,
> >> for powerpoint :)
> >>
> >> If I submit a patch and it gets -1 "tests failed" on the Windows slave,
> >> how am I supposed to proceed?
> >>
> >> I think a reasonable compromise would be that the tests should always
> >> *build* on Windows before commit, and contributors should do their best
> >>to
> >> look at the test logs for any Windows-specific failures. But, beyond
> >> looking at the logs, a "-1 Tests failed on windows" should not block a
> >> commit.
> >>
> >> Those contributors who are interested in Windows being a first-class
> >> platform should be responsible for watching the Windows builds and
> >> debugging/fixing any regressions that might be Windows-specific.
> >>
> >> I also think the KDE model that Harsh pointed out is an interesting one
> >>--
> >> ie the idea that we would not merge windows support to trunk, but rather
> >> treat is as a "parallel code line" which lives in the ASF and has its
> >>own
> >> builds and releases. The windows team would periodically merge
> >>trunk->win
> >> to pick up any new changes, and do a separate test/release process. I'm
> >>not
> >> convinced this is the best idea, but worth discussion of pros and cons.
> >>
> >> -Todd
> >>
> >>
> >> >
> >> > On Wed, Feb 27, 2013 at 11:56 AM, Eli Collins <el...@cloudera.com>
> >>wrote:
> >> >
> >> > > Bobby raises some good questions.  A related one, since most current
> >> > > developers won't add Windows support for new features that are
> >> > > platform specific is it assumed that Windows development will either
> >> > > lag or will people actively work on keeping Windows up with the
> >> > > latest?  And vice versa in case Windows support is implemented
> >>first.
> >> > >
> >> > > Is there a jira for resolving the outstanding TODOs in the code base
> >> > > (similar to HDFS-2148)?  Looks like this merge doesn't introduce
> >> > > many which is great (just did a quick diff and grep).
> >> > >
> >> > > Thanks,
> >> > > Eli
> >> > >
> >> > > On Wed, Feb 27, 2013 at 8:17 AM, Robert Evans <ev...@yahoo-inc.com>
> >> > wrote:
> >> > > > After this is merged in is Windows still going to be a second
> >> > > > class citizen but happens to work for more than just development
> >> > > > or is it a fully supported platform where if something breaks it
> >> > > > can block a
> >> > > release?
> >> > > >  How do we as a community intend to keep Windows support from
> >> breaking?
> >> > > > We don't have any Jenkins slaves to be able to run nightly tests
> >> > > > to validate everything still compiles/runs.  This is not a blocker
> >> > > > for me because we often rely on individuals and groups to test
> >> > > > Hadoop, but I
> >> > do
> >> > > > think we need to have this discussion before we put it in.
> >> > > >
> >> > > > --Bobby
> >> > > >
> >> > > > On 2/26/13 4:55 PM, "Suresh Srinivas" <su...@hortonworks.com>
> >> wrote:
> >> > > >
> >> > > >>I had posted heads up about merging branch-trunk-win to trunk on
> >> > > >>Feb
> >> > 8th.
> >> > > >>I
> >> > > >>am happy to announce that we are ready for the merge.
> >> > > >>
> >> > > >>Here is a brief recap on the highlights of the work done:
> >> > > >>- Command-line scripts for the Hadoop surface area
> >> > > >>- Mapping the HDFS permissions model to Windows
> >> > > >>- Abstracted and reconciled mismatches around differences in Path
> >> > > >>semantics in Java and Windows
> >> > > >>- Native Task Controller for Windows
> >> > > >>- Implementation of a Block Placement Policy to support cloud
> >> > > >>environments, more specifically Azure.
> >> > > >>- Implementation of Hadoop native libraries for Windows
> >> > > >>(compression codecs, native I/O)
> >> > > >>- Several reliability issues, including race-conditions,
> >> > > >>intermittent
> >> > > test
> >> > > >>failures, resource leaks.
> >> > > >>- Several new unit test cases written for the above changes
> >> > > >>
> >> > > >>Please find the details of the work in
> >> > > >>CHANGES.branch-trunk-win.txt - Common
> >> > > >>changes<http://bit.ly/Xe7Ynv>, HDFS changes<
> >> > http://bit.ly/13QOSo9
> >> > > >,
> >> > > >>and YARN and MapReduce changes <http://bit.ly/128zzMt>. This is
> >> > > >>the
> >> > work
> >> > > >>ported from branch-1-win to a branch based on trunk.
> >> > > >>
> >> > > >>For details of the testing done, please see the thread -
> >> > > >>http://bit.ly/WpavJ4. Merge patch for this is available on
> >> > HADOOP-8562<
> >> > > >>https://issues.apache.org/jira/browse/HADOOP-8562>.
> >> > > >>
> >> > > >>This was a large undertaking that involved developing code,
> >> > > >>testing the entire Hadoop stack, including scale tests. This is
> >> > > >>made possible only with the contribution from many many folks in
> >> > > >>the community. Following
> >> > people
> >> > > >>contributed to this work: Ivan Mitic, Chuan Liu, Ramya Sunil,
> >> > > >>Bikas
> >> > Saha,
> >> > > >>Kanna Karanam, John Gordon, Brandon Li, Chris Nauroth, David Lao,
> >> > > Sumadhur
> >> > > >>Reddy Bolli, Arpit Agarwal, Ahmed El Baz, Mike Liddell, Jing Zhao,
> >> > Thejas
> >> > > >>Nair, Steve Maine, Ganeshan Iyer, Raja Aluri, Giridharan Kesavan,
> >> > > >>Ramya Bharathi Nimmagadda, Daryn Sharp, Arun Murthy, Tsz-Wo
> >> > > >>Nicholas Sze,
> >> > > Suresh
> >> > > >>Srinivas and Sanjay Radia. There are many others who contributed
> >> > > >>as
> >> > well
> >> > > >>providing feedback and comments on numerous jiras.
> >> > > >>
> >> > > >>The vote will run for seven days and will end on March 5, 6:00PM
> >>PST.
> >> > > >>
> >> > > >>Regards,
> >> > > >>Suresh
> >> > > >>
> >> > > >>
> >> > > >>
> >> > > >>
> >> > > >>On Thu, Feb 7, 2013 at 6:41 PM, Mahadevan Venkatraman
> >> > > >><ma...@microsoft.com>wrote:
> >> > > >>
> >> > > >>> It is super exciting to look at the prospect of these changes
> >> > > >>>being merged  to trunk. Having Windows as one of the supported
> >> > > >>>Hadoop platforms is
> >> > a
> >> > > >>> fantastic opportunity both for the Hadoop project and Microsoft
> >> > > >>>customers.
> >> > > >>>
> >> > > >>> This work began around a year back when a few of us started with
> >> > > >>> a
> >> > > basic
> >> > > >>> port of Hadoop on Windows. Ever since, the Hadoop team in
> >> > > >>> Microsoft
> >> > > have
> >> > > >>> made significant progress in the following areas:
> >> > > >>> (PS: Some of these items are already included in Suresh's email,
> >> > > >>> but including again for completeness)
> >> > > >>>
> >> > > >>> - Command-line scripts for the Hadoop surface area
> >> > > >>> - Mapping the HDFS permissions model to Windows
> >> > > >>> - Abstracted and reconciled mismatches around differences in
> >> > > >>> Path semantics in Java and Windows
> >> > > >>> - Native Task Controller for Windows
> >> > > >>> - Implementation of a Block Placement Policy to support cloud
> >> > > >>> environments, more specifically Azure.
> >> > > >>> - Implementation of Hadoop native libraries for Windows
> >> > > >>> (compression codecs, native I/O) - Several reliability issues,
> >> > > >>> including race-conditions, intermittent test failures, resource
> >> leaks.
> >> > > >>> - Several new unit test cases written for the above changes
> >> > > >>>
> >> > > >>> In the process, we have closely engaged with the Apache open
> >> > > >>> source community and have got great support and assistance from
> >> > > >>> the
> >> > community
> >> > > >>>in
> >> > > >>> terms of contributing fixes, code review comments and commits.
> >> > > >>>
> >> > > >>> In addition, the Hadoop team at Microsoft has also made good
> >> > > >>> progress
> >> > > in
> >> > > >>> other projects including Hive, Pig, Sqoop, Oozie, HCat and
> >>HBase.
> >> > Many
> >> > > >>>of
> >> > > >>> these changes have already been committed to the respective
> >> > > >>>trunks
> >> > with
> >> > > >>> help from various committers and contributors. It is great to
> >> > > >>> see the commitment of the community to support multiple
> >> > > >>> platforms, and we
> >> > look
> >> > > >>> forward to the day when a developer/customer is able to
> >> > > >>>successfully deploy  a complete solution stack based on Apache
> >> > > >>>Hadoop releases.
> >> > > >>>
> >> > > >>> Next Steps:
> >> > > >>>
> >> > > >>> All of the above changes are part of the Windows Azure HDInsight
> >> > > >>>and  HDInsight Server products from Microsoft. We have
> >> > > >>>successfully on-boarded  several internal customers and have been
> >> > > >>>running production workloads
> >> > > on
> >> > > >>> Windows Azure HDInsight. Our vision is to create a big data
> >> > > >>>platform based  on Hadoop, and we are committed to helping make
> >> > > >>>Hadoop a world-class  solution that anyone can use to solve their
> >> > > >>>biggest data challenges.
> >> > > >>>
> >> > > >>> As an immediate next step, we would like to have a discussion
> >> > > >>> around
> >> > > how
> >> > > >>> we can ensure that the quality of the mainline Hadoop branches
> >> > > >>>on Windows  is maintained. To this end, we would like to get to
> >> > > >>>the state where
> >> > we
> >> > > >>>have
> >> > > >>> pre-checkin validation gates and nightly test runs enabled on
> >> > Windows.
> >> > > >>>If
> >> > > >>> you have any suggestions around this, please do send an email.
> >> > > >>>We
> >> > are
> >> > > >>> committed to helping sustain the long-term quality of Hadoop on
> >> > > >>>both Linux  and Windows.
> >> > > >>>
> >> > > >>> We sincerely thank the community for their contribution and
> >> > > >>> support
> >> > so
> >> > > >>> far. And hope to continue having a close engagement in the
> >>future.
> >> > > >>>
> >> > > >>> -Microsoft HDInsight Team
> >> > > >>>
> >> > > >>>
> >> > > >>> -----Original Message-----
> >> > > >>> From: Suresh Srinivas [mailto:suresh@hortonworks.com]
> >> > > >>> Sent: Thursday, February 7, 2013 5:42 PM
> >> > > >>> To: common-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org;
> >> > > >>> hdfs-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
> >> > > >>> Subject: Heads up - merge branch-trunk-win to trunk
> >> > > >>>
> >> > > >>> The support for Hadoop on Windows was proposed in HADOOP-8079<
> >> > > >>> https://issues.apache.org/jira/browse/HADOOP-8079> almost a
> year
> >> > ago.
> >> > > >>>The
> >> > > >>> goal was to make Hadoop natively integrated, full-featured, and
> >> > > >>>performance  and scalability tuned on Windows Server or Windows
> >> > > >>>Azure.
> >> > > >>> We are happy to announce that a lot of progress has been made in
> >> > > >>>this  regard.
> >> > > >>>
> >> > > >>> Initial work started in a feature branch, branch-1-win, based on
> >> > > >>>branch-1.
> >> > > >>> The details related to the work done in the branch can be seen
> >> > > >>>in  CHANGES.txt<
> >> > > >>>
> >> > > >>>
> >> > >
> http://svn.apache.org/viewvc/hadoop/common/branches/branch-1-win/CHA
> >> > > NGES
> >> > .
> >> > > >>>branch-1-win.txt?view=markup
> >> > > >>> >.
> >> > > >>> This work has been ported to a branch, branch-trunk-win, based
> >> > > >>> on
> >> > > trunk.
> >> > > >>> Merge patch for this is available on
> >> > > >>> HADOOP-8562<https://issues.apache.org/jira/browse/HADOOP-8562>
> >> > > >>> .
> >> > > >>>
> >> > > >>> Highlights of the work done so far:
> >> > > >>> 1. Necessary changes in Hadoop to run natively on Windows. These
> >> > > changes
> >> > > >>> handle differences in platforms related to path names,
> >> > > >>>process/task  management etc.
> >> > > >>> 2. Addition of winutils tools for managing file permissions and
> >> > > >>>ownership,  user group mapping, hardlinks, symbolic links, chmod,
> >> > > >>>disk
> >> > utilization,
> >> > > >>>and
> >> > > >>> process/task management.
> >> > > >>> 3. Added cmd scripts equivalent to existing shell scripts
> >> > > >>>hadoop-daemon.sh, start and stop scripts.
> >> > > >>> 4. Addition of block placement policy implemnation to support
> >> > > >>>cloud  enviroment, more specifically Azure.
> >> > > >>>
> >> > > >>> We are very close to wrapping up the work in branch-trunk-win
> >> > > >>>and getting  ready for a merge. Currently the merge patch is
> >> > > >>>passing close to 100%
> >> > > of
> >> > > >>> unit tests on Linux. Soon I will call for a vote to merge this
> >> > > >>>branch into  trunk.
> >> > > >>>
> >> > > >>> Next steps:
> >> > > >>> 1. Call for vote to merge branch-trunk-win to trunk, when the
> >> > > >>> work completes and precommit build is clean.
> >> > > >>> 2. Start a discussion on adding Jenkins precommit builds on
> >> > > >>> windows
> >> > and
> >> > > >>> how to integrate that with the existing commit process.
> >> > > >>>
> >> > > >>> Let me know if you have any questions.
> >> > > >>>
> >> > > >>> Regards,
> >> > > >>> Suresh
> >> > > >>>
> >> > > >>>
> >> > > >>
> >> > > >>
> >> > > >>--
> >> > > >>http://hortonworks.com/download/
> >> > > >
> >> > >
> >> >
> >> >
> >> >
> >> > --
> >> > http://hortonworks.com/download/
> >> >
> >>
> >>
> >>
> >> --
> >> Todd Lipcon
> >> Software Engineer, Cloudera
> >>
> >>
> >>
> >>
> >>
>
>

Re: [Vote] Merge branch-trunk-win to trunk

[Robert Evans <ev...@yahoo-inc.com>]

My initial question was mostly intended to understand the desired new
classification of Windows after the merge, and how we plan to maintain
Windows support.  I am happy to hear that hardware for Jenkins will be
provided.  I am also fine, at least initially, with us trying to treat
Windows as a first class supported platform.  But I realize that there are
a lot of people that do not have easy access to Windows for
development/debugging, myself included. I also don't want to slow down the
pace of development too much because of this.  It will cause some
organizations that do not use or support Windows to be more likely to run
software that has diverged from an official release.  It also has the
potential to make the patch submission process even more difficult, which
increases the likelihood of submitters abandoning patches.  However, the
great thing about being in a community is we can change if we need to.

I am +0 for the merge.  I am not a Windows expert so I don't feel
comfortable giving it a true +1.

--Bobby


On 2/28/13 10:45 AM, "Chris Nauroth" <cn...@hortonworks.com> wrote:

>I'd like to share a few anecdotes about developing cross-platform,
>hopefully to address some of the concerns about adding overhead to the
>development process.  By reviewing past cases of cross-platform Linux vs.
>Windows bugs, we can get a sense for how the development process could
>look
>in the future.
>
>HADOOP-9131: TestLocalFileSystem#testListStatusWithColons cannot run on
>Windows.  As part of an earlier jira, HADOOP-8962, there was a new test
>committed on trunk covering the case of a local file system interaction on
>a file containing a ':'.  On Windows, ':' in a path has special meaning as
>part of the drive specifier (i.e. C:), so this test cannot pass when
>running on Windows.  In this kind of case, the cross-platform bug is
>obvious, and the fix is obvious (assumeTrue(!Shell.WINDOWS)).  Ideally,
>this would get fixed pre-commit after seeing a -1 from the Windows Jenkins
>slave.
>
>HDFS-4274: BlockPoolSliceScanner does not close verification log during
>shutdown.  This caused problems for MiniDFSCluster-based tests running on
>Windows.  Failure to close the verification log meant that we didn't
>release file locks, so the tests couldn't delete/recreate working
>directories during teardown/setup.  Arguably, this was always a bug, and
>running on Windows just exposed it because of its stricter rules about
>file
>locking.  This is a more complex fix, but it doesn't require
>platform-specific knowledge.  If some future patch accidentally regresses
>this, then we'll likely see +1 from Linux Jenkins and -1 from Windows
>Jenkins.  Ideally, it would get fixed pre-commit, because it doesn't
>require Windows-specific knowledge.  There is also the matter of impact.
> Re-breaking this would re-break many test suites on Windows.
>
>HADOOP-9232: JniBasedUnixGroupsMappingWithFallback fails on Windows with
>UnsatisfiedLinkError.  This was introduced by HADOOP-8712, which switched
>to JniBasedUnixGroupsMappingWithFallback as the default
>hadoop.security.group.mapping, but did not provide a Windows
>implementation
>of the JNI function.  In this case, there was a strong desire to get
>HADOOP-8712 into a release, fixing it on Windows required native Windows
>API knowledge, and Windows users had a simple workaround available by
>changing their configs back to ShellBasedUnixGroupsMapping.  I think this
>is the kind of situation where we could allow HADOOP-8712 to commit
>despite
>-1 from Windows Jenkins, with fairly quick follow-up from an engineer with
>the Windows expertise to fix it.
>
>To summarize, I don't think it needs to differ greatly from our current
>development process.  We're all responsible for breadth of understanding
>and maintenance of the whole codebase, but we also rely on specific
>individuals with deep expertise in particular areas for certain issues.
> Sometimes we commit despite a -1 from Jenkins, based on the community's
>judgment.
>
>Virtualization greatly simplifies cross-platform development.  I use
>VirtualBox on a Mac host and run VMs for Windows and Ubuntu with a shared
>drive so that they can all see the same copy of the source code.  There
>are
>plenty of variations on this depending on your preference, such as
>offloading the VMs to a separate server or cloud service to free up local
>RAM.  I'm planning on submitting BUILDING.txt changes later today that
>fully describe how to build on Windows.  After some initial setup, it's
>nearly identical to the mvn commands that you already use today.
>
>Hope this helps,
>--Chris
>
>
>On Thu, Feb 28, 2013 at 3:25 AM, John Gordon
><Jo...@microsoft.com>wrote:
>
>> +1 (non-binding)
>>
>> I want to share my vote of confidence in this community.  If motivated
>>to
>> do so, this community can keep this project cross-platform and continue
>>to
>> rapidly innovate without breaking a sweat.
>>
>> The day we started working on this, I saw the foundations of greatness
>>in
>> the quality and volume of dev tests, the code itself, and the Apache
>>values
>> themselves.
>>
>> 1.) Hadoop's unit tests and their frameworks are very well thought out
>>and
>> the consideration and energy that went into their design is worthy of
>> praise.  The MiniCluster abstractions utilize very few resources and put
>> all the processes into one JVM for easy debugging.  It is very easy to
>> select specific tests from the full suite to reproduce an issue
>>reported in
>> another environment - like the Jenkins build server or another
>> contributor's environment.
>> 2.) This community has done an excellent job of incorporating
>>well-placed
>> log messages to make it easy to post mortem troubleshoot most failures.
>>  The logs are very useful, and it is extremely rare that
>>troubleshooting a
>> failure requires debugging a live repro.
>> 3.) Hadoop is written primarily in Java, a cross-platform language that
>> provides its own platform in the form of the JVM to insulate most of the
>> code from the specifics of the OS layer.
>> 4.) CoPDoC - The right priorities, and well stated.
>>
>>
>> Thank you,
>>
>> John
>>
>> -----Original Message-----
>> From: Ivan Mitic [mailto:ivanmi@microsoft.com]
>> Sent: Wednesday, February 27, 2013 6:32 PM
>> To: mapreduce-dev@hadoop.apache.org; common-dev@hadoop.apache.org
>> Cc: yarn-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org
>> Subject: RE: [Vote] Merge branch-trunk-win to trunk
>>
>> +1 (non-binding)
>>
>> I am really glad to see this happening! As people already mentioned,
>>this
>> has been a great engineering effort involving many people!
>>
>>
>> Folks raised some valid concerns below and I thought it would be good to
>> share my 2 cents. In my opinion, we don't have to solve all these
>>problems
>> right now. As we move forward with two platforms, we can start
>>addressing
>> one problem at a time and incrementally improve. In the first iteration,
>> maintaining Hadoop on Windows could be just everyone trying to do their
>> best effort (make sure Jenkins build succeeds at least). We already have
>> people who are building/running trunk on Windows daily, so they would
>>jump
>> in and fix problems as needed (we've been doing this in branch-trunk-win
>> for a while now). Although I see that the problems could arise with
>> platform specific features/optimizations, I don't think these are
>>frequent,
>> so in most cases everything will just work. Merging the two branches
>>sooner
>> rather than later does seems like the right thing to do if the ultimate
>> goal is to have Hadoop on both platforms. Now that the port has
>>completed,
>> we will have people in Microsoft (and elsewhere) wanting to contribute
>> features/improvements to the trunk branch. A separate branch would just
>> make things more difficult and confusing for everyone :) Hope this makes
>> sense.
>>
>> -----Original Message-----
>> From: Todd Lipcon [mailto:todd@cloudera.com]
>> Sent: Wednesday, February 27, 2013 3:43 PM
>> To: common-dev@hadoop.apache.org
>> Cc: yarn-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org;
>> mapreduce-dev@hadoop.apache.org
>> Subject: Re: [Vote] Merge branch-trunk-win to trunk
>>
>> On Wed, Feb 27, 2013 at 2:54 PM, Suresh Srinivas <suresh@hortonworks.com
>> >wrote:
>>
>> > With that we need to decide how our precommit process looks.
>> > My inclination is to wait for +1 from precommit builds on both the
>> > platforms to ensure no issues are introduced.
>> > Thoughts?
>> >
>> > 2. Feature development impact
>> > Some questions have been raised about would new features need to be
>> > supported on both the platforms. Yes. I do not see a reason why
>> > features cannot work on both the platforms, with the exception of
>> > platform specific optimizations. This what Java gives us.
>> >
>> >
>> I'm concerned about the above. Personally, I don't have access to any
>> Windows boxes with development tools, and I know nothing about
>>developing
>> on Windows. The only Windows I run is an 8GB VM with 1 GB RAM allocated,
>> for powerpoint :)
>>
>> If I submit a patch and it gets -1 "tests failed" on the Windows slave,
>> how am I supposed to proceed?
>>
>> I think a reasonable compromise would be that the tests should always
>> *build* on Windows before commit, and contributors should do their best
>>to
>> look at the test logs for any Windows-specific failures. But, beyond
>> looking at the logs, a "-1 Tests failed on windows" should not block a
>> commit.
>>
>> Those contributors who are interested in Windows being a first-class
>> platform should be responsible for watching the Windows builds and
>> debugging/fixing any regressions that might be Windows-specific.
>>
>> I also think the KDE model that Harsh pointed out is an interesting one
>>--
>> ie the idea that we would not merge windows support to trunk, but rather
>> treat is as a "parallel code line" which lives in the ASF and has its
>>own
>> builds and releases. The windows team would periodically merge
>>trunk->win
>> to pick up any new changes, and do a separate test/release process. I'm
>>not
>> convinced this is the best idea, but worth discussion of pros and cons.
>>
>> -Todd
>>
>>
>> >
>> > On Wed, Feb 27, 2013 at 11:56 AM, Eli Collins <el...@cloudera.com>
>>wrote:
>> >
>> > > Bobby raises some good questions.  A related one, since most current
>> > > developers won't add Windows support for new features that are
>> > > platform specific is it assumed that Windows development will either
>> > > lag or will people actively work on keeping Windows up with the
>> > > latest?  And vice versa in case Windows support is implemented
>>first.
>> > >
>> > > Is there a jira for resolving the outstanding TODOs in the code base
>> > > (similar to HDFS-2148)?  Looks like this merge doesn't introduce
>> > > many which is great (just did a quick diff and grep).
>> > >
>> > > Thanks,
>> > > Eli
>> > >
>> > > On Wed, Feb 27, 2013 at 8:17 AM, Robert Evans <ev...@yahoo-inc.com>
>> > wrote:
>> > > > After this is merged in is Windows still going to be a second
>> > > > class citizen but happens to work for more than just development
>> > > > or is it a fully supported platform where if something breaks it
>> > > > can block a
>> > > release?
>> > > >  How do we as a community intend to keep Windows support from
>> breaking?
>> > > > We don't have any Jenkins slaves to be able to run nightly tests
>> > > > to validate everything still compiles/runs.  This is not a blocker
>> > > > for me because we often rely on individuals and groups to test
>> > > > Hadoop, but I
>> > do
>> > > > think we need to have this discussion before we put it in.
>> > > >
>> > > > --Bobby
>> > > >
>> > > > On 2/26/13 4:55 PM, "Suresh Srinivas" <su...@hortonworks.com>
>> wrote:
>> > > >
>> > > >>I had posted heads up about merging branch-trunk-win to trunk on
>> > > >>Feb
>> > 8th.
>> > > >>I
>> > > >>am happy to announce that we are ready for the merge.
>> > > >>
>> > > >>Here is a brief recap on the highlights of the work done:
>> > > >>- Command-line scripts for the Hadoop surface area
>> > > >>- Mapping the HDFS permissions model to Windows
>> > > >>- Abstracted and reconciled mismatches around differences in Path
>> > > >>semantics in Java and Windows
>> > > >>- Native Task Controller for Windows
>> > > >>- Implementation of a Block Placement Policy to support cloud
>> > > >>environments, more specifically Azure.
>> > > >>- Implementation of Hadoop native libraries for Windows
>> > > >>(compression codecs, native I/O)
>> > > >>- Several reliability issues, including race-conditions,
>> > > >>intermittent
>> > > test
>> > > >>failures, resource leaks.
>> > > >>- Several new unit test cases written for the above changes
>> > > >>
>> > > >>Please find the details of the work in
>> > > >>CHANGES.branch-trunk-win.txt - Common
>> > > >>changes<http://bit.ly/Xe7Ynv>, HDFS changes<
>> > http://bit.ly/13QOSo9
>> > > >,
>> > > >>and YARN and MapReduce changes <http://bit.ly/128zzMt>. This is
>> > > >>the
>> > work
>> > > >>ported from branch-1-win to a branch based on trunk.
>> > > >>
>> > > >>For details of the testing done, please see the thread -
>> > > >>http://bit.ly/WpavJ4. Merge patch for this is available on
>> > HADOOP-8562<
>> > > >>https://issues.apache.org/jira/browse/HADOOP-8562>.
>> > > >>
>> > > >>This was a large undertaking that involved developing code,
>> > > >>testing the entire Hadoop stack, including scale tests. This is
>> > > >>made possible only with the contribution from many many folks in
>> > > >>the community. Following
>> > people
>> > > >>contributed to this work: Ivan Mitic, Chuan Liu, Ramya Sunil,
>> > > >>Bikas
>> > Saha,
>> > > >>Kanna Karanam, John Gordon, Brandon Li, Chris Nauroth, David Lao,
>> > > Sumadhur
>> > > >>Reddy Bolli, Arpit Agarwal, Ahmed El Baz, Mike Liddell, Jing Zhao,
>> > Thejas
>> > > >>Nair, Steve Maine, Ganeshan Iyer, Raja Aluri, Giridharan Kesavan,
>> > > >>Ramya Bharathi Nimmagadda, Daryn Sharp, Arun Murthy, Tsz-Wo
>> > > >>Nicholas Sze,
>> > > Suresh
>> > > >>Srinivas and Sanjay Radia. There are many others who contributed
>> > > >>as
>> > well
>> > > >>providing feedback and comments on numerous jiras.
>> > > >>
>> > > >>The vote will run for seven days and will end on March 5, 6:00PM
>>PST.
>> > > >>
>> > > >>Regards,
>> > > >>Suresh
>> > > >>
>> > > >>
>> > > >>
>> > > >>
>> > > >>On Thu, Feb 7, 2013 at 6:41 PM, Mahadevan Venkatraman
>> > > >><ma...@microsoft.com>wrote:
>> > > >>
>> > > >>> It is super exciting to look at the prospect of these changes
>> > > >>>being merged  to trunk. Having Windows as one of the supported
>> > > >>>Hadoop platforms is
>> > a
>> > > >>> fantastic opportunity both for the Hadoop project and Microsoft
>> > > >>>customers.
>> > > >>>
>> > > >>> This work began around a year back when a few of us started with
>> > > >>> a
>> > > basic
>> > > >>> port of Hadoop on Windows. Ever since, the Hadoop team in
>> > > >>> Microsoft
>> > > have
>> > > >>> made significant progress in the following areas:
>> > > >>> (PS: Some of these items are already included in Suresh's email,
>> > > >>> but including again for completeness)
>> > > >>>
>> > > >>> - Command-line scripts for the Hadoop surface area
>> > > >>> - Mapping the HDFS permissions model to Windows
>> > > >>> - Abstracted and reconciled mismatches around differences in
>> > > >>> Path semantics in Java and Windows
>> > > >>> - Native Task Controller for Windows
>> > > >>> - Implementation of a Block Placement Policy to support cloud
>> > > >>> environments, more specifically Azure.
>> > > >>> - Implementation of Hadoop native libraries for Windows
>> > > >>> (compression codecs, native I/O) - Several reliability issues,
>> > > >>> including race-conditions, intermittent test failures, resource
>> leaks.
>> > > >>> - Several new unit test cases written for the above changes
>> > > >>>
>> > > >>> In the process, we have closely engaged with the Apache open
>> > > >>> source community and have got great support and assistance from
>> > > >>> the
>> > community
>> > > >>>in
>> > > >>> terms of contributing fixes, code review comments and commits.
>> > > >>>
>> > > >>> In addition, the Hadoop team at Microsoft has also made good
>> > > >>> progress
>> > > in
>> > > >>> other projects including Hive, Pig, Sqoop, Oozie, HCat and
>>HBase.
>> > Many
>> > > >>>of
>> > > >>> these changes have already been committed to the respective
>> > > >>>trunks
>> > with
>> > > >>> help from various committers and contributors. It is great to
>> > > >>> see the commitment of the community to support multiple
>> > > >>> platforms, and we
>> > look
>> > > >>> forward to the day when a developer/customer is able to
>> > > >>>successfully deploy  a complete solution stack based on Apache
>> > > >>>Hadoop releases.
>> > > >>>
>> > > >>> Next Steps:
>> > > >>>
>> > > >>> All of the above changes are part of the Windows Azure HDInsight
>> > > >>>and  HDInsight Server products from Microsoft. We have
>> > > >>>successfully on-boarded  several internal customers and have been
>> > > >>>running production workloads
>> > > on
>> > > >>> Windows Azure HDInsight. Our vision is to create a big data
>> > > >>>platform based  on Hadoop, and we are committed to helping make
>> > > >>>Hadoop a world-class  solution that anyone can use to solve their
>> > > >>>biggest data challenges.
>> > > >>>
>> > > >>> As an immediate next step, we would like to have a discussion
>> > > >>> around
>> > > how
>> > > >>> we can ensure that the quality of the mainline Hadoop branches
>> > > >>>on Windows  is maintained. To this end, we would like to get to
>> > > >>>the state where
>> > we
>> > > >>>have
>> > > >>> pre-checkin validation gates and nightly test runs enabled on
>> > Windows.
>> > > >>>If
>> > > >>> you have any suggestions around this, please do send an email.
>> > > >>>We
>> > are
>> > > >>> committed to helping sustain the long-term quality of Hadoop on
>> > > >>>both Linux  and Windows.
>> > > >>>
>> > > >>> We sincerely thank the community for their contribution and
>> > > >>> support
>> > so
>> > > >>> far. And hope to continue having a close engagement in the
>>future.
>> > > >>>
>> > > >>> -Microsoft HDInsight Team
>> > > >>>
>> > > >>>
>> > > >>> -----Original Message-----
>> > > >>> From: Suresh Srinivas [mailto:suresh@hortonworks.com]
>> > > >>> Sent: Thursday, February 7, 2013 5:42 PM
>> > > >>> To: common-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org;
>> > > >>> hdfs-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
>> > > >>> Subject: Heads up - merge branch-trunk-win to trunk
>> > > >>>
>> > > >>> The support for Hadoop on Windows was proposed in HADOOP-8079<
>> > > >>> https://issues.apache.org/jira/browse/HADOOP-8079> almost a year
>> > ago.
>> > > >>>The
>> > > >>> goal was to make Hadoop natively integrated, full-featured, and
>> > > >>>performance  and scalability tuned on Windows Server or Windows
>> > > >>>Azure.
>> > > >>> We are happy to announce that a lot of progress has been made in
>> > > >>>this  regard.
>> > > >>>
>> > > >>> Initial work started in a feature branch, branch-1-win, based on
>> > > >>>branch-1.
>> > > >>> The details related to the work done in the branch can be seen
>> > > >>>in  CHANGES.txt<
>> > > >>>
>> > > >>>
>> > > http://svn.apache.org/viewvc/hadoop/common/branches/branch-1-win/CHA
>> > > NGES
>> > .
>> > > >>>branch-1-win.txt?view=markup
>> > > >>> >.
>> > > >>> This work has been ported to a branch, branch-trunk-win, based
>> > > >>> on
>> > > trunk.
>> > > >>> Merge patch for this is available on
>> > > >>> HADOOP-8562<https://issues.apache.org/jira/browse/HADOOP-8562>
>> > > >>> .
>> > > >>>
>> > > >>> Highlights of the work done so far:
>> > > >>> 1. Necessary changes in Hadoop to run natively on Windows. These
>> > > changes
>> > > >>> handle differences in platforms related to path names,
>> > > >>>process/task  management etc.
>> > > >>> 2. Addition of winutils tools for managing file permissions and
>> > > >>>ownership,  user group mapping, hardlinks, symbolic links, chmod,
>> > > >>>disk
>> > utilization,
>> > > >>>and
>> > > >>> process/task management.
>> > > >>> 3. Added cmd scripts equivalent to existing shell scripts
>> > > >>>hadoop-daemon.sh, start and stop scripts.
>> > > >>> 4. Addition of block placement policy implemnation to support
>> > > >>>cloud  enviroment, more specifically Azure.
>> > > >>>
>> > > >>> We are very close to wrapping up the work in branch-trunk-win
>> > > >>>and getting  ready for a merge. Currently the merge patch is
>> > > >>>passing close to 100%
>> > > of
>> > > >>> unit tests on Linux. Soon I will call for a vote to merge this
>> > > >>>branch into  trunk.
>> > > >>>
>> > > >>> Next steps:
>> > > >>> 1. Call for vote to merge branch-trunk-win to trunk, when the
>> > > >>> work completes and precommit build is clean.
>> > > >>> 2. Start a discussion on adding Jenkins precommit builds on
>> > > >>> windows
>> > and
>> > > >>> how to integrate that with the existing commit process.
>> > > >>>
>> > > >>> Let me know if you have any questions.
>> > > >>>
>> > > >>> Regards,
>> > > >>> Suresh
>> > > >>>
>> > > >>>
>> > > >>
>> > > >>
>> > > >>--
>> > > >>http://hortonworks.com/download/
>> > > >
>> > >
>> >
>> >
>> >
>> > --
>> > http://hortonworks.com/download/
>> >
>>
>>
>>
>> --
>> Todd Lipcon
>> Software Engineer, Cloudera
>>
>>
>>
>>
>>

Re: [Vote] Merge branch-trunk-win to trunk

[Robert Evans <ev...@yahoo-inc.com>]

My initial question was mostly intended to understand the desired new
classification of Windows after the merge, and how we plan to maintain
Windows support.  I am happy to hear that hardware for Jenkins will be
provided.  I am also fine, at least initially, with us trying to treat
Windows as a first class supported platform.  But I realize that there are
a lot of people that do not have easy access to Windows for
development/debugging, myself included. I also don't want to slow down the
pace of development too much because of this.  It will cause some
organizations that do not use or support Windows to be more likely to run
software that has diverged from an official release.  It also has the
potential to make the patch submission process even more difficult, which
increases the likelihood of submitters abandoning patches.  However, the
great thing about being in a community is we can change if we need to.

I am +0 for the merge.  I am not a Windows expert so I don't feel
comfortable giving it a true +1.

--Bobby


On 2/28/13 10:45 AM, "Chris Nauroth" <cn...@hortonworks.com> wrote:

>I'd like to share a few anecdotes about developing cross-platform,
>hopefully to address some of the concerns about adding overhead to the
>development process.  By reviewing past cases of cross-platform Linux vs.
>Windows bugs, we can get a sense for how the development process could
>look
>in the future.
>
>HADOOP-9131: TestLocalFileSystem#testListStatusWithColons cannot run on
>Windows.  As part of an earlier jira, HADOOP-8962, there was a new test
>committed on trunk covering the case of a local file system interaction on
>a file containing a ':'.  On Windows, ':' in a path has special meaning as
>part of the drive specifier (i.e. C:), so this test cannot pass when
>running on Windows.  In this kind of case, the cross-platform bug is
>obvious, and the fix is obvious (assumeTrue(!Shell.WINDOWS)).  Ideally,
>this would get fixed pre-commit after seeing a -1 from the Windows Jenkins
>slave.
>
>HDFS-4274: BlockPoolSliceScanner does not close verification log during
>shutdown.  This caused problems for MiniDFSCluster-based tests running on
>Windows.  Failure to close the verification log meant that we didn't
>release file locks, so the tests couldn't delete/recreate working
>directories during teardown/setup.  Arguably, this was always a bug, and
>running on Windows just exposed it because of its stricter rules about
>file
>locking.  This is a more complex fix, but it doesn't require
>platform-specific knowledge.  If some future patch accidentally regresses
>this, then we'll likely see +1 from Linux Jenkins and -1 from Windows
>Jenkins.  Ideally, it would get fixed pre-commit, because it doesn't
>require Windows-specific knowledge.  There is also the matter of impact.
> Re-breaking this would re-break many test suites on Windows.
>
>HADOOP-9232: JniBasedUnixGroupsMappingWithFallback fails on Windows with
>UnsatisfiedLinkError.  This was introduced by HADOOP-8712, which switched
>to JniBasedUnixGroupsMappingWithFallback as the default
>hadoop.security.group.mapping, but did not provide a Windows
>implementation
>of the JNI function.  In this case, there was a strong desire to get
>HADOOP-8712 into a release, fixing it on Windows required native Windows
>API knowledge, and Windows users had a simple workaround available by
>changing their configs back to ShellBasedUnixGroupsMapping.  I think this
>is the kind of situation where we could allow HADOOP-8712 to commit
>despite
>-1 from Windows Jenkins, with fairly quick follow-up from an engineer with
>the Windows expertise to fix it.
>
>To summarize, I don't think it needs to differ greatly from our current
>development process.  We're all responsible for breadth of understanding
>and maintenance of the whole codebase, but we also rely on specific
>individuals with deep expertise in particular areas for certain issues.
> Sometimes we commit despite a -1 from Jenkins, based on the community's
>judgment.
>
>Virtualization greatly simplifies cross-platform development.  I use
>VirtualBox on a Mac host and run VMs for Windows and Ubuntu with a shared
>drive so that they can all see the same copy of the source code.  There
>are
>plenty of variations on this depending on your preference, such as
>offloading the VMs to a separate server or cloud service to free up local
>RAM.  I'm planning on submitting BUILDING.txt changes later today that
>fully describe how to build on Windows.  After some initial setup, it's
>nearly identical to the mvn commands that you already use today.
>
>Hope this helps,
>--Chris
>
>
>On Thu, Feb 28, 2013 at 3:25 AM, John Gordon
><Jo...@microsoft.com>wrote:
>
>> +1 (non-binding)
>>
>> I want to share my vote of confidence in this community.  If motivated
>>to
>> do so, this community can keep this project cross-platform and continue
>>to
>> rapidly innovate without breaking a sweat.
>>
>> The day we started working on this, I saw the foundations of greatness
>>in
>> the quality and volume of dev tests, the code itself, and the Apache
>>values
>> themselves.
>>
>> 1.) Hadoop's unit tests and their frameworks are very well thought out
>>and
>> the consideration and energy that went into their design is worthy of
>> praise.  The MiniCluster abstractions utilize very few resources and put
>> all the processes into one JVM for easy debugging.  It is very easy to
>> select specific tests from the full suite to reproduce an issue
>>reported in
>> another environment - like the Jenkins build server or another
>> contributor's environment.
>> 2.) This community has done an excellent job of incorporating
>>well-placed
>> log messages to make it easy to post mortem troubleshoot most failures.
>>  The logs are very useful, and it is extremely rare that
>>troubleshooting a
>> failure requires debugging a live repro.
>> 3.) Hadoop is written primarily in Java, a cross-platform language that
>> provides its own platform in the form of the JVM to insulate most of the
>> code from the specifics of the OS layer.
>> 4.) CoPDoC - The right priorities, and well stated.
>>
>>
>> Thank you,
>>
>> John
>>
>> -----Original Message-----
>> From: Ivan Mitic [mailto:ivanmi@microsoft.com]
>> Sent: Wednesday, February 27, 2013 6:32 PM
>> To: mapreduce-dev@hadoop.apache.org; common-dev@hadoop.apache.org
>> Cc: yarn-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org
>> Subject: RE: [Vote] Merge branch-trunk-win to trunk
>>
>> +1 (non-binding)
>>
>> I am really glad to see this happening! As people already mentioned,
>>this
>> has been a great engineering effort involving many people!
>>
>>
>> Folks raised some valid concerns below and I thought it would be good to
>> share my 2 cents. In my opinion, we don't have to solve all these
>>problems
>> right now. As we move forward with two platforms, we can start
>>addressing
>> one problem at a time and incrementally improve. In the first iteration,
>> maintaining Hadoop on Windows could be just everyone trying to do their
>> best effort (make sure Jenkins build succeeds at least). We already have
>> people who are building/running trunk on Windows daily, so they would
>>jump
>> in and fix problems as needed (we've been doing this in branch-trunk-win
>> for a while now). Although I see that the problems could arise with
>> platform specific features/optimizations, I don't think these are
>>frequent,
>> so in most cases everything will just work. Merging the two branches
>>sooner
>> rather than later does seems like the right thing to do if the ultimate
>> goal is to have Hadoop on both platforms. Now that the port has
>>completed,
>> we will have people in Microsoft (and elsewhere) wanting to contribute
>> features/improvements to the trunk branch. A separate branch would just
>> make things more difficult and confusing for everyone :) Hope this makes
>> sense.
>>
>> -----Original Message-----
>> From: Todd Lipcon [mailto:todd@cloudera.com]
>> Sent: Wednesday, February 27, 2013 3:43 PM
>> To: common-dev@hadoop.apache.org
>> Cc: yarn-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org;
>> mapreduce-dev@hadoop.apache.org
>> Subject: Re: [Vote] Merge branch-trunk-win to trunk
>>
>> On Wed, Feb 27, 2013 at 2:54 PM, Suresh Srinivas <suresh@hortonworks.com
>> >wrote:
>>
>> > With that we need to decide how our precommit process looks.
>> > My inclination is to wait for +1 from precommit builds on both the
>> > platforms to ensure no issues are introduced.
>> > Thoughts?
>> >
>> > 2. Feature development impact
>> > Some questions have been raised about would new features need to be
>> > supported on both the platforms. Yes. I do not see a reason why
>> > features cannot work on both the platforms, with the exception of
>> > platform specific optimizations. This what Java gives us.
>> >
>> >
>> I'm concerned about the above. Personally, I don't have access to any
>> Windows boxes with development tools, and I know nothing about
>>developing
>> on Windows. The only Windows I run is an 8GB VM with 1 GB RAM allocated,
>> for powerpoint :)
>>
>> If I submit a patch and it gets -1 "tests failed" on the Windows slave,
>> how am I supposed to proceed?
>>
>> I think a reasonable compromise would be that the tests should always
>> *build* on Windows before commit, and contributors should do their best
>>to
>> look at the test logs for any Windows-specific failures. But, beyond
>> looking at the logs, a "-1 Tests failed on windows" should not block a
>> commit.
>>
>> Those contributors who are interested in Windows being a first-class
>> platform should be responsible for watching the Windows builds and
>> debugging/fixing any regressions that might be Windows-specific.
>>
>> I also think the KDE model that Harsh pointed out is an interesting one
>>--
>> ie the idea that we would not merge windows support to trunk, but rather
>> treat is as a "parallel code line" which lives in the ASF and has its
>>own
>> builds and releases. The windows team would periodically merge
>>trunk->win
>> to pick up any new changes, and do a separate test/release process. I'm
>>not
>> convinced this is the best idea, but worth discussion of pros and cons.
>>
>> -Todd
>>
>>
>> >
>> > On Wed, Feb 27, 2013 at 11:56 AM, Eli Collins <el...@cloudera.com>
>>wrote:
>> >
>> > > Bobby raises some good questions.  A related one, since most current
>> > > developers won't add Windows support for new features that are
>> > > platform specific is it assumed that Windows development will either
>> > > lag or will people actively work on keeping Windows up with the
>> > > latest?  And vice versa in case Windows support is implemented
>>first.
>> > >
>> > > Is there a jira for resolving the outstanding TODOs in the code base
>> > > (similar to HDFS-2148)?  Looks like this merge doesn't introduce
>> > > many which is great (just did a quick diff and grep).
>> > >
>> > > Thanks,
>> > > Eli
>> > >
>> > > On Wed, Feb 27, 2013 at 8:17 AM, Robert Evans <ev...@yahoo-inc.com>
>> > wrote:
>> > > > After this is merged in is Windows still going to be a second
>> > > > class citizen but happens to work for more than just development
>> > > > or is it a fully supported platform where if something breaks it
>> > > > can block a
>> > > release?
>> > > >  How do we as a community intend to keep Windows support from
>> breaking?
>> > > > We don't have any Jenkins slaves to be able to run nightly tests
>> > > > to validate everything still compiles/runs.  This is not a blocker
>> > > > for me because we often rely on individuals and groups to test
>> > > > Hadoop, but I
>> > do
>> > > > think we need to have this discussion before we put it in.
>> > > >
>> > > > --Bobby
>> > > >
>> > > > On 2/26/13 4:55 PM, "Suresh Srinivas" <su...@hortonworks.com>
>> wrote:
>> > > >
>> > > >>I had posted heads up about merging branch-trunk-win to trunk on
>> > > >>Feb
>> > 8th.
>> > > >>I
>> > > >>am happy to announce that we are ready for the merge.
>> > > >>
>> > > >>Here is a brief recap on the highlights of the work done:
>> > > >>- Command-line scripts for the Hadoop surface area
>> > > >>- Mapping the HDFS permissions model to Windows
>> > > >>- Abstracted and reconciled mismatches around differences in Path
>> > > >>semantics in Java and Windows
>> > > >>- Native Task Controller for Windows
>> > > >>- Implementation of a Block Placement Policy to support cloud
>> > > >>environments, more specifically Azure.
>> > > >>- Implementation of Hadoop native libraries for Windows
>> > > >>(compression codecs, native I/O)
>> > > >>- Several reliability issues, including race-conditions,
>> > > >>intermittent
>> > > test
>> > > >>failures, resource leaks.
>> > > >>- Several new unit test cases written for the above changes
>> > > >>
>> > > >>Please find the details of the work in
>> > > >>CHANGES.branch-trunk-win.txt - Common
>> > > >>changes<http://bit.ly/Xe7Ynv>, HDFS changes<
>> > http://bit.ly/13QOSo9
>> > > >,
>> > > >>and YARN and MapReduce changes <http://bit.ly/128zzMt>. This is
>> > > >>the
>> > work
>> > > >>ported from branch-1-win to a branch based on trunk.
>> > > >>
>> > > >>For details of the testing done, please see the thread -
>> > > >>http://bit.ly/WpavJ4. Merge patch for this is available on
>> > HADOOP-8562<
>> > > >>https://issues.apache.org/jira/browse/HADOOP-8562>.
>> > > >>
>> > > >>This was a large undertaking that involved developing code,
>> > > >>testing the entire Hadoop stack, including scale tests. This is
>> > > >>made possible only with the contribution from many many folks in
>> > > >>the community. Following
>> > people
>> > > >>contributed to this work: Ivan Mitic, Chuan Liu, Ramya Sunil,
>> > > >>Bikas
>> > Saha,
>> > > >>Kanna Karanam, John Gordon, Brandon Li, Chris Nauroth, David Lao,
>> > > Sumadhur
>> > > >>Reddy Bolli, Arpit Agarwal, Ahmed El Baz, Mike Liddell, Jing Zhao,
>> > Thejas
>> > > >>Nair, Steve Maine, Ganeshan Iyer, Raja Aluri, Giridharan Kesavan,
>> > > >>Ramya Bharathi Nimmagadda, Daryn Sharp, Arun Murthy, Tsz-Wo
>> > > >>Nicholas Sze,
>> > > Suresh
>> > > >>Srinivas and Sanjay Radia. There are many others who contributed
>> > > >>as
>> > well
>> > > >>providing feedback and comments on numerous jiras.
>> > > >>
>> > > >>The vote will run for seven days and will end on March 5, 6:00PM
>>PST.
>> > > >>
>> > > >>Regards,
>> > > >>Suresh
>> > > >>
>> > > >>
>> > > >>
>> > > >>
>> > > >>On Thu, Feb 7, 2013 at 6:41 PM, Mahadevan Venkatraman
>> > > >><ma...@microsoft.com>wrote:
>> > > >>
>> > > >>> It is super exciting to look at the prospect of these changes
>> > > >>>being merged  to trunk. Having Windows as one of the supported
>> > > >>>Hadoop platforms is
>> > a
>> > > >>> fantastic opportunity both for the Hadoop project and Microsoft
>> > > >>>customers.
>> > > >>>
>> > > >>> This work began around a year back when a few of us started with
>> > > >>> a
>> > > basic
>> > > >>> port of Hadoop on Windows. Ever since, the Hadoop team in
>> > > >>> Microsoft
>> > > have
>> > > >>> made significant progress in the following areas:
>> > > >>> (PS: Some of these items are already included in Suresh's email,
>> > > >>> but including again for completeness)
>> > > >>>
>> > > >>> - Command-line scripts for the Hadoop surface area
>> > > >>> - Mapping the HDFS permissions model to Windows
>> > > >>> - Abstracted and reconciled mismatches around differences in
>> > > >>> Path semantics in Java and Windows
>> > > >>> - Native Task Controller for Windows
>> > > >>> - Implementation of a Block Placement Policy to support cloud
>> > > >>> environments, more specifically Azure.
>> > > >>> - Implementation of Hadoop native libraries for Windows
>> > > >>> (compression codecs, native I/O) - Several reliability issues,
>> > > >>> including race-conditions, intermittent test failures, resource
>> leaks.
>> > > >>> - Several new unit test cases written for the above changes
>> > > >>>
>> > > >>> In the process, we have closely engaged with the Apache open
>> > > >>> source community and have got great support and assistance from
>> > > >>> the
>> > community
>> > > >>>in
>> > > >>> terms of contributing fixes, code review comments and commits.
>> > > >>>
>> > > >>> In addition, the Hadoop team at Microsoft has also made good
>> > > >>> progress
>> > > in
>> > > >>> other projects including Hive, Pig, Sqoop, Oozie, HCat and
>>HBase.
>> > Many
>> > > >>>of
>> > > >>> these changes have already been committed to the respective
>> > > >>>trunks
>> > with
>> > > >>> help from various committers and contributors. It is great to
>> > > >>> see the commitment of the community to support multiple
>> > > >>> platforms, and we
>> > look
>> > > >>> forward to the day when a developer/customer is able to
>> > > >>>successfully deploy  a complete solution stack based on Apache
>> > > >>>Hadoop releases.
>> > > >>>
>> > > >>> Next Steps:
>> > > >>>
>> > > >>> All of the above changes are part of the Windows Azure HDInsight
>> > > >>>and  HDInsight Server products from Microsoft. We have
>> > > >>>successfully on-boarded  several internal customers and have been
>> > > >>>running production workloads
>> > > on
>> > > >>> Windows Azure HDInsight. Our vision is to create a big data
>> > > >>>platform based  on Hadoop, and we are committed to helping make
>> > > >>>Hadoop a world-class  solution that anyone can use to solve their
>> > > >>>biggest data challenges.
>> > > >>>
>> > > >>> As an immediate next step, we would like to have a discussion
>> > > >>> around
>> > > how
>> > > >>> we can ensure that the quality of the mainline Hadoop branches
>> > > >>>on Windows  is maintained. To this end, we would like to get to
>> > > >>>the state where
>> > we
>> > > >>>have
>> > > >>> pre-checkin validation gates and nightly test runs enabled on
>> > Windows.
>> > > >>>If
>> > > >>> you have any suggestions around this, please do send an email.
>> > > >>>We
>> > are
>> > > >>> committed to helping sustain the long-term quality of Hadoop on
>> > > >>>both Linux  and Windows.
>> > > >>>
>> > > >>> We sincerely thank the community for their contribution and
>> > > >>> support
>> > so
>> > > >>> far. And hope to continue having a close engagement in the
>>future.
>> > > >>>
>> > > >>> -Microsoft HDInsight Team
>> > > >>>
>> > > >>>
>> > > >>> -----Original Message-----
>> > > >>> From: Suresh Srinivas [mailto:suresh@hortonworks.com]
>> > > >>> Sent: Thursday, February 7, 2013 5:42 PM
>> > > >>> To: common-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org;
>> > > >>> hdfs-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
>> > > >>> Subject: Heads up - merge branch-trunk-win to trunk
>> > > >>>
>> > > >>> The support for Hadoop on Windows was proposed in HADOOP-8079<
>> > > >>> https://issues.apache.org/jira/browse/HADOOP-8079> almost a year
>> > ago.
>> > > >>>The
>> > > >>> goal was to make Hadoop natively integrated, full-featured, and
>> > > >>>performance  and scalability tuned on Windows Server or Windows
>> > > >>>Azure.
>> > > >>> We are happy to announce that a lot of progress has been made in
>> > > >>>this  regard.
>> > > >>>
>> > > >>> Initial work started in a feature branch, branch-1-win, based on
>> > > >>>branch-1.
>> > > >>> The details related to the work done in the branch can be seen
>> > > >>>in  CHANGES.txt<
>> > > >>>
>> > > >>>
>> > > http://svn.apache.org/viewvc/hadoop/common/branches/branch-1-win/CHA
>> > > NGES
>> > .
>> > > >>>branch-1-win.txt?view=markup
>> > > >>> >.
>> > > >>> This work has been ported to a branch, branch-trunk-win, based
>> > > >>> on
>> > > trunk.
>> > > >>> Merge patch for this is available on
>> > > >>> HADOOP-8562<https://issues.apache.org/jira/browse/HADOOP-8562>
>> > > >>> .
>> > > >>>
>> > > >>> Highlights of the work done so far:
>> > > >>> 1. Necessary changes in Hadoop to run natively on Windows. These
>> > > changes
>> > > >>> handle differences in platforms related to path names,
>> > > >>>process/task  management etc.
>> > > >>> 2. Addition of winutils tools for managing file permissions and
>> > > >>>ownership,  user group mapping, hardlinks, symbolic links, chmod,
>> > > >>>disk
>> > utilization,
>> > > >>>and
>> > > >>> process/task management.
>> > > >>> 3. Added cmd scripts equivalent to existing shell scripts
>> > > >>>hadoop-daemon.sh, start and stop scripts.
>> > > >>> 4. Addition of block placement policy implemnation to support
>> > > >>>cloud  enviroment, more specifically Azure.
>> > > >>>
>> > > >>> We are very close to wrapping up the work in branch-trunk-win
>> > > >>>and getting  ready for a merge. Currently the merge patch is
>> > > >>>passing close to 100%
>> > > of
>> > > >>> unit tests on Linux. Soon I will call for a vote to merge this
>> > > >>>branch into  trunk.
>> > > >>>
>> > > >>> Next steps:
>> > > >>> 1. Call for vote to merge branch-trunk-win to trunk, when the
>> > > >>> work completes and precommit build is clean.
>> > > >>> 2. Start a discussion on adding Jenkins precommit builds on
>> > > >>> windows
>> > and
>> > > >>> how to integrate that with the existing commit process.
>> > > >>>
>> > > >>> Let me know if you have any questions.
>> > > >>>
>> > > >>> Regards,
>> > > >>> Suresh
>> > > >>>
>> > > >>>
>> > > >>
>> > > >>
>> > > >>--
>> > > >>http://hortonworks.com/download/
>> > > >
>> > >
>> >
>> >
>> >
>> > --
>> > http://hortonworks.com/download/
>> >
>>
>>
>>
>> --
>> Todd Lipcon
>> Software Engineer, Cloudera
>>
>>
>>
>>
>>

Re: [Vote] Merge branch-trunk-win to trunk

[Robert Evans <ev...@yahoo-inc.com>]

My initial question was mostly intended to understand the desired new
classification of Windows after the merge, and how we plan to maintain
Windows support.  I am happy to hear that hardware for Jenkins will be
provided.  I am also fine, at least initially, with us trying to treat
Windows as a first class supported platform.  But I realize that there are
a lot of people that do not have easy access to Windows for
development/debugging, myself included. I also don't want to slow down the
pace of development too much because of this.  It will cause some
organizations that do not use or support Windows to be more likely to run
software that has diverged from an official release.  It also has the
potential to make the patch submission process even more difficult, which
increases the likelihood of submitters abandoning patches.  However, the
great thing about being in a community is we can change if we need to.

I am +0 for the merge.  I am not a Windows expert so I don't feel
comfortable giving it a true +1.

--Bobby


On 2/28/13 10:45 AM, "Chris Nauroth" <cn...@hortonworks.com> wrote:

>I'd like to share a few anecdotes about developing cross-platform,
>hopefully to address some of the concerns about adding overhead to the
>development process.  By reviewing past cases of cross-platform Linux vs.
>Windows bugs, we can get a sense for how the development process could
>look
>in the future.
>
>HADOOP-9131: TestLocalFileSystem#testListStatusWithColons cannot run on
>Windows.  As part of an earlier jira, HADOOP-8962, there was a new test
>committed on trunk covering the case of a local file system interaction on
>a file containing a ':'.  On Windows, ':' in a path has special meaning as
>part of the drive specifier (i.e. C:), so this test cannot pass when
>running on Windows.  In this kind of case, the cross-platform bug is
>obvious, and the fix is obvious (assumeTrue(!Shell.WINDOWS)).  Ideally,
>this would get fixed pre-commit after seeing a -1 from the Windows Jenkins
>slave.
>
>HDFS-4274: BlockPoolSliceScanner does not close verification log during
>shutdown.  This caused problems for MiniDFSCluster-based tests running on
>Windows.  Failure to close the verification log meant that we didn't
>release file locks, so the tests couldn't delete/recreate working
>directories during teardown/setup.  Arguably, this was always a bug, and
>running on Windows just exposed it because of its stricter rules about
>file
>locking.  This is a more complex fix, but it doesn't require
>platform-specific knowledge.  If some future patch accidentally regresses
>this, then we'll likely see +1 from Linux Jenkins and -1 from Windows
>Jenkins.  Ideally, it would get fixed pre-commit, because it doesn't
>require Windows-specific knowledge.  There is also the matter of impact.
> Re-breaking this would re-break many test suites on Windows.
>
>HADOOP-9232: JniBasedUnixGroupsMappingWithFallback fails on Windows with
>UnsatisfiedLinkError.  This was introduced by HADOOP-8712, which switched
>to JniBasedUnixGroupsMappingWithFallback as the default
>hadoop.security.group.mapping, but did not provide a Windows
>implementation
>of the JNI function.  In this case, there was a strong desire to get
>HADOOP-8712 into a release, fixing it on Windows required native Windows
>API knowledge, and Windows users had a simple workaround available by
>changing their configs back to ShellBasedUnixGroupsMapping.  I think this
>is the kind of situation where we could allow HADOOP-8712 to commit
>despite
>-1 from Windows Jenkins, with fairly quick follow-up from an engineer with
>the Windows expertise to fix it.
>
>To summarize, I don't think it needs to differ greatly from our current
>development process.  We're all responsible for breadth of understanding
>and maintenance of the whole codebase, but we also rely on specific
>individuals with deep expertise in particular areas for certain issues.
> Sometimes we commit despite a -1 from Jenkins, based on the community's
>judgment.
>
>Virtualization greatly simplifies cross-platform development.  I use
>VirtualBox on a Mac host and run VMs for Windows and Ubuntu with a shared
>drive so that they can all see the same copy of the source code.  There
>are
>plenty of variations on this depending on your preference, such as
>offloading the VMs to a separate server or cloud service to free up local
>RAM.  I'm planning on submitting BUILDING.txt changes later today that
>fully describe how to build on Windows.  After some initial setup, it's
>nearly identical to the mvn commands that you already use today.
>
>Hope this helps,
>--Chris
>
>
>On Thu, Feb 28, 2013 at 3:25 AM, John Gordon
><Jo...@microsoft.com>wrote:
>
>> +1 (non-binding)
>>
>> I want to share my vote of confidence in this community.  If motivated
>>to
>> do so, this community can keep this project cross-platform and continue
>>to
>> rapidly innovate without breaking a sweat.
>>
>> The day we started working on this, I saw the foundations of greatness
>>in
>> the quality and volume of dev tests, the code itself, and the Apache
>>values
>> themselves.
>>
>> 1.) Hadoop's unit tests and their frameworks are very well thought out
>>and
>> the consideration and energy that went into their design is worthy of
>> praise.  The MiniCluster abstractions utilize very few resources and put
>> all the processes into one JVM for easy debugging.  It is very easy to
>> select specific tests from the full suite to reproduce an issue
>>reported in
>> another environment - like the Jenkins build server or another
>> contributor's environment.
>> 2.) This community has done an excellent job of incorporating
>>well-placed
>> log messages to make it easy to post mortem troubleshoot most failures.
>>  The logs are very useful, and it is extremely rare that
>>troubleshooting a
>> failure requires debugging a live repro.
>> 3.) Hadoop is written primarily in Java, a cross-platform language that
>> provides its own platform in the form of the JVM to insulate most of the
>> code from the specifics of the OS layer.
>> 4.) CoPDoC - The right priorities, and well stated.
>>
>>
>> Thank you,
>>
>> John
>>
>> -----Original Message-----
>> From: Ivan Mitic [mailto:ivanmi@microsoft.com]
>> Sent: Wednesday, February 27, 2013 6:32 PM
>> To: mapreduce-dev@hadoop.apache.org; common-dev@hadoop.apache.org
>> Cc: yarn-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org
>> Subject: RE: [Vote] Merge branch-trunk-win to trunk
>>
>> +1 (non-binding)
>>
>> I am really glad to see this happening! As people already mentioned,
>>this
>> has been a great engineering effort involving many people!
>>
>>
>> Folks raised some valid concerns below and I thought it would be good to
>> share my 2 cents. In my opinion, we don't have to solve all these
>>problems
>> right now. As we move forward with two platforms, we can start
>>addressing
>> one problem at a time and incrementally improve. In the first iteration,
>> maintaining Hadoop on Windows could be just everyone trying to do their
>> best effort (make sure Jenkins build succeeds at least). We already have
>> people who are building/running trunk on Windows daily, so they would
>>jump
>> in and fix problems as needed (we've been doing this in branch-trunk-win
>> for a while now). Although I see that the problems could arise with
>> platform specific features/optimizations, I don't think these are
>>frequent,
>> so in most cases everything will just work. Merging the two branches
>>sooner
>> rather than later does seems like the right thing to do if the ultimate
>> goal is to have Hadoop on both platforms. Now that the port has
>>completed,
>> we will have people in Microsoft (and elsewhere) wanting to contribute
>> features/improvements to the trunk branch. A separate branch would just
>> make things more difficult and confusing for everyone :) Hope this makes
>> sense.
>>
>> -----Original Message-----
>> From: Todd Lipcon [mailto:todd@cloudera.com]
>> Sent: Wednesday, February 27, 2013 3:43 PM
>> To: common-dev@hadoop.apache.org
>> Cc: yarn-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org;
>> mapreduce-dev@hadoop.apache.org
>> Subject: Re: [Vote] Merge branch-trunk-win to trunk
>>
>> On Wed, Feb 27, 2013 at 2:54 PM, Suresh Srinivas <suresh@hortonworks.com
>> >wrote:
>>
>> > With that we need to decide how our precommit process looks.
>> > My inclination is to wait for +1 from precommit builds on both the
>> > platforms to ensure no issues are introduced.
>> > Thoughts?
>> >
>> > 2. Feature development impact
>> > Some questions have been raised about would new features need to be
>> > supported on both the platforms. Yes. I do not see a reason why
>> > features cannot work on both the platforms, with the exception of
>> > platform specific optimizations. This what Java gives us.
>> >
>> >
>> I'm concerned about the above. Personally, I don't have access to any
>> Windows boxes with development tools, and I know nothing about
>>developing
>> on Windows. The only Windows I run is an 8GB VM with 1 GB RAM allocated,
>> for powerpoint :)
>>
>> If I submit a patch and it gets -1 "tests failed" on the Windows slave,
>> how am I supposed to proceed?
>>
>> I think a reasonable compromise would be that the tests should always
>> *build* on Windows before commit, and contributors should do their best
>>to
>> look at the test logs for any Windows-specific failures. But, beyond
>> looking at the logs, a "-1 Tests failed on windows" should not block a
>> commit.
>>
>> Those contributors who are interested in Windows being a first-class
>> platform should be responsible for watching the Windows builds and
>> debugging/fixing any regressions that might be Windows-specific.
>>
>> I also think the KDE model that Harsh pointed out is an interesting one
>>--
>> ie the idea that we would not merge windows support to trunk, but rather
>> treat is as a "parallel code line" which lives in the ASF and has its
>>own
>> builds and releases. The windows team would periodically merge
>>trunk->win
>> to pick up any new changes, and do a separate test/release process. I'm
>>not
>> convinced this is the best idea, but worth discussion of pros and cons.
>>
>> -Todd
>>
>>
>> >
>> > On Wed, Feb 27, 2013 at 11:56 AM, Eli Collins <el...@cloudera.com>
>>wrote:
>> >
>> > > Bobby raises some good questions.  A related one, since most current
>> > > developers won't add Windows support for new features that are
>> > > platform specific is it assumed that Windows development will either
>> > > lag or will people actively work on keeping Windows up with the
>> > > latest?  And vice versa in case Windows support is implemented
>>first.
>> > >
>> > > Is there a jira for resolving the outstanding TODOs in the code base
>> > > (similar to HDFS-2148)?  Looks like this merge doesn't introduce
>> > > many which is great (just did a quick diff and grep).
>> > >
>> > > Thanks,
>> > > Eli
>> > >
>> > > On Wed, Feb 27, 2013 at 8:17 AM, Robert Evans <ev...@yahoo-inc.com>
>> > wrote:
>> > > > After this is merged in is Windows still going to be a second
>> > > > class citizen but happens to work for more than just development
>> > > > or is it a fully supported platform where if something breaks it
>> > > > can block a
>> > > release?
>> > > >  How do we as a community intend to keep Windows support from
>> breaking?
>> > > > We don't have any Jenkins slaves to be able to run nightly tests
>> > > > to validate everything still compiles/runs.  This is not a blocker
>> > > > for me because we often rely on individuals and groups to test
>> > > > Hadoop, but I
>> > do
>> > > > think we need to have this discussion before we put it in.
>> > > >
>> > > > --Bobby
>> > > >
>> > > > On 2/26/13 4:55 PM, "Suresh Srinivas" <su...@hortonworks.com>
>> wrote:
>> > > >
>> > > >>I had posted heads up about merging branch-trunk-win to trunk on
>> > > >>Feb
>> > 8th.
>> > > >>I
>> > > >>am happy to announce that we are ready for the merge.
>> > > >>
>> > > >>Here is a brief recap on the highlights of the work done:
>> > > >>- Command-line scripts for the Hadoop surface area
>> > > >>- Mapping the HDFS permissions model to Windows
>> > > >>- Abstracted and reconciled mismatches around differences in Path
>> > > >>semantics in Java and Windows
>> > > >>- Native Task Controller for Windows
>> > > >>- Implementation of a Block Placement Policy to support cloud
>> > > >>environments, more specifically Azure.
>> > > >>- Implementation of Hadoop native libraries for Windows
>> > > >>(compression codecs, native I/O)
>> > > >>- Several reliability issues, including race-conditions,
>> > > >>intermittent
>> > > test
>> > > >>failures, resource leaks.
>> > > >>- Several new unit test cases written for the above changes
>> > > >>
>> > > >>Please find the details of the work in
>> > > >>CHANGES.branch-trunk-win.txt - Common
>> > > >>changes<http://bit.ly/Xe7Ynv>, HDFS changes<
>> > http://bit.ly/13QOSo9
>> > > >,
>> > > >>and YARN and MapReduce changes <http://bit.ly/128zzMt>. This is
>> > > >>the
>> > work
>> > > >>ported from branch-1-win to a branch based on trunk.
>> > > >>
>> > > >>For details of the testing done, please see the thread -
>> > > >>http://bit.ly/WpavJ4. Merge patch for this is available on
>> > HADOOP-8562<
>> > > >>https://issues.apache.org/jira/browse/HADOOP-8562>.
>> > > >>
>> > > >>This was a large undertaking that involved developing code,
>> > > >>testing the entire Hadoop stack, including scale tests. This is
>> > > >>made possible only with the contribution from many many folks in
>> > > >>the community. Following
>> > people
>> > > >>contributed to this work: Ivan Mitic, Chuan Liu, Ramya Sunil,
>> > > >>Bikas
>> > Saha,
>> > > >>Kanna Karanam, John Gordon, Brandon Li, Chris Nauroth, David Lao,
>> > > Sumadhur
>> > > >>Reddy Bolli, Arpit Agarwal, Ahmed El Baz, Mike Liddell, Jing Zhao,
>> > Thejas
>> > > >>Nair, Steve Maine, Ganeshan Iyer, Raja Aluri, Giridharan Kesavan,
>> > > >>Ramya Bharathi Nimmagadda, Daryn Sharp, Arun Murthy, Tsz-Wo
>> > > >>Nicholas Sze,
>> > > Suresh
>> > > >>Srinivas and Sanjay Radia. There are many others who contributed
>> > > >>as
>> > well
>> > > >>providing feedback and comments on numerous jiras.
>> > > >>
>> > > >>The vote will run for seven days and will end on March 5, 6:00PM
>>PST.
>> > > >>
>> > > >>Regards,
>> > > >>Suresh
>> > > >>
>> > > >>
>> > > >>
>> > > >>
>> > > >>On Thu, Feb 7, 2013 at 6:41 PM, Mahadevan Venkatraman
>> > > >><ma...@microsoft.com>wrote:
>> > > >>
>> > > >>> It is super exciting to look at the prospect of these changes
>> > > >>>being merged  to trunk. Having Windows as one of the supported
>> > > >>>Hadoop platforms is
>> > a
>> > > >>> fantastic opportunity both for the Hadoop project and Microsoft
>> > > >>>customers.
>> > > >>>
>> > > >>> This work began around a year back when a few of us started with
>> > > >>> a
>> > > basic
>> > > >>> port of Hadoop on Windows. Ever since, the Hadoop team in
>> > > >>> Microsoft
>> > > have
>> > > >>> made significant progress in the following areas:
>> > > >>> (PS: Some of these items are already included in Suresh's email,
>> > > >>> but including again for completeness)
>> > > >>>
>> > > >>> - Command-line scripts for the Hadoop surface area
>> > > >>> - Mapping the HDFS permissions model to Windows
>> > > >>> - Abstracted and reconciled mismatches around differences in
>> > > >>> Path semantics in Java and Windows
>> > > >>> - Native Task Controller for Windows
>> > > >>> - Implementation of a Block Placement Policy to support cloud
>> > > >>> environments, more specifically Azure.
>> > > >>> - Implementation of Hadoop native libraries for Windows
>> > > >>> (compression codecs, native I/O) - Several reliability issues,
>> > > >>> including race-conditions, intermittent test failures, resource
>> leaks.
>> > > >>> - Several new unit test cases written for the above changes
>> > > >>>
>> > > >>> In the process, we have closely engaged with the Apache open
>> > > >>> source community and have got great support and assistance from
>> > > >>> the
>> > community
>> > > >>>in
>> > > >>> terms of contributing fixes, code review comments and commits.
>> > > >>>
>> > > >>> In addition, the Hadoop team at Microsoft has also made good
>> > > >>> progress
>> > > in
>> > > >>> other projects including Hive, Pig, Sqoop, Oozie, HCat and
>>HBase.
>> > Many
>> > > >>>of
>> > > >>> these changes have already been committed to the respective
>> > > >>>trunks
>> > with
>> > > >>> help from various committers and contributors. It is great to
>> > > >>> see the commitment of the community to support multiple
>> > > >>> platforms, and we
>> > look
>> > > >>> forward to the day when a developer/customer is able to
>> > > >>>successfully deploy  a complete solution stack based on Apache
>> > > >>>Hadoop releases.
>> > > >>>
>> > > >>> Next Steps:
>> > > >>>
>> > > >>> All of the above changes are part of the Windows Azure HDInsight
>> > > >>>and  HDInsight Server products from Microsoft. We have
>> > > >>>successfully on-boarded  several internal customers and have been
>> > > >>>running production workloads
>> > > on
>> > > >>> Windows Azure HDInsight. Our vision is to create a big data
>> > > >>>platform based  on Hadoop, and we are committed to helping make
>> > > >>>Hadoop a world-class  solution that anyone can use to solve their
>> > > >>>biggest data challenges.
>> > > >>>
>> > > >>> As an immediate next step, we would like to have a discussion
>> > > >>> around
>> > > how
>> > > >>> we can ensure that the quality of the mainline Hadoop branches
>> > > >>>on Windows  is maintained. To this end, we would like to get to
>> > > >>>the state where
>> > we
>> > > >>>have
>> > > >>> pre-checkin validation gates and nightly test runs enabled on
>> > Windows.
>> > > >>>If
>> > > >>> you have any suggestions around this, please do send an email.
>> > > >>>We
>> > are
>> > > >>> committed to helping sustain the long-term quality of Hadoop on
>> > > >>>both Linux  and Windows.
>> > > >>>
>> > > >>> We sincerely thank the community for their contribution and
>> > > >>> support
>> > so
>> > > >>> far. And hope to continue having a close engagement in the
>>future.
>> > > >>>
>> > > >>> -Microsoft HDInsight Team
>> > > >>>
>> > > >>>
>> > > >>> -----Original Message-----
>> > > >>> From: Suresh Srinivas [mailto:suresh@hortonworks.com]
>> > > >>> Sent: Thursday, February 7, 2013 5:42 PM
>> > > >>> To: common-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org;
>> > > >>> hdfs-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
>> > > >>> Subject: Heads up - merge branch-trunk-win to trunk
>> > > >>>
>> > > >>> The support for Hadoop on Windows was proposed in HADOOP-8079<
>> > > >>> https://issues.apache.org/jira/browse/HADOOP-8079> almost a year
>> > ago.
>> > > >>>The
>> > > >>> goal was to make Hadoop natively integrated, full-featured, and
>> > > >>>performance  and scalability tuned on Windows Server or Windows
>> > > >>>Azure.
>> > > >>> We are happy to announce that a lot of progress has been made in
>> > > >>>this  regard.
>> > > >>>
>> > > >>> Initial work started in a feature branch, branch-1-win, based on
>> > > >>>branch-1.
>> > > >>> The details related to the work done in the branch can be seen
>> > > >>>in  CHANGES.txt<
>> > > >>>
>> > > >>>
>> > > http://svn.apache.org/viewvc/hadoop/common/branches/branch-1-win/CHA
>> > > NGES
>> > .
>> > > >>>branch-1-win.txt?view=markup
>> > > >>> >.
>> > > >>> This work has been ported to a branch, branch-trunk-win, based
>> > > >>> on
>> > > trunk.
>> > > >>> Merge patch for this is available on
>> > > >>> HADOOP-8562<https://issues.apache.org/jira/browse/HADOOP-8562>
>> > > >>> .
>> > > >>>
>> > > >>> Highlights of the work done so far:
>> > > >>> 1. Necessary changes in Hadoop to run natively on Windows. These
>> > > changes
>> > > >>> handle differences in platforms related to path names,
>> > > >>>process/task  management etc.
>> > > >>> 2. Addition of winutils tools for managing file permissions and
>> > > >>>ownership,  user group mapping, hardlinks, symbolic links, chmod,
>> > > >>>disk
>> > utilization,
>> > > >>>and
>> > > >>> process/task management.
>> > > >>> 3. Added cmd scripts equivalent to existing shell scripts
>> > > >>>hadoop-daemon.sh, start and stop scripts.
>> > > >>> 4. Addition of block placement policy implemnation to support
>> > > >>>cloud  enviroment, more specifically Azure.
>> > > >>>
>> > > >>> We are very close to wrapping up the work in branch-trunk-win
>> > > >>>and getting  ready for a merge. Currently the merge patch is
>> > > >>>passing close to 100%
>> > > of
>> > > >>> unit tests on Linux. Soon I will call for a vote to merge this
>> > > >>>branch into  trunk.
>> > > >>>
>> > > >>> Next steps:
>> > > >>> 1. Call for vote to merge branch-trunk-win to trunk, when the
>> > > >>> work completes and precommit build is clean.
>> > > >>> 2. Start a discussion on adding Jenkins precommit builds on
>> > > >>> windows
>> > and
>> > > >>> how to integrate that with the existing commit process.
>> > > >>>
>> > > >>> Let me know if you have any questions.
>> > > >>>
>> > > >>> Regards,
>> > > >>> Suresh
>> > > >>>
>> > > >>>
>> > > >>
>> > > >>
>> > > >>--
>> > > >>http://hortonworks.com/download/
>> > > >
>> > >
>> >
>> >
>> >
>> > --
>> > http://hortonworks.com/download/
>> >
>>
>>
>>
>> --
>> Todd Lipcon
>> Software Engineer, Cloudera
>>
>>
>>
>>
>>

Re: [Vote] Merge branch-trunk-win to trunk

[Robert Evans <ev...@yahoo-inc.com>]

My initial question was mostly intended to understand the desired new
classification of Windows after the merge, and how we plan to maintain
Windows support.  I am happy to hear that hardware for Jenkins will be
provided.  I am also fine, at least initially, with us trying to treat
Windows as a first class supported platform.  But I realize that there are
a lot of people that do not have easy access to Windows for
development/debugging, myself included. I also don't want to slow down the
pace of development too much because of this.  It will cause some
organizations that do not use or support Windows to be more likely to run
software that has diverged from an official release.  It also has the
potential to make the patch submission process even more difficult, which
increases the likelihood of submitters abandoning patches.  However, the
great thing about being in a community is we can change if we need to.

I am +0 for the merge.  I am not a Windows expert so I don't feel
comfortable giving it a true +1.

--Bobby


On 2/28/13 10:45 AM, "Chris Nauroth" <cn...@hortonworks.com> wrote:

>I'd like to share a few anecdotes about developing cross-platform,
>hopefully to address some of the concerns about adding overhead to the
>development process.  By reviewing past cases of cross-platform Linux vs.
>Windows bugs, we can get a sense for how the development process could
>look
>in the future.
>
>HADOOP-9131: TestLocalFileSystem#testListStatusWithColons cannot run on
>Windows.  As part of an earlier jira, HADOOP-8962, there was a new test
>committed on trunk covering the case of a local file system interaction on
>a file containing a ':'.  On Windows, ':' in a path has special meaning as
>part of the drive specifier (i.e. C:), so this test cannot pass when
>running on Windows.  In this kind of case, the cross-platform bug is
>obvious, and the fix is obvious (assumeTrue(!Shell.WINDOWS)).  Ideally,
>this would get fixed pre-commit after seeing a -1 from the Windows Jenkins
>slave.
>
>HDFS-4274: BlockPoolSliceScanner does not close verification log during
>shutdown.  This caused problems for MiniDFSCluster-based tests running on
>Windows.  Failure to close the verification log meant that we didn't
>release file locks, so the tests couldn't delete/recreate working
>directories during teardown/setup.  Arguably, this was always a bug, and
>running on Windows just exposed it because of its stricter rules about
>file
>locking.  This is a more complex fix, but it doesn't require
>platform-specific knowledge.  If some future patch accidentally regresses
>this, then we'll likely see +1 from Linux Jenkins and -1 from Windows
>Jenkins.  Ideally, it would get fixed pre-commit, because it doesn't
>require Windows-specific knowledge.  There is also the matter of impact.
> Re-breaking this would re-break many test suites on Windows.
>
>HADOOP-9232: JniBasedUnixGroupsMappingWithFallback fails on Windows with
>UnsatisfiedLinkError.  This was introduced by HADOOP-8712, which switched
>to JniBasedUnixGroupsMappingWithFallback as the default
>hadoop.security.group.mapping, but did not provide a Windows
>implementation
>of the JNI function.  In this case, there was a strong desire to get
>HADOOP-8712 into a release, fixing it on Windows required native Windows
>API knowledge, and Windows users had a simple workaround available by
>changing their configs back to ShellBasedUnixGroupsMapping.  I think this
>is the kind of situation where we could allow HADOOP-8712 to commit
>despite
>-1 from Windows Jenkins, with fairly quick follow-up from an engineer with
>the Windows expertise to fix it.
>
>To summarize, I don't think it needs to differ greatly from our current
>development process.  We're all responsible for breadth of understanding
>and maintenance of the whole codebase, but we also rely on specific
>individuals with deep expertise in particular areas for certain issues.
> Sometimes we commit despite a -1 from Jenkins, based on the community's
>judgment.
>
>Virtualization greatly simplifies cross-platform development.  I use
>VirtualBox on a Mac host and run VMs for Windows and Ubuntu with a shared
>drive so that they can all see the same copy of the source code.  There
>are
>plenty of variations on this depending on your preference, such as
>offloading the VMs to a separate server or cloud service to free up local
>RAM.  I'm planning on submitting BUILDING.txt changes later today that
>fully describe how to build on Windows.  After some initial setup, it's
>nearly identical to the mvn commands that you already use today.
>
>Hope this helps,
>--Chris
>
>
>On Thu, Feb 28, 2013 at 3:25 AM, John Gordon
><Jo...@microsoft.com>wrote:
>
>> +1 (non-binding)
>>
>> I want to share my vote of confidence in this community.  If motivated
>>to
>> do so, this community can keep this project cross-platform and continue
>>to
>> rapidly innovate without breaking a sweat.
>>
>> The day we started working on this, I saw the foundations of greatness
>>in
>> the quality and volume of dev tests, the code itself, and the Apache
>>values
>> themselves.
>>
>> 1.) Hadoop's unit tests and their frameworks are very well thought out
>>and
>> the consideration and energy that went into their design is worthy of
>> praise.  The MiniCluster abstractions utilize very few resources and put
>> all the processes into one JVM for easy debugging.  It is very easy to
>> select specific tests from the full suite to reproduce an issue
>>reported in
>> another environment - like the Jenkins build server or another
>> contributor's environment.
>> 2.) This community has done an excellent job of incorporating
>>well-placed
>> log messages to make it easy to post mortem troubleshoot most failures.
>>  The logs are very useful, and it is extremely rare that
>>troubleshooting a
>> failure requires debugging a live repro.
>> 3.) Hadoop is written primarily in Java, a cross-platform language that
>> provides its own platform in the form of the JVM to insulate most of the
>> code from the specifics of the OS layer.
>> 4.) CoPDoC - The right priorities, and well stated.
>>
>>
>> Thank you,
>>
>> John
>>
>> -----Original Message-----
>> From: Ivan Mitic [mailto:ivanmi@microsoft.com]
>> Sent: Wednesday, February 27, 2013 6:32 PM
>> To: mapreduce-dev@hadoop.apache.org; common-dev@hadoop.apache.org
>> Cc: yarn-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org
>> Subject: RE: [Vote] Merge branch-trunk-win to trunk
>>
>> +1 (non-binding)
>>
>> I am really glad to see this happening! As people already mentioned,
>>this
>> has been a great engineering effort involving many people!
>>
>>
>> Folks raised some valid concerns below and I thought it would be good to
>> share my 2 cents. In my opinion, we don't have to solve all these
>>problems
>> right now. As we move forward with two platforms, we can start
>>addressing
>> one problem at a time and incrementally improve. In the first iteration,
>> maintaining Hadoop on Windows could be just everyone trying to do their
>> best effort (make sure Jenkins build succeeds at least). We already have
>> people who are building/running trunk on Windows daily, so they would
>>jump
>> in and fix problems as needed (we've been doing this in branch-trunk-win
>> for a while now). Although I see that the problems could arise with
>> platform specific features/optimizations, I don't think these are
>>frequent,
>> so in most cases everything will just work. Merging the two branches
>>sooner
>> rather than later does seems like the right thing to do if the ultimate
>> goal is to have Hadoop on both platforms. Now that the port has
>>completed,
>> we will have people in Microsoft (and elsewhere) wanting to contribute
>> features/improvements to the trunk branch. A separate branch would just
>> make things more difficult and confusing for everyone :) Hope this makes
>> sense.
>>
>> -----Original Message-----
>> From: Todd Lipcon [mailto:todd@cloudera.com]
>> Sent: Wednesday, February 27, 2013 3:43 PM
>> To: common-dev@hadoop.apache.org
>> Cc: yarn-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org;
>> mapreduce-dev@hadoop.apache.org
>> Subject: Re: [Vote] Merge branch-trunk-win to trunk
>>
>> On Wed, Feb 27, 2013 at 2:54 PM, Suresh Srinivas <suresh@hortonworks.com
>> >wrote:
>>
>> > With that we need to decide how our precommit process looks.
>> > My inclination is to wait for +1 from precommit builds on both the
>> > platforms to ensure no issues are introduced.
>> > Thoughts?
>> >
>> > 2. Feature development impact
>> > Some questions have been raised about would new features need to be
>> > supported on both the platforms. Yes. I do not see a reason why
>> > features cannot work on both the platforms, with the exception of
>> > platform specific optimizations. This what Java gives us.
>> >
>> >
>> I'm concerned about the above. Personally, I don't have access to any
>> Windows boxes with development tools, and I know nothing about
>>developing
>> on Windows. The only Windows I run is an 8GB VM with 1 GB RAM allocated,
>> for powerpoint :)
>>
>> If I submit a patch and it gets -1 "tests failed" on the Windows slave,
>> how am I supposed to proceed?
>>
>> I think a reasonable compromise would be that the tests should always
>> *build* on Windows before commit, and contributors should do their best
>>to
>> look at the test logs for any Windows-specific failures. But, beyond
>> looking at the logs, a "-1 Tests failed on windows" should not block a
>> commit.
>>
>> Those contributors who are interested in Windows being a first-class
>> platform should be responsible for watching the Windows builds and
>> debugging/fixing any regressions that might be Windows-specific.
>>
>> I also think the KDE model that Harsh pointed out is an interesting one
>>--
>> ie the idea that we would not merge windows support to trunk, but rather
>> treat is as a "parallel code line" which lives in the ASF and has its
>>own
>> builds and releases. The windows team would periodically merge
>>trunk->win
>> to pick up any new changes, and do a separate test/release process. I'm
>>not
>> convinced this is the best idea, but worth discussion of pros and cons.
>>
>> -Todd
>>
>>
>> >
>> > On Wed, Feb 27, 2013 at 11:56 AM, Eli Collins <el...@cloudera.com>
>>wrote:
>> >
>> > > Bobby raises some good questions.  A related one, since most current
>> > > developers won't add Windows support for new features that are
>> > > platform specific is it assumed that Windows development will either
>> > > lag or will people actively work on keeping Windows up with the
>> > > latest?  And vice versa in case Windows support is implemented
>>first.
>> > >
>> > > Is there a jira for resolving the outstanding TODOs in the code base
>> > > (similar to HDFS-2148)?  Looks like this merge doesn't introduce
>> > > many which is great (just did a quick diff and grep).
>> > >
>> > > Thanks,
>> > > Eli
>> > >
>> > > On Wed, Feb 27, 2013 at 8:17 AM, Robert Evans <ev...@yahoo-inc.com>
>> > wrote:
>> > > > After this is merged in is Windows still going to be a second
>> > > > class citizen but happens to work for more than just development
>> > > > or is it a fully supported platform where if something breaks it
>> > > > can block a
>> > > release?
>> > > >  How do we as a community intend to keep Windows support from
>> breaking?
>> > > > We don't have any Jenkins slaves to be able to run nightly tests
>> > > > to validate everything still compiles/runs.  This is not a blocker
>> > > > for me because we often rely on individuals and groups to test
>> > > > Hadoop, but I
>> > do
>> > > > think we need to have this discussion before we put it in.
>> > > >
>> > > > --Bobby
>> > > >
>> > > > On 2/26/13 4:55 PM, "Suresh Srinivas" <su...@hortonworks.com>
>> wrote:
>> > > >
>> > > >>I had posted heads up about merging branch-trunk-win to trunk on
>> > > >>Feb
>> > 8th.
>> > > >>I
>> > > >>am happy to announce that we are ready for the merge.
>> > > >>
>> > > >>Here is a brief recap on the highlights of the work done:
>> > > >>- Command-line scripts for the Hadoop surface area
>> > > >>- Mapping the HDFS permissions model to Windows
>> > > >>- Abstracted and reconciled mismatches around differences in Path
>> > > >>semantics in Java and Windows
>> > > >>- Native Task Controller for Windows
>> > > >>- Implementation of a Block Placement Policy to support cloud
>> > > >>environments, more specifically Azure.
>> > > >>- Implementation of Hadoop native libraries for Windows
>> > > >>(compression codecs, native I/O)
>> > > >>- Several reliability issues, including race-conditions,
>> > > >>intermittent
>> > > test
>> > > >>failures, resource leaks.
>> > > >>- Several new unit test cases written for the above changes
>> > > >>
>> > > >>Please find the details of the work in
>> > > >>CHANGES.branch-trunk-win.txt - Common
>> > > >>changes<http://bit.ly/Xe7Ynv>, HDFS changes<
>> > http://bit.ly/13QOSo9
>> > > >,
>> > > >>and YARN and MapReduce changes <http://bit.ly/128zzMt>. This is
>> > > >>the
>> > work
>> > > >>ported from branch-1-win to a branch based on trunk.
>> > > >>
>> > > >>For details of the testing done, please see the thread -
>> > > >>http://bit.ly/WpavJ4. Merge patch for this is available on
>> > HADOOP-8562<
>> > > >>https://issues.apache.org/jira/browse/HADOOP-8562>.
>> > > >>
>> > > >>This was a large undertaking that involved developing code,
>> > > >>testing the entire Hadoop stack, including scale tests. This is
>> > > >>made possible only with the contribution from many many folks in
>> > > >>the community. Following
>> > people
>> > > >>contributed to this work: Ivan Mitic, Chuan Liu, Ramya Sunil,
>> > > >>Bikas
>> > Saha,
>> > > >>Kanna Karanam, John Gordon, Brandon Li, Chris Nauroth, David Lao,
>> > > Sumadhur
>> > > >>Reddy Bolli, Arpit Agarwal, Ahmed El Baz, Mike Liddell, Jing Zhao,
>> > Thejas
>> > > >>Nair, Steve Maine, Ganeshan Iyer, Raja Aluri, Giridharan Kesavan,
>> > > >>Ramya Bharathi Nimmagadda, Daryn Sharp, Arun Murthy, Tsz-Wo
>> > > >>Nicholas Sze,
>> > > Suresh
>> > > >>Srinivas and Sanjay Radia. There are many others who contributed
>> > > >>as
>> > well
>> > > >>providing feedback and comments on numerous jiras.
>> > > >>
>> > > >>The vote will run for seven days and will end on March 5, 6:00PM
>>PST.
>> > > >>
>> > > >>Regards,
>> > > >>Suresh
>> > > >>
>> > > >>
>> > > >>
>> > > >>
>> > > >>On Thu, Feb 7, 2013 at 6:41 PM, Mahadevan Venkatraman
>> > > >><ma...@microsoft.com>wrote:
>> > > >>
>> > > >>> It is super exciting to look at the prospect of these changes
>> > > >>>being merged  to trunk. Having Windows as one of the supported
>> > > >>>Hadoop platforms is
>> > a
>> > > >>> fantastic opportunity both for the Hadoop project and Microsoft
>> > > >>>customers.
>> > > >>>
>> > > >>> This work began around a year back when a few of us started with
>> > > >>> a
>> > > basic
>> > > >>> port of Hadoop on Windows. Ever since, the Hadoop team in
>> > > >>> Microsoft
>> > > have
>> > > >>> made significant progress in the following areas:
>> > > >>> (PS: Some of these items are already included in Suresh's email,
>> > > >>> but including again for completeness)
>> > > >>>
>> > > >>> - Command-line scripts for the Hadoop surface area
>> > > >>> - Mapping the HDFS permissions model to Windows
>> > > >>> - Abstracted and reconciled mismatches around differences in
>> > > >>> Path semantics in Java and Windows
>> > > >>> - Native Task Controller for Windows
>> > > >>> - Implementation of a Block Placement Policy to support cloud
>> > > >>> environments, more specifically Azure.
>> > > >>> - Implementation of Hadoop native libraries for Windows
>> > > >>> (compression codecs, native I/O) - Several reliability issues,
>> > > >>> including race-conditions, intermittent test failures, resource
>> leaks.
>> > > >>> - Several new unit test cases written for the above changes
>> > > >>>
>> > > >>> In the process, we have closely engaged with the Apache open
>> > > >>> source community and have got great support and assistance from
>> > > >>> the
>> > community
>> > > >>>in
>> > > >>> terms of contributing fixes, code review comments and commits.
>> > > >>>
>> > > >>> In addition, the Hadoop team at Microsoft has also made good
>> > > >>> progress
>> > > in
>> > > >>> other projects including Hive, Pig, Sqoop, Oozie, HCat and
>>HBase.
>> > Many
>> > > >>>of
>> > > >>> these changes have already been committed to the respective
>> > > >>>trunks
>> > with
>> > > >>> help from various committers and contributors. It is great to
>> > > >>> see the commitment of the community to support multiple
>> > > >>> platforms, and we
>> > look
>> > > >>> forward to the day when a developer/customer is able to
>> > > >>>successfully deploy  a complete solution stack based on Apache
>> > > >>>Hadoop releases.
>> > > >>>
>> > > >>> Next Steps:
>> > > >>>
>> > > >>> All of the above changes are part of the Windows Azure HDInsight
>> > > >>>and  HDInsight Server products from Microsoft. We have
>> > > >>>successfully on-boarded  several internal customers and have been
>> > > >>>running production workloads
>> > > on
>> > > >>> Windows Azure HDInsight. Our vision is to create a big data
>> > > >>>platform based  on Hadoop, and we are committed to helping make
>> > > >>>Hadoop a world-class  solution that anyone can use to solve their
>> > > >>>biggest data challenges.
>> > > >>>
>> > > >>> As an immediate next step, we would like to have a discussion
>> > > >>> around
>> > > how
>> > > >>> we can ensure that the quality of the mainline Hadoop branches
>> > > >>>on Windows  is maintained. To this end, we would like to get to
>> > > >>>the state where
>> > we
>> > > >>>have
>> > > >>> pre-checkin validation gates and nightly test runs enabled on
>> > Windows.
>> > > >>>If
>> > > >>> you have any suggestions around this, please do send an email.
>> > > >>>We
>> > are
>> > > >>> committed to helping sustain the long-term quality of Hadoop on
>> > > >>>both Linux  and Windows.
>> > > >>>
>> > > >>> We sincerely thank the community for their contribution and
>> > > >>> support
>> > so
>> > > >>> far. And hope to continue having a close engagement in the
>>future.
>> > > >>>
>> > > >>> -Microsoft HDInsight Team
>> > > >>>
>> > > >>>
>> > > >>> -----Original Message-----
>> > > >>> From: Suresh Srinivas [mailto:suresh@hortonworks.com]
>> > > >>> Sent: Thursday, February 7, 2013 5:42 PM
>> > > >>> To: common-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org;
>> > > >>> hdfs-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
>> > > >>> Subject: Heads up - merge branch-trunk-win to trunk
>> > > >>>
>> > > >>> The support for Hadoop on Windows was proposed in HADOOP-8079<
>> > > >>> https://issues.apache.org/jira/browse/HADOOP-8079> almost a year
>> > ago.
>> > > >>>The
>> > > >>> goal was to make Hadoop natively integrated, full-featured, and
>> > > >>>performance  and scalability tuned on Windows Server or Windows
>> > > >>>Azure.
>> > > >>> We are happy to announce that a lot of progress has been made in
>> > > >>>this  regard.
>> > > >>>
>> > > >>> Initial work started in a feature branch, branch-1-win, based on
>> > > >>>branch-1.
>> > > >>> The details related to the work done in the branch can be seen
>> > > >>>in  CHANGES.txt<
>> > > >>>
>> > > >>>
>> > > http://svn.apache.org/viewvc/hadoop/common/branches/branch-1-win/CHA
>> > > NGES
>> > .
>> > > >>>branch-1-win.txt?view=markup
>> > > >>> >.
>> > > >>> This work has been ported to a branch, branch-trunk-win, based
>> > > >>> on
>> > > trunk.
>> > > >>> Merge patch for this is available on
>> > > >>> HADOOP-8562<https://issues.apache.org/jira/browse/HADOOP-8562>
>> > > >>> .
>> > > >>>
>> > > >>> Highlights of the work done so far:
>> > > >>> 1. Necessary changes in Hadoop to run natively on Windows. These
>> > > changes
>> > > >>> handle differences in platforms related to path names,
>> > > >>>process/task  management etc.
>> > > >>> 2. Addition of winutils tools for managing file permissions and
>> > > >>>ownership,  user group mapping, hardlinks, symbolic links, chmod,
>> > > >>>disk
>> > utilization,
>> > > >>>and
>> > > >>> process/task management.
>> > > >>> 3. Added cmd scripts equivalent to existing shell scripts
>> > > >>>hadoop-daemon.sh, start and stop scripts.
>> > > >>> 4. Addition of block placement policy implemnation to support
>> > > >>>cloud  enviroment, more specifically Azure.
>> > > >>>
>> > > >>> We are very close to wrapping up the work in branch-trunk-win
>> > > >>>and getting  ready for a merge. Currently the merge patch is
>> > > >>>passing close to 100%
>> > > of
>> > > >>> unit tests on Linux. Soon I will call for a vote to merge this
>> > > >>>branch into  trunk.
>> > > >>>
>> > > >>> Next steps:
>> > > >>> 1. Call for vote to merge branch-trunk-win to trunk, when the
>> > > >>> work completes and precommit build is clean.
>> > > >>> 2. Start a discussion on adding Jenkins precommit builds on
>> > > >>> windows
>> > and
>> > > >>> how to integrate that with the existing commit process.
>> > > >>>
>> > > >>> Let me know if you have any questions.
>> > > >>>
>> > > >>> Regards,
>> > > >>> Suresh
>> > > >>>
>> > > >>>
>> > > >>
>> > > >>
>> > > >>--
>> > > >>http://hortonworks.com/download/
>> > > >
>> > >
>> >
>> >
>> >
>> > --
>> > http://hortonworks.com/download/
>> >
>>
>>
>>
>> --
>> Todd Lipcon
>> Software Engineer, Cloudera
>>
>>
>>
>>
>>

Re: [Vote] Merge branch-trunk-win to trunk

[Chris Nauroth <cn...@hortonworks.com>]

I'd like to share a few anecdotes about developing cross-platform,
hopefully to address some of the concerns about adding overhead to the
development process.  By reviewing past cases of cross-platform Linux vs.
Windows bugs, we can get a sense for how the development process could look
in the future.

HADOOP-9131: TestLocalFileSystem#testListStatusWithColons cannot run on
Windows.  As part of an earlier jira, HADOOP-8962, there was a new test
committed on trunk covering the case of a local file system interaction on
a file containing a ':'.  On Windows, ':' in a path has special meaning as
part of the drive specifier (i.e. C:), so this test cannot pass when
running on Windows.  In this kind of case, the cross-platform bug is
obvious, and the fix is obvious (assumeTrue(!Shell.WINDOWS)).  Ideally,
this would get fixed pre-commit after seeing a -1 from the Windows Jenkins
slave.

HDFS-4274: BlockPoolSliceScanner does not close verification log during
shutdown.  This caused problems for MiniDFSCluster-based tests running on
Windows.  Failure to close the verification log meant that we didn't
release file locks, so the tests couldn't delete/recreate working
directories during teardown/setup.  Arguably, this was always a bug, and
running on Windows just exposed it because of its stricter rules about file
locking.  This is a more complex fix, but it doesn't require
platform-specific knowledge.  If some future patch accidentally regresses
this, then we'll likely see +1 from Linux Jenkins and -1 from Windows
Jenkins.  Ideally, it would get fixed pre-commit, because it doesn't
require Windows-specific knowledge.  There is also the matter of impact.
 Re-breaking this would re-break many test suites on Windows.

HADOOP-9232: JniBasedUnixGroupsMappingWithFallback fails on Windows with
UnsatisfiedLinkError.  This was introduced by HADOOP-8712, which switched
to JniBasedUnixGroupsMappingWithFallback as the default
hadoop.security.group.mapping, but did not provide a Windows implementation
of the JNI function.  In this case, there was a strong desire to get
HADOOP-8712 into a release, fixing it on Windows required native Windows
API knowledge, and Windows users had a simple workaround available by
changing their configs back to ShellBasedUnixGroupsMapping.  I think this
is the kind of situation where we could allow HADOOP-8712 to commit despite
-1 from Windows Jenkins, with fairly quick follow-up from an engineer with
the Windows expertise to fix it.

To summarize, I don't think it needs to differ greatly from our current
development process.  We're all responsible for breadth of understanding
and maintenance of the whole codebase, but we also rely on specific
individuals with deep expertise in particular areas for certain issues.
 Sometimes we commit despite a -1 from Jenkins, based on the community's
judgment.

Virtualization greatly simplifies cross-platform development.  I use
VirtualBox on a Mac host and run VMs for Windows and Ubuntu with a shared
drive so that they can all see the same copy of the source code.  There are
plenty of variations on this depending on your preference, such as
offloading the VMs to a separate server or cloud service to free up local
RAM.  I'm planning on submitting BUILDING.txt changes later today that
fully describe how to build on Windows.  After some initial setup, it's
nearly identical to the mvn commands that you already use today.

Hope this helps,
--Chris


On Thu, Feb 28, 2013 at 3:25 AM, John Gordon <Jo...@microsoft.com>wrote:

> +1 (non-binding)
>
> I want to share my vote of confidence in this community.  If motivated to
> do so, this community can keep this project cross-platform and continue to
> rapidly innovate without breaking a sweat.
>
> The day we started working on this, I saw the foundations of greatness in
> the quality and volume of dev tests, the code itself, and the Apache values
> themselves.
>
> 1.) Hadoop's unit tests and their frameworks are very well thought out and
> the consideration and energy that went into their design is worthy of
> praise.  The MiniCluster abstractions utilize very few resources and put
> all the processes into one JVM for easy debugging.  It is very easy to
> select specific tests from the full suite to reproduce an issue reported in
> another environment - like the Jenkins build server or another
> contributor's environment.
> 2.) This community has done an excellent job of incorporating well-placed
> log messages to make it easy to post mortem troubleshoot most failures.
>  The logs are very useful, and it is extremely rare that troubleshooting a
> failure requires debugging a live repro.
> 3.) Hadoop is written primarily in Java, a cross-platform language that
> provides its own platform in the form of the JVM to insulate most of the
> code from the specifics of the OS layer.
> 4.) CoPDoC - The right priorities, and well stated.
>
>
> Thank you,
>
> John
>
> -----Original Message-----
> From: Ivan Mitic [mailto:ivanmi@microsoft.com]
> Sent: Wednesday, February 27, 2013 6:32 PM
> To: mapreduce-dev@hadoop.apache.org; common-dev@hadoop.apache.org
> Cc: yarn-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org
> Subject: RE: [Vote] Merge branch-trunk-win to trunk
>
> +1 (non-binding)
>
> I am really glad to see this happening! As people already mentioned, this
> has been a great engineering effort involving many people!
>
>
> Folks raised some valid concerns below and I thought it would be good to
> share my 2 cents. In my opinion, we don't have to solve all these problems
> right now. As we move forward with two platforms, we can start addressing
> one problem at a time and incrementally improve. In the first iteration,
> maintaining Hadoop on Windows could be just everyone trying to do their
> best effort (make sure Jenkins build succeeds at least). We already have
> people who are building/running trunk on Windows daily, so they would jump
> in and fix problems as needed (we've been doing this in branch-trunk-win
> for a while now). Although I see that the problems could arise with
> platform specific features/optimizations, I don't think these are frequent,
> so in most cases everything will just work. Merging the two branches sooner
> rather than later does seems like the right thing to do if the ultimate
> goal is to have Hadoop on both platforms. Now that the port has completed,
> we will have people in Microsoft (and elsewhere) wanting to contribute
> features/improvements to the trunk branch. A separate branch would just
> make things more difficult and confusing for everyone :) Hope this makes
> sense.
>
> -----Original Message-----
> From: Todd Lipcon [mailto:todd@cloudera.com]
> Sent: Wednesday, February 27, 2013 3:43 PM
> To: common-dev@hadoop.apache.org
> Cc: yarn-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org;
> mapreduce-dev@hadoop.apache.org
> Subject: Re: [Vote] Merge branch-trunk-win to trunk
>
> On Wed, Feb 27, 2013 at 2:54 PM, Suresh Srinivas <suresh@hortonworks.com
> >wrote:
>
> > With that we need to decide how our precommit process looks.
> > My inclination is to wait for +1 from precommit builds on both the
> > platforms to ensure no issues are introduced.
> > Thoughts?
> >
> > 2. Feature development impact
> > Some questions have been raised about would new features need to be
> > supported on both the platforms. Yes. I do not see a reason why
> > features cannot work on both the platforms, with the exception of
> > platform specific optimizations. This what Java gives us.
> >
> >
> I'm concerned about the above. Personally, I don't have access to any
> Windows boxes with development tools, and I know nothing about developing
> on Windows. The only Windows I run is an 8GB VM with 1 GB RAM allocated,
> for powerpoint :)
>
> If I submit a patch and it gets -1 "tests failed" on the Windows slave,
> how am I supposed to proceed?
>
> I think a reasonable compromise would be that the tests should always
> *build* on Windows before commit, and contributors should do their best to
> look at the test logs for any Windows-specific failures. But, beyond
> looking at the logs, a "-1 Tests failed on windows" should not block a
> commit.
>
> Those contributors who are interested in Windows being a first-class
> platform should be responsible for watching the Windows builds and
> debugging/fixing any regressions that might be Windows-specific.
>
> I also think the KDE model that Harsh pointed out is an interesting one --
> ie the idea that we would not merge windows support to trunk, but rather
> treat is as a "parallel code line" which lives in the ASF and has its own
> builds and releases. The windows team would periodically merge trunk->win
> to pick up any new changes, and do a separate test/release process. I'm not
> convinced this is the best idea, but worth discussion of pros and cons.
>
> -Todd
>
>
> >
> > On Wed, Feb 27, 2013 at 11:56 AM, Eli Collins <el...@cloudera.com> wrote:
> >
> > > Bobby raises some good questions.  A related one, since most current
> > > developers won't add Windows support for new features that are
> > > platform specific is it assumed that Windows development will either
> > > lag or will people actively work on keeping Windows up with the
> > > latest?  And vice versa in case Windows support is implemented first.
> > >
> > > Is there a jira for resolving the outstanding TODOs in the code base
> > > (similar to HDFS-2148)?  Looks like this merge doesn't introduce
> > > many which is great (just did a quick diff and grep).
> > >
> > > Thanks,
> > > Eli
> > >
> > > On Wed, Feb 27, 2013 at 8:17 AM, Robert Evans <ev...@yahoo-inc.com>
> > wrote:
> > > > After this is merged in is Windows still going to be a second
> > > > class citizen but happens to work for more than just development
> > > > or is it a fully supported platform where if something breaks it
> > > > can block a
> > > release?
> > > >  How do we as a community intend to keep Windows support from
> breaking?
> > > > We don't have any Jenkins slaves to be able to run nightly tests
> > > > to validate everything still compiles/runs.  This is not a blocker
> > > > for me because we often rely on individuals and groups to test
> > > > Hadoop, but I
> > do
> > > > think we need to have this discussion before we put it in.
> > > >
> > > > --Bobby
> > > >
> > > > On 2/26/13 4:55 PM, "Suresh Srinivas" <su...@hortonworks.com>
> wrote:
> > > >
> > > >>I had posted heads up about merging branch-trunk-win to trunk on
> > > >>Feb
> > 8th.
> > > >>I
> > > >>am happy to announce that we are ready for the merge.
> > > >>
> > > >>Here is a brief recap on the highlights of the work done:
> > > >>- Command-line scripts for the Hadoop surface area
> > > >>- Mapping the HDFS permissions model to Windows
> > > >>- Abstracted and reconciled mismatches around differences in Path
> > > >>semantics in Java and Windows
> > > >>- Native Task Controller for Windows
> > > >>- Implementation of a Block Placement Policy to support cloud
> > > >>environments, more specifically Azure.
> > > >>- Implementation of Hadoop native libraries for Windows
> > > >>(compression codecs, native I/O)
> > > >>- Several reliability issues, including race-conditions,
> > > >>intermittent
> > > test
> > > >>failures, resource leaks.
> > > >>- Several new unit test cases written for the above changes
> > > >>
> > > >>Please find the details of the work in
> > > >>CHANGES.branch-trunk-win.txt - Common
> > > >>changes<http://bit.ly/Xe7Ynv>, HDFS changes<
> > http://bit.ly/13QOSo9
> > > >,
> > > >>and YARN and MapReduce changes <http://bit.ly/128zzMt>. This is
> > > >>the
> > work
> > > >>ported from branch-1-win to a branch based on trunk.
> > > >>
> > > >>For details of the testing done, please see the thread -
> > > >>http://bit.ly/WpavJ4. Merge patch for this is available on
> > HADOOP-8562<
> > > >>https://issues.apache.org/jira/browse/HADOOP-8562>.
> > > >>
> > > >>This was a large undertaking that involved developing code,
> > > >>testing the entire Hadoop stack, including scale tests. This is
> > > >>made possible only with the contribution from many many folks in
> > > >>the community. Following
> > people
> > > >>contributed to this work: Ivan Mitic, Chuan Liu, Ramya Sunil,
> > > >>Bikas
> > Saha,
> > > >>Kanna Karanam, John Gordon, Brandon Li, Chris Nauroth, David Lao,
> > > Sumadhur
> > > >>Reddy Bolli, Arpit Agarwal, Ahmed El Baz, Mike Liddell, Jing Zhao,
> > Thejas
> > > >>Nair, Steve Maine, Ganeshan Iyer, Raja Aluri, Giridharan Kesavan,
> > > >>Ramya Bharathi Nimmagadda, Daryn Sharp, Arun Murthy, Tsz-Wo
> > > >>Nicholas Sze,
> > > Suresh
> > > >>Srinivas and Sanjay Radia. There are many others who contributed
> > > >>as
> > well
> > > >>providing feedback and comments on numerous jiras.
> > > >>
> > > >>The vote will run for seven days and will end on March 5, 6:00PM PST.
> > > >>
> > > >>Regards,
> > > >>Suresh
> > > >>
> > > >>
> > > >>
> > > >>
> > > >>On Thu, Feb 7, 2013 at 6:41 PM, Mahadevan Venkatraman
> > > >><ma...@microsoft.com>wrote:
> > > >>
> > > >>> It is super exciting to look at the prospect of these changes
> > > >>>being merged  to trunk. Having Windows as one of the supported
> > > >>>Hadoop platforms is
> > a
> > > >>> fantastic opportunity both for the Hadoop project and Microsoft
> > > >>>customers.
> > > >>>
> > > >>> This work began around a year back when a few of us started with
> > > >>> a
> > > basic
> > > >>> port of Hadoop on Windows. Ever since, the Hadoop team in
> > > >>> Microsoft
> > > have
> > > >>> made significant progress in the following areas:
> > > >>> (PS: Some of these items are already included in Suresh's email,
> > > >>> but including again for completeness)
> > > >>>
> > > >>> - Command-line scripts for the Hadoop surface area
> > > >>> - Mapping the HDFS permissions model to Windows
> > > >>> - Abstracted and reconciled mismatches around differences in
> > > >>> Path semantics in Java and Windows
> > > >>> - Native Task Controller for Windows
> > > >>> - Implementation of a Block Placement Policy to support cloud
> > > >>> environments, more specifically Azure.
> > > >>> - Implementation of Hadoop native libraries for Windows
> > > >>> (compression codecs, native I/O) - Several reliability issues,
> > > >>> including race-conditions, intermittent test failures, resource
> leaks.
> > > >>> - Several new unit test cases written for the above changes
> > > >>>
> > > >>> In the process, we have closely engaged with the Apache open
> > > >>> source community and have got great support and assistance from
> > > >>> the
> > community
> > > >>>in
> > > >>> terms of contributing fixes, code review comments and commits.
> > > >>>
> > > >>> In addition, the Hadoop team at Microsoft has also made good
> > > >>> progress
> > > in
> > > >>> other projects including Hive, Pig, Sqoop, Oozie, HCat and HBase.
> > Many
> > > >>>of
> > > >>> these changes have already been committed to the respective
> > > >>>trunks
> > with
> > > >>> help from various committers and contributors. It is great to
> > > >>> see the commitment of the community to support multiple
> > > >>> platforms, and we
> > look
> > > >>> forward to the day when a developer/customer is able to
> > > >>>successfully deploy  a complete solution stack based on Apache
> > > >>>Hadoop releases.
> > > >>>
> > > >>> Next Steps:
> > > >>>
> > > >>> All of the above changes are part of the Windows Azure HDInsight
> > > >>>and  HDInsight Server products from Microsoft. We have
> > > >>>successfully on-boarded  several internal customers and have been
> > > >>>running production workloads
> > > on
> > > >>> Windows Azure HDInsight. Our vision is to create a big data
> > > >>>platform based  on Hadoop, and we are committed to helping make
> > > >>>Hadoop a world-class  solution that anyone can use to solve their
> > > >>>biggest data challenges.
> > > >>>
> > > >>> As an immediate next step, we would like to have a discussion
> > > >>> around
> > > how
> > > >>> we can ensure that the quality of the mainline Hadoop branches
> > > >>>on Windows  is maintained. To this end, we would like to get to
> > > >>>the state where
> > we
> > > >>>have
> > > >>> pre-checkin validation gates and nightly test runs enabled on
> > Windows.
> > > >>>If
> > > >>> you have any suggestions around this, please do send an email.
> > > >>>We
> > are
> > > >>> committed to helping sustain the long-term quality of Hadoop on
> > > >>>both Linux  and Windows.
> > > >>>
> > > >>> We sincerely thank the community for their contribution and
> > > >>> support
> > so
> > > >>> far. And hope to continue having a close engagement in the future.
> > > >>>
> > > >>> -Microsoft HDInsight Team
> > > >>>
> > > >>>
> > > >>> -----Original Message-----
> > > >>> From: Suresh Srinivas [mailto:suresh@hortonworks.com]
> > > >>> Sent: Thursday, February 7, 2013 5:42 PM
> > > >>> To: common-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org;
> > > >>> hdfs-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
> > > >>> Subject: Heads up - merge branch-trunk-win to trunk
> > > >>>
> > > >>> The support for Hadoop on Windows was proposed in HADOOP-8079<
> > > >>> https://issues.apache.org/jira/browse/HADOOP-8079> almost a year
> > ago.
> > > >>>The
> > > >>> goal was to make Hadoop natively integrated, full-featured, and
> > > >>>performance  and scalability tuned on Windows Server or Windows
> > > >>>Azure.
> > > >>> We are happy to announce that a lot of progress has been made in
> > > >>>this  regard.
> > > >>>
> > > >>> Initial work started in a feature branch, branch-1-win, based on
> > > >>>branch-1.
> > > >>> The details related to the work done in the branch can be seen
> > > >>>in  CHANGES.txt<
> > > >>>
> > > >>>
> > > http://svn.apache.org/viewvc/hadoop/common/branches/branch-1-win/CHA
> > > NGES
> > .
> > > >>>branch-1-win.txt?view=markup
> > > >>> >.
> > > >>> This work has been ported to a branch, branch-trunk-win, based
> > > >>> on
> > > trunk.
> > > >>> Merge patch for this is available on
> > > >>> HADOOP-8562<https://issues.apache.org/jira/browse/HADOOP-8562>
> > > >>> .
> > > >>>
> > > >>> Highlights of the work done so far:
> > > >>> 1. Necessary changes in Hadoop to run natively on Windows. These
> > > changes
> > > >>> handle differences in platforms related to path names,
> > > >>>process/task  management etc.
> > > >>> 2. Addition of winutils tools for managing file permissions and
> > > >>>ownership,  user group mapping, hardlinks, symbolic links, chmod,
> > > >>>disk
> > utilization,
> > > >>>and
> > > >>> process/task management.
> > > >>> 3. Added cmd scripts equivalent to existing shell scripts
> > > >>>hadoop-daemon.sh, start and stop scripts.
> > > >>> 4. Addition of block placement policy implemnation to support
> > > >>>cloud  enviroment, more specifically Azure.
> > > >>>
> > > >>> We are very close to wrapping up the work in branch-trunk-win
> > > >>>and getting  ready for a merge. Currently the merge patch is
> > > >>>passing close to 100%
> > > of
> > > >>> unit tests on Linux. Soon I will call for a vote to merge this
> > > >>>branch into  trunk.
> > > >>>
> > > >>> Next steps:
> > > >>> 1. Call for vote to merge branch-trunk-win to trunk, when the
> > > >>> work completes and precommit build is clean.
> > > >>> 2. Start a discussion on adding Jenkins precommit builds on
> > > >>> windows
> > and
> > > >>> how to integrate that with the existing commit process.
> > > >>>
> > > >>> Let me know if you have any questions.
> > > >>>
> > > >>> Regards,
> > > >>> Suresh
> > > >>>
> > > >>>
> > > >>
> > > >>
> > > >>--
> > > >>http://hortonworks.com/download/
> > > >
> > >
> >
> >
> >
> > --
> > http://hortonworks.com/download/
> >
>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
>
>
>
>
>

Re: [Vote] Merge branch-trunk-win to trunk

[Chris Nauroth <cn...@hortonworks.com>]

I'd like to share a few anecdotes about developing cross-platform,
hopefully to address some of the concerns about adding overhead to the
development process.  By reviewing past cases of cross-platform Linux vs.
Windows bugs, we can get a sense for how the development process could look
in the future.

HADOOP-9131: TestLocalFileSystem#testListStatusWithColons cannot run on
Windows.  As part of an earlier jira, HADOOP-8962, there was a new test
committed on trunk covering the case of a local file system interaction on
a file containing a ':'.  On Windows, ':' in a path has special meaning as
part of the drive specifier (i.e. C:), so this test cannot pass when
running on Windows.  In this kind of case, the cross-platform bug is
obvious, and the fix is obvious (assumeTrue(!Shell.WINDOWS)).  Ideally,
this would get fixed pre-commit after seeing a -1 from the Windows Jenkins
slave.

HDFS-4274: BlockPoolSliceScanner does not close verification log during
shutdown.  This caused problems for MiniDFSCluster-based tests running on
Windows.  Failure to close the verification log meant that we didn't
release file locks, so the tests couldn't delete/recreate working
directories during teardown/setup.  Arguably, this was always a bug, and
running on Windows just exposed it because of its stricter rules about file
locking.  This is a more complex fix, but it doesn't require
platform-specific knowledge.  If some future patch accidentally regresses
this, then we'll likely see +1 from Linux Jenkins and -1 from Windows
Jenkins.  Ideally, it would get fixed pre-commit, because it doesn't
require Windows-specific knowledge.  There is also the matter of impact.
 Re-breaking this would re-break many test suites on Windows.

HADOOP-9232: JniBasedUnixGroupsMappingWithFallback fails on Windows with
UnsatisfiedLinkError.  This was introduced by HADOOP-8712, which switched
to JniBasedUnixGroupsMappingWithFallback as the default
hadoop.security.group.mapping, but did not provide a Windows implementation
of the JNI function.  In this case, there was a strong desire to get
HADOOP-8712 into a release, fixing it on Windows required native Windows
API knowledge, and Windows users had a simple workaround available by
changing their configs back to ShellBasedUnixGroupsMapping.  I think this
is the kind of situation where we could allow HADOOP-8712 to commit despite
-1 from Windows Jenkins, with fairly quick follow-up from an engineer with
the Windows expertise to fix it.

To summarize, I don't think it needs to differ greatly from our current
development process.  We're all responsible for breadth of understanding
and maintenance of the whole codebase, but we also rely on specific
individuals with deep expertise in particular areas for certain issues.
 Sometimes we commit despite a -1 from Jenkins, based on the community's
judgment.

Virtualization greatly simplifies cross-platform development.  I use
VirtualBox on a Mac host and run VMs for Windows and Ubuntu with a shared
drive so that they can all see the same copy of the source code.  There are
plenty of variations on this depending on your preference, such as
offloading the VMs to a separate server or cloud service to free up local
RAM.  I'm planning on submitting BUILDING.txt changes later today that
fully describe how to build on Windows.  After some initial setup, it's
nearly identical to the mvn commands that you already use today.

Hope this helps,
--Chris


On Thu, Feb 28, 2013 at 3:25 AM, John Gordon <Jo...@microsoft.com>wrote:

> +1 (non-binding)
>
> I want to share my vote of confidence in this community.  If motivated to
> do so, this community can keep this project cross-platform and continue to
> rapidly innovate without breaking a sweat.
>
> The day we started working on this, I saw the foundations of greatness in
> the quality and volume of dev tests, the code itself, and the Apache values
> themselves.
>
> 1.) Hadoop's unit tests and their frameworks are very well thought out and
> the consideration and energy that went into their design is worthy of
> praise.  The MiniCluster abstractions utilize very few resources and put
> all the processes into one JVM for easy debugging.  It is very easy to
> select specific tests from the full suite to reproduce an issue reported in
> another environment - like the Jenkins build server or another
> contributor's environment.
> 2.) This community has done an excellent job of incorporating well-placed
> log messages to make it easy to post mortem troubleshoot most failures.
>  The logs are very useful, and it is extremely rare that troubleshooting a
> failure requires debugging a live repro.
> 3.) Hadoop is written primarily in Java, a cross-platform language that
> provides its own platform in the form of the JVM to insulate most of the
> code from the specifics of the OS layer.
> 4.) CoPDoC - The right priorities, and well stated.
>
>
> Thank you,
>
> John
>
> -----Original Message-----
> From: Ivan Mitic [mailto:ivanmi@microsoft.com]
> Sent: Wednesday, February 27, 2013 6:32 PM
> To: mapreduce-dev@hadoop.apache.org; common-dev@hadoop.apache.org
> Cc: yarn-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org
> Subject: RE: [Vote] Merge branch-trunk-win to trunk
>
> +1 (non-binding)
>
> I am really glad to see this happening! As people already mentioned, this
> has been a great engineering effort involving many people!
>
>
> Folks raised some valid concerns below and I thought it would be good to
> share my 2 cents. In my opinion, we don't have to solve all these problems
> right now. As we move forward with two platforms, we can start addressing
> one problem at a time and incrementally improve. In the first iteration,
> maintaining Hadoop on Windows could be just everyone trying to do their
> best effort (make sure Jenkins build succeeds at least). We already have
> people who are building/running trunk on Windows daily, so they would jump
> in and fix problems as needed (we've been doing this in branch-trunk-win
> for a while now). Although I see that the problems could arise with
> platform specific features/optimizations, I don't think these are frequent,
> so in most cases everything will just work. Merging the two branches sooner
> rather than later does seems like the right thing to do if the ultimate
> goal is to have Hadoop on both platforms. Now that the port has completed,
> we will have people in Microsoft (and elsewhere) wanting to contribute
> features/improvements to the trunk branch. A separate branch would just
> make things more difficult and confusing for everyone :) Hope this makes
> sense.
>
> -----Original Message-----
> From: Todd Lipcon [mailto:todd@cloudera.com]
> Sent: Wednesday, February 27, 2013 3:43 PM
> To: common-dev@hadoop.apache.org
> Cc: yarn-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org;
> mapreduce-dev@hadoop.apache.org
> Subject: Re: [Vote] Merge branch-trunk-win to trunk
>
> On Wed, Feb 27, 2013 at 2:54 PM, Suresh Srinivas <suresh@hortonworks.com
> >wrote:
>
> > With that we need to decide how our precommit process looks.
> > My inclination is to wait for +1 from precommit builds on both the
> > platforms to ensure no issues are introduced.
> > Thoughts?
> >
> > 2. Feature development impact
> > Some questions have been raised about would new features need to be
> > supported on both the platforms. Yes. I do not see a reason why
> > features cannot work on both the platforms, with the exception of
> > platform specific optimizations. This what Java gives us.
> >
> >
> I'm concerned about the above. Personally, I don't have access to any
> Windows boxes with development tools, and I know nothing about developing
> on Windows. The only Windows I run is an 8GB VM with 1 GB RAM allocated,
> for powerpoint :)
>
> If I submit a patch and it gets -1 "tests failed" on the Windows slave,
> how am I supposed to proceed?
>
> I think a reasonable compromise would be that the tests should always
> *build* on Windows before commit, and contributors should do their best to
> look at the test logs for any Windows-specific failures. But, beyond
> looking at the logs, a "-1 Tests failed on windows" should not block a
> commit.
>
> Those contributors who are interested in Windows being a first-class
> platform should be responsible for watching the Windows builds and
> debugging/fixing any regressions that might be Windows-specific.
>
> I also think the KDE model that Harsh pointed out is an interesting one --
> ie the idea that we would not merge windows support to trunk, but rather
> treat is as a "parallel code line" which lives in the ASF and has its own
> builds and releases. The windows team would periodically merge trunk->win
> to pick up any new changes, and do a separate test/release process. I'm not
> convinced this is the best idea, but worth discussion of pros and cons.
>
> -Todd
>
>
> >
> > On Wed, Feb 27, 2013 at 11:56 AM, Eli Collins <el...@cloudera.com> wrote:
> >
> > > Bobby raises some good questions.  A related one, since most current
> > > developers won't add Windows support for new features that are
> > > platform specific is it assumed that Windows development will either
> > > lag or will people actively work on keeping Windows up with the
> > > latest?  And vice versa in case Windows support is implemented first.
> > >
> > > Is there a jira for resolving the outstanding TODOs in the code base
> > > (similar to HDFS-2148)?  Looks like this merge doesn't introduce
> > > many which is great (just did a quick diff and grep).
> > >
> > > Thanks,
> > > Eli
> > >
> > > On Wed, Feb 27, 2013 at 8:17 AM, Robert Evans <ev...@yahoo-inc.com>
> > wrote:
> > > > After this is merged in is Windows still going to be a second
> > > > class citizen but happens to work for more than just development
> > > > or is it a fully supported platform where if something breaks it
> > > > can block a
> > > release?
> > > >  How do we as a community intend to keep Windows support from
> breaking?
> > > > We don't have any Jenkins slaves to be able to run nightly tests
> > > > to validate everything still compiles/runs.  This is not a blocker
> > > > for me because we often rely on individuals and groups to test
> > > > Hadoop, but I
> > do
> > > > think we need to have this discussion before we put it in.
> > > >
> > > > --Bobby
> > > >
> > > > On 2/26/13 4:55 PM, "Suresh Srinivas" <su...@hortonworks.com>
> wrote:
> > > >
> > > >>I had posted heads up about merging branch-trunk-win to trunk on
> > > >>Feb
> > 8th.
> > > >>I
> > > >>am happy to announce that we are ready for the merge.
> > > >>
> > > >>Here is a brief recap on the highlights of the work done:
> > > >>- Command-line scripts for the Hadoop surface area
> > > >>- Mapping the HDFS permissions model to Windows
> > > >>- Abstracted and reconciled mismatches around differences in Path
> > > >>semantics in Java and Windows
> > > >>- Native Task Controller for Windows
> > > >>- Implementation of a Block Placement Policy to support cloud
> > > >>environments, more specifically Azure.
> > > >>- Implementation of Hadoop native libraries for Windows
> > > >>(compression codecs, native I/O)
> > > >>- Several reliability issues, including race-conditions,
> > > >>intermittent
> > > test
> > > >>failures, resource leaks.
> > > >>- Several new unit test cases written for the above changes
> > > >>
> > > >>Please find the details of the work in
> > > >>CHANGES.branch-trunk-win.txt - Common
> > > >>changes<http://bit.ly/Xe7Ynv>, HDFS changes<
> > http://bit.ly/13QOSo9
> > > >,
> > > >>and YARN and MapReduce changes <http://bit.ly/128zzMt>. This is
> > > >>the
> > work
> > > >>ported from branch-1-win to a branch based on trunk.
> > > >>
> > > >>For details of the testing done, please see the thread -
> > > >>http://bit.ly/WpavJ4. Merge patch for this is available on
> > HADOOP-8562<
> > > >>https://issues.apache.org/jira/browse/HADOOP-8562>.
> > > >>
> > > >>This was a large undertaking that involved developing code,
> > > >>testing the entire Hadoop stack, including scale tests. This is
> > > >>made possible only with the contribution from many many folks in
> > > >>the community. Following
> > people
> > > >>contributed to this work: Ivan Mitic, Chuan Liu, Ramya Sunil,
> > > >>Bikas
> > Saha,
> > > >>Kanna Karanam, John Gordon, Brandon Li, Chris Nauroth, David Lao,
> > > Sumadhur
> > > >>Reddy Bolli, Arpit Agarwal, Ahmed El Baz, Mike Liddell, Jing Zhao,
> > Thejas
> > > >>Nair, Steve Maine, Ganeshan Iyer, Raja Aluri, Giridharan Kesavan,
> > > >>Ramya Bharathi Nimmagadda, Daryn Sharp, Arun Murthy, Tsz-Wo
> > > >>Nicholas Sze,
> > > Suresh
> > > >>Srinivas and Sanjay Radia. There are many others who contributed
> > > >>as
> > well
> > > >>providing feedback and comments on numerous jiras.
> > > >>
> > > >>The vote will run for seven days and will end on March 5, 6:00PM PST.
> > > >>
> > > >>Regards,
> > > >>Suresh
> > > >>
> > > >>
> > > >>
> > > >>
> > > >>On Thu, Feb 7, 2013 at 6:41 PM, Mahadevan Venkatraman
> > > >><ma...@microsoft.com>wrote:
> > > >>
> > > >>> It is super exciting to look at the prospect of these changes
> > > >>>being merged  to trunk. Having Windows as one of the supported
> > > >>>Hadoop platforms is
> > a
> > > >>> fantastic opportunity both for the Hadoop project and Microsoft
> > > >>>customers.
> > > >>>
> > > >>> This work began around a year back when a few of us started with
> > > >>> a
> > > basic
> > > >>> port of Hadoop on Windows. Ever since, the Hadoop team in
> > > >>> Microsoft
> > > have
> > > >>> made significant progress in the following areas:
> > > >>> (PS: Some of these items are already included in Suresh's email,
> > > >>> but including again for completeness)
> > > >>>
> > > >>> - Command-line scripts for the Hadoop surface area
> > > >>> - Mapping the HDFS permissions model to Windows
> > > >>> - Abstracted and reconciled mismatches around differences in
> > > >>> Path semantics in Java and Windows
> > > >>> - Native Task Controller for Windows
> > > >>> - Implementation of a Block Placement Policy to support cloud
> > > >>> environments, more specifically Azure.
> > > >>> - Implementation of Hadoop native libraries for Windows
> > > >>> (compression codecs, native I/O) - Several reliability issues,
> > > >>> including race-conditions, intermittent test failures, resource
> leaks.
> > > >>> - Several new unit test cases written for the above changes
> > > >>>
> > > >>> In the process, we have closely engaged with the Apache open
> > > >>> source community and have got great support and assistance from
> > > >>> the
> > community
> > > >>>in
> > > >>> terms of contributing fixes, code review comments and commits.
> > > >>>
> > > >>> In addition, the Hadoop team at Microsoft has also made good
> > > >>> progress
> > > in
> > > >>> other projects including Hive, Pig, Sqoop, Oozie, HCat and HBase.
> > Many
> > > >>>of
> > > >>> these changes have already been committed to the respective
> > > >>>trunks
> > with
> > > >>> help from various committers and contributors. It is great to
> > > >>> see the commitment of the community to support multiple
> > > >>> platforms, and we
> > look
> > > >>> forward to the day when a developer/customer is able to
> > > >>>successfully deploy  a complete solution stack based on Apache
> > > >>>Hadoop releases.
> > > >>>
> > > >>> Next Steps:
> > > >>>
> > > >>> All of the above changes are part of the Windows Azure HDInsight
> > > >>>and  HDInsight Server products from Microsoft. We have
> > > >>>successfully on-boarded  several internal customers and have been
> > > >>>running production workloads
> > > on
> > > >>> Windows Azure HDInsight. Our vision is to create a big data
> > > >>>platform based  on Hadoop, and we are committed to helping make
> > > >>>Hadoop a world-class  solution that anyone can use to solve their
> > > >>>biggest data challenges.
> > > >>>
> > > >>> As an immediate next step, we would like to have a discussion
> > > >>> around
> > > how
> > > >>> we can ensure that the quality of the mainline Hadoop branches
> > > >>>on Windows  is maintained. To this end, we would like to get to
> > > >>>the state where
> > we
> > > >>>have
> > > >>> pre-checkin validation gates and nightly test runs enabled on
> > Windows.
> > > >>>If
> > > >>> you have any suggestions around this, please do send an email.
> > > >>>We
> > are
> > > >>> committed to helping sustain the long-term quality of Hadoop on
> > > >>>both Linux  and Windows.
> > > >>>
> > > >>> We sincerely thank the community for their contribution and
> > > >>> support
> > so
> > > >>> far. And hope to continue having a close engagement in the future.
> > > >>>
> > > >>> -Microsoft HDInsight Team
> > > >>>
> > > >>>
> > > >>> -----Original Message-----
> > > >>> From: Suresh Srinivas [mailto:suresh@hortonworks.com]
> > > >>> Sent: Thursday, February 7, 2013 5:42 PM
> > > >>> To: common-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org;
> > > >>> hdfs-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
> > > >>> Subject: Heads up - merge branch-trunk-win to trunk
> > > >>>
> > > >>> The support for Hadoop on Windows was proposed in HADOOP-8079<
> > > >>> https://issues.apache.org/jira/browse/HADOOP-8079> almost a year
> > ago.
> > > >>>The
> > > >>> goal was to make Hadoop natively integrated, full-featured, and
> > > >>>performance  and scalability tuned on Windows Server or Windows
> > > >>>Azure.
> > > >>> We are happy to announce that a lot of progress has been made in
> > > >>>this  regard.
> > > >>>
> > > >>> Initial work started in a feature branch, branch-1-win, based on
> > > >>>branch-1.
> > > >>> The details related to the work done in the branch can be seen
> > > >>>in  CHANGES.txt<
> > > >>>
> > > >>>
> > > http://svn.apache.org/viewvc/hadoop/common/branches/branch-1-win/CHA
> > > NGES
> > .
> > > >>>branch-1-win.txt?view=markup
> > > >>> >.
> > > >>> This work has been ported to a branch, branch-trunk-win, based
> > > >>> on
> > > trunk.
> > > >>> Merge patch for this is available on
> > > >>> HADOOP-8562<https://issues.apache.org/jira/browse/HADOOP-8562>
> > > >>> .
> > > >>>
> > > >>> Highlights of the work done so far:
> > > >>> 1. Necessary changes in Hadoop to run natively on Windows. These
> > > changes
> > > >>> handle differences in platforms related to path names,
> > > >>>process/task  management etc.
> > > >>> 2. Addition of winutils tools for managing file permissions and
> > > >>>ownership,  user group mapping, hardlinks, symbolic links, chmod,
> > > >>>disk
> > utilization,
> > > >>>and
> > > >>> process/task management.
> > > >>> 3. Added cmd scripts equivalent to existing shell scripts
> > > >>>hadoop-daemon.sh, start and stop scripts.
> > > >>> 4. Addition of block placement policy implemnation to support
> > > >>>cloud  enviroment, more specifically Azure.
> > > >>>
> > > >>> We are very close to wrapping up the work in branch-trunk-win
> > > >>>and getting  ready for a merge. Currently the merge patch is
> > > >>>passing close to 100%
> > > of
> > > >>> unit tests on Linux. Soon I will call for a vote to merge this
> > > >>>branch into  trunk.
> > > >>>
> > > >>> Next steps:
> > > >>> 1. Call for vote to merge branch-trunk-win to trunk, when the
> > > >>> work completes and precommit build is clean.
> > > >>> 2. Start a discussion on adding Jenkins precommit builds on
> > > >>> windows
> > and
> > > >>> how to integrate that with the existing commit process.
> > > >>>
> > > >>> Let me know if you have any questions.
> > > >>>
> > > >>> Regards,
> > > >>> Suresh
> > > >>>
> > > >>>
> > > >>
> > > >>
> > > >>--
> > > >>http://hortonworks.com/download/
> > > >
> > >
> >
> >
> >
> > --
> > http://hortonworks.com/download/
> >
>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
>
>
>
>
>

Re: [Vote] Merge branch-trunk-win to trunk

[Chris Nauroth <cn...@hortonworks.com>]

I'd like to share a few anecdotes about developing cross-platform,
hopefully to address some of the concerns about adding overhead to the
development process.  By reviewing past cases of cross-platform Linux vs.
Windows bugs, we can get a sense for how the development process could look
in the future.

HADOOP-9131: TestLocalFileSystem#testListStatusWithColons cannot run on
Windows.  As part of an earlier jira, HADOOP-8962, there was a new test
committed on trunk covering the case of a local file system interaction on
a file containing a ':'.  On Windows, ':' in a path has special meaning as
part of the drive specifier (i.e. C:), so this test cannot pass when
running on Windows.  In this kind of case, the cross-platform bug is
obvious, and the fix is obvious (assumeTrue(!Shell.WINDOWS)).  Ideally,
this would get fixed pre-commit after seeing a -1 from the Windows Jenkins
slave.

HDFS-4274: BlockPoolSliceScanner does not close verification log during
shutdown.  This caused problems for MiniDFSCluster-based tests running on
Windows.  Failure to close the verification log meant that we didn't
release file locks, so the tests couldn't delete/recreate working
directories during teardown/setup.  Arguably, this was always a bug, and
running on Windows just exposed it because of its stricter rules about file
locking.  This is a more complex fix, but it doesn't require
platform-specific knowledge.  If some future patch accidentally regresses
this, then we'll likely see +1 from Linux Jenkins and -1 from Windows
Jenkins.  Ideally, it would get fixed pre-commit, because it doesn't
require Windows-specific knowledge.  There is also the matter of impact.
 Re-breaking this would re-break many test suites on Windows.

HADOOP-9232: JniBasedUnixGroupsMappingWithFallback fails on Windows with
UnsatisfiedLinkError.  This was introduced by HADOOP-8712, which switched
to JniBasedUnixGroupsMappingWithFallback as the default
hadoop.security.group.mapping, but did not provide a Windows implementation
of the JNI function.  In this case, there was a strong desire to get
HADOOP-8712 into a release, fixing it on Windows required native Windows
API knowledge, and Windows users had a simple workaround available by
changing their configs back to ShellBasedUnixGroupsMapping.  I think this
is the kind of situation where we could allow HADOOP-8712 to commit despite
-1 from Windows Jenkins, with fairly quick follow-up from an engineer with
the Windows expertise to fix it.

To summarize, I don't think it needs to differ greatly from our current
development process.  We're all responsible for breadth of understanding
and maintenance of the whole codebase, but we also rely on specific
individuals with deep expertise in particular areas for certain issues.
 Sometimes we commit despite a -1 from Jenkins, based on the community's
judgment.

Virtualization greatly simplifies cross-platform development.  I use
VirtualBox on a Mac host and run VMs for Windows and Ubuntu with a shared
drive so that they can all see the same copy of the source code.  There are
plenty of variations on this depending on your preference, such as
offloading the VMs to a separate server or cloud service to free up local
RAM.  I'm planning on submitting BUILDING.txt changes later today that
fully describe how to build on Windows.  After some initial setup, it's
nearly identical to the mvn commands that you already use today.

Hope this helps,
--Chris


On Thu, Feb 28, 2013 at 3:25 AM, John Gordon <Jo...@microsoft.com>wrote:

> +1 (non-binding)
>
> I want to share my vote of confidence in this community.  If motivated to
> do so, this community can keep this project cross-platform and continue to
> rapidly innovate without breaking a sweat.
>
> The day we started working on this, I saw the foundations of greatness in
> the quality and volume of dev tests, the code itself, and the Apache values
> themselves.
>
> 1.) Hadoop's unit tests and their frameworks are very well thought out and
> the consideration and energy that went into their design is worthy of
> praise.  The MiniCluster abstractions utilize very few resources and put
> all the processes into one JVM for easy debugging.  It is very easy to
> select specific tests from the full suite to reproduce an issue reported in
> another environment - like the Jenkins build server or another
> contributor's environment.
> 2.) This community has done an excellent job of incorporating well-placed
> log messages to make it easy to post mortem troubleshoot most failures.
>  The logs are very useful, and it is extremely rare that troubleshooting a
> failure requires debugging a live repro.
> 3.) Hadoop is written primarily in Java, a cross-platform language that
> provides its own platform in the form of the JVM to insulate most of the
> code from the specifics of the OS layer.
> 4.) CoPDoC - The right priorities, and well stated.
>
>
> Thank you,
>
> John
>
> -----Original Message-----
> From: Ivan Mitic [mailto:ivanmi@microsoft.com]
> Sent: Wednesday, February 27, 2013 6:32 PM
> To: mapreduce-dev@hadoop.apache.org; common-dev@hadoop.apache.org
> Cc: yarn-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org
> Subject: RE: [Vote] Merge branch-trunk-win to trunk
>
> +1 (non-binding)
>
> I am really glad to see this happening! As people already mentioned, this
> has been a great engineering effort involving many people!
>
>
> Folks raised some valid concerns below and I thought it would be good to
> share my 2 cents. In my opinion, we don't have to solve all these problems
> right now. As we move forward with two platforms, we can start addressing
> one problem at a time and incrementally improve. In the first iteration,
> maintaining Hadoop on Windows could be just everyone trying to do their
> best effort (make sure Jenkins build succeeds at least). We already have
> people who are building/running trunk on Windows daily, so they would jump
> in and fix problems as needed (we've been doing this in branch-trunk-win
> for a while now). Although I see that the problems could arise with
> platform specific features/optimizations, I don't think these are frequent,
> so in most cases everything will just work. Merging the two branches sooner
> rather than later does seems like the right thing to do if the ultimate
> goal is to have Hadoop on both platforms. Now that the port has completed,
> we will have people in Microsoft (and elsewhere) wanting to contribute
> features/improvements to the trunk branch. A separate branch would just
> make things more difficult and confusing for everyone :) Hope this makes
> sense.
>
> -----Original Message-----
> From: Todd Lipcon [mailto:todd@cloudera.com]
> Sent: Wednesday, February 27, 2013 3:43 PM
> To: common-dev@hadoop.apache.org
> Cc: yarn-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org;
> mapreduce-dev@hadoop.apache.org
> Subject: Re: [Vote] Merge branch-trunk-win to trunk
>
> On Wed, Feb 27, 2013 at 2:54 PM, Suresh Srinivas <suresh@hortonworks.com
> >wrote:
>
> > With that we need to decide how our precommit process looks.
> > My inclination is to wait for +1 from precommit builds on both the
> > platforms to ensure no issues are introduced.
> > Thoughts?
> >
> > 2. Feature development impact
> > Some questions have been raised about would new features need to be
> > supported on both the platforms. Yes. I do not see a reason why
> > features cannot work on both the platforms, with the exception of
> > platform specific optimizations. This what Java gives us.
> >
> >
> I'm concerned about the above. Personally, I don't have access to any
> Windows boxes with development tools, and I know nothing about developing
> on Windows. The only Windows I run is an 8GB VM with 1 GB RAM allocated,
> for powerpoint :)
>
> If I submit a patch and it gets -1 "tests failed" on the Windows slave,
> how am I supposed to proceed?
>
> I think a reasonable compromise would be that the tests should always
> *build* on Windows before commit, and contributors should do their best to
> look at the test logs for any Windows-specific failures. But, beyond
> looking at the logs, a "-1 Tests failed on windows" should not block a
> commit.
>
> Those contributors who are interested in Windows being a first-class
> platform should be responsible for watching the Windows builds and
> debugging/fixing any regressions that might be Windows-specific.
>
> I also think the KDE model that Harsh pointed out is an interesting one --
> ie the idea that we would not merge windows support to trunk, but rather
> treat is as a "parallel code line" which lives in the ASF and has its own
> builds and releases. The windows team would periodically merge trunk->win
> to pick up any new changes, and do a separate test/release process. I'm not
> convinced this is the best idea, but worth discussion of pros and cons.
>
> -Todd
>
>
> >
> > On Wed, Feb 27, 2013 at 11:56 AM, Eli Collins <el...@cloudera.com> wrote:
> >
> > > Bobby raises some good questions.  A related one, since most current
> > > developers won't add Windows support for new features that are
> > > platform specific is it assumed that Windows development will either
> > > lag or will people actively work on keeping Windows up with the
> > > latest?  And vice versa in case Windows support is implemented first.
> > >
> > > Is there a jira for resolving the outstanding TODOs in the code base
> > > (similar to HDFS-2148)?  Looks like this merge doesn't introduce
> > > many which is great (just did a quick diff and grep).
> > >
> > > Thanks,
> > > Eli
> > >
> > > On Wed, Feb 27, 2013 at 8:17 AM, Robert Evans <ev...@yahoo-inc.com>
> > wrote:
> > > > After this is merged in is Windows still going to be a second
> > > > class citizen but happens to work for more than just development
> > > > or is it a fully supported platform where if something breaks it
> > > > can block a
> > > release?
> > > >  How do we as a community intend to keep Windows support from
> breaking?
> > > > We don't have any Jenkins slaves to be able to run nightly tests
> > > > to validate everything still compiles/runs.  This is not a blocker
> > > > for me because we often rely on individuals and groups to test
> > > > Hadoop, but I
> > do
> > > > think we need to have this discussion before we put it in.
> > > >
> > > > --Bobby
> > > >
> > > > On 2/26/13 4:55 PM, "Suresh Srinivas" <su...@hortonworks.com>
> wrote:
> > > >
> > > >>I had posted heads up about merging branch-trunk-win to trunk on
> > > >>Feb
> > 8th.
> > > >>I
> > > >>am happy to announce that we are ready for the merge.
> > > >>
> > > >>Here is a brief recap on the highlights of the work done:
> > > >>- Command-line scripts for the Hadoop surface area
> > > >>- Mapping the HDFS permissions model to Windows
> > > >>- Abstracted and reconciled mismatches around differences in Path
> > > >>semantics in Java and Windows
> > > >>- Native Task Controller for Windows
> > > >>- Implementation of a Block Placement Policy to support cloud
> > > >>environments, more specifically Azure.
> > > >>- Implementation of Hadoop native libraries for Windows
> > > >>(compression codecs, native I/O)
> > > >>- Several reliability issues, including race-conditions,
> > > >>intermittent
> > > test
> > > >>failures, resource leaks.
> > > >>- Several new unit test cases written for the above changes
> > > >>
> > > >>Please find the details of the work in
> > > >>CHANGES.branch-trunk-win.txt - Common
> > > >>changes<http://bit.ly/Xe7Ynv>, HDFS changes<
> > http://bit.ly/13QOSo9
> > > >,
> > > >>and YARN and MapReduce changes <http://bit.ly/128zzMt>. This is
> > > >>the
> > work
> > > >>ported from branch-1-win to a branch based on trunk.
> > > >>
> > > >>For details of the testing done, please see the thread -
> > > >>http://bit.ly/WpavJ4. Merge patch for this is available on
> > HADOOP-8562<
> > > >>https://issues.apache.org/jira/browse/HADOOP-8562>.
> > > >>
> > > >>This was a large undertaking that involved developing code,
> > > >>testing the entire Hadoop stack, including scale tests. This is
> > > >>made possible only with the contribution from many many folks in
> > > >>the community. Following
> > people
> > > >>contributed to this work: Ivan Mitic, Chuan Liu, Ramya Sunil,
> > > >>Bikas
> > Saha,
> > > >>Kanna Karanam, John Gordon, Brandon Li, Chris Nauroth, David Lao,
> > > Sumadhur
> > > >>Reddy Bolli, Arpit Agarwal, Ahmed El Baz, Mike Liddell, Jing Zhao,
> > Thejas
> > > >>Nair, Steve Maine, Ganeshan Iyer, Raja Aluri, Giridharan Kesavan,
> > > >>Ramya Bharathi Nimmagadda, Daryn Sharp, Arun Murthy, Tsz-Wo
> > > >>Nicholas Sze,
> > > Suresh
> > > >>Srinivas and Sanjay Radia. There are many others who contributed
> > > >>as
> > well
> > > >>providing feedback and comments on numerous jiras.
> > > >>
> > > >>The vote will run for seven days and will end on March 5, 6:00PM PST.
> > > >>
> > > >>Regards,
> > > >>Suresh
> > > >>
> > > >>
> > > >>
> > > >>
> > > >>On Thu, Feb 7, 2013 at 6:41 PM, Mahadevan Venkatraman
> > > >><ma...@microsoft.com>wrote:
> > > >>
> > > >>> It is super exciting to look at the prospect of these changes
> > > >>>being merged  to trunk. Having Windows as one of the supported
> > > >>>Hadoop platforms is
> > a
> > > >>> fantastic opportunity both for the Hadoop project and Microsoft
> > > >>>customers.
> > > >>>
> > > >>> This work began around a year back when a few of us started with
> > > >>> a
> > > basic
> > > >>> port of Hadoop on Windows. Ever since, the Hadoop team in
> > > >>> Microsoft
> > > have
> > > >>> made significant progress in the following areas:
> > > >>> (PS: Some of these items are already included in Suresh's email,
> > > >>> but including again for completeness)
> > > >>>
> > > >>> - Command-line scripts for the Hadoop surface area
> > > >>> - Mapping the HDFS permissions model to Windows
> > > >>> - Abstracted and reconciled mismatches around differences in
> > > >>> Path semantics in Java and Windows
> > > >>> - Native Task Controller for Windows
> > > >>> - Implementation of a Block Placement Policy to support cloud
> > > >>> environments, more specifically Azure.
> > > >>> - Implementation of Hadoop native libraries for Windows
> > > >>> (compression codecs, native I/O) - Several reliability issues,
> > > >>> including race-conditions, intermittent test failures, resource
> leaks.
> > > >>> - Several new unit test cases written for the above changes
> > > >>>
> > > >>> In the process, we have closely engaged with the Apache open
> > > >>> source community and have got great support and assistance from
> > > >>> the
> > community
> > > >>>in
> > > >>> terms of contributing fixes, code review comments and commits.
> > > >>>
> > > >>> In addition, the Hadoop team at Microsoft has also made good
> > > >>> progress
> > > in
> > > >>> other projects including Hive, Pig, Sqoop, Oozie, HCat and HBase.
> > Many
> > > >>>of
> > > >>> these changes have already been committed to the respective
> > > >>>trunks
> > with
> > > >>> help from various committers and contributors. It is great to
> > > >>> see the commitment of the community to support multiple
> > > >>> platforms, and we
> > look
> > > >>> forward to the day when a developer/customer is able to
> > > >>>successfully deploy  a complete solution stack based on Apache
> > > >>>Hadoop releases.
> > > >>>
> > > >>> Next Steps:
> > > >>>
> > > >>> All of the above changes are part of the Windows Azure HDInsight
> > > >>>and  HDInsight Server products from Microsoft. We have
> > > >>>successfully on-boarded  several internal customers and have been
> > > >>>running production workloads
> > > on
> > > >>> Windows Azure HDInsight. Our vision is to create a big data
> > > >>>platform based  on Hadoop, and we are committed to helping make
> > > >>>Hadoop a world-class  solution that anyone can use to solve their
> > > >>>biggest data challenges.
> > > >>>
> > > >>> As an immediate next step, we would like to have a discussion
> > > >>> around
> > > how
> > > >>> we can ensure that the quality of the mainline Hadoop branches
> > > >>>on Windows  is maintained. To this end, we would like to get to
> > > >>>the state where
> > we
> > > >>>have
> > > >>> pre-checkin validation gates and nightly test runs enabled on
> > Windows.
> > > >>>If
> > > >>> you have any suggestions around this, please do send an email.
> > > >>>We
> > are
> > > >>> committed to helping sustain the long-term quality of Hadoop on
> > > >>>both Linux  and Windows.
> > > >>>
> > > >>> We sincerely thank the community for their contribution and
> > > >>> support
> > so
> > > >>> far. And hope to continue having a close engagement in the future.
> > > >>>
> > > >>> -Microsoft HDInsight Team
> > > >>>
> > > >>>
> > > >>> -----Original Message-----
> > > >>> From: Suresh Srinivas [mailto:suresh@hortonworks.com]
> > > >>> Sent: Thursday, February 7, 2013 5:42 PM
> > > >>> To: common-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org;
> > > >>> hdfs-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
> > > >>> Subject: Heads up - merge branch-trunk-win to trunk
> > > >>>
> > > >>> The support for Hadoop on Windows was proposed in HADOOP-8079<
> > > >>> https://issues.apache.org/jira/browse/HADOOP-8079> almost a year
> > ago.
> > > >>>The
> > > >>> goal was to make Hadoop natively integrated, full-featured, and
> > > >>>performance  and scalability tuned on Windows Server or Windows
> > > >>>Azure.
> > > >>> We are happy to announce that a lot of progress has been made in
> > > >>>this  regard.
> > > >>>
> > > >>> Initial work started in a feature branch, branch-1-win, based on
> > > >>>branch-1.
> > > >>> The details related to the work done in the branch can be seen
> > > >>>in  CHANGES.txt<
> > > >>>
> > > >>>
> > > http://svn.apache.org/viewvc/hadoop/common/branches/branch-1-win/CHA
> > > NGES
> > .
> > > >>>branch-1-win.txt?view=markup
> > > >>> >.
> > > >>> This work has been ported to a branch, branch-trunk-win, based
> > > >>> on
> > > trunk.
> > > >>> Merge patch for this is available on
> > > >>> HADOOP-8562<https://issues.apache.org/jira/browse/HADOOP-8562>
> > > >>> .
> > > >>>
> > > >>> Highlights of the work done so far:
> > > >>> 1. Necessary changes in Hadoop to run natively on Windows. These
> > > changes
> > > >>> handle differences in platforms related to path names,
> > > >>>process/task  management etc.
> > > >>> 2. Addition of winutils tools for managing file permissions and
> > > >>>ownership,  user group mapping, hardlinks, symbolic links, chmod,
> > > >>>disk
> > utilization,
> > > >>>and
> > > >>> process/task management.
> > > >>> 3. Added cmd scripts equivalent to existing shell scripts
> > > >>>hadoop-daemon.sh, start and stop scripts.
> > > >>> 4. Addition of block placement policy implemnation to support
> > > >>>cloud  enviroment, more specifically Azure.
> > > >>>
> > > >>> We are very close to wrapping up the work in branch-trunk-win
> > > >>>and getting  ready for a merge. Currently the merge patch is
> > > >>>passing close to 100%
> > > of
> > > >>> unit tests on Linux. Soon I will call for a vote to merge this
> > > >>>branch into  trunk.
> > > >>>
> > > >>> Next steps:
> > > >>> 1. Call for vote to merge branch-trunk-win to trunk, when the
> > > >>> work completes and precommit build is clean.
> > > >>> 2. Start a discussion on adding Jenkins precommit builds on
> > > >>> windows
> > and
> > > >>> how to integrate that with the existing commit process.
> > > >>>
> > > >>> Let me know if you have any questions.
> > > >>>
> > > >>> Regards,
> > > >>> Suresh
> > > >>>
> > > >>>
> > > >>
> > > >>
> > > >>--
> > > >>http://hortonworks.com/download/
> > > >
> > >
> >
> >
> >
> > --
> > http://hortonworks.com/download/
> >
>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
>
>
>
>
>

Re: [Vote] Merge branch-trunk-win to trunk

[Chris Nauroth <cn...@hortonworks.com>]

I'd like to share a few anecdotes about developing cross-platform,
hopefully to address some of the concerns about adding overhead to the
development process.  By reviewing past cases of cross-platform Linux vs.
Windows bugs, we can get a sense for how the development process could look
in the future.

HADOOP-9131: TestLocalFileSystem#testListStatusWithColons cannot run on
Windows.  As part of an earlier jira, HADOOP-8962, there was a new test
committed on trunk covering the case of a local file system interaction on
a file containing a ':'.  On Windows, ':' in a path has special meaning as
part of the drive specifier (i.e. C:), so this test cannot pass when
running on Windows.  In this kind of case, the cross-platform bug is
obvious, and the fix is obvious (assumeTrue(!Shell.WINDOWS)).  Ideally,
this would get fixed pre-commit after seeing a -1 from the Windows Jenkins
slave.

HDFS-4274: BlockPoolSliceScanner does not close verification log during
shutdown.  This caused problems for MiniDFSCluster-based tests running on
Windows.  Failure to close the verification log meant that we didn't
release file locks, so the tests couldn't delete/recreate working
directories during teardown/setup.  Arguably, this was always a bug, and
running on Windows just exposed it because of its stricter rules about file
locking.  This is a more complex fix, but it doesn't require
platform-specific knowledge.  If some future patch accidentally regresses
this, then we'll likely see +1 from Linux Jenkins and -1 from Windows
Jenkins.  Ideally, it would get fixed pre-commit, because it doesn't
require Windows-specific knowledge.  There is also the matter of impact.
 Re-breaking this would re-break many test suites on Windows.

HADOOP-9232: JniBasedUnixGroupsMappingWithFallback fails on Windows with
UnsatisfiedLinkError.  This was introduced by HADOOP-8712, which switched
to JniBasedUnixGroupsMappingWithFallback as the default
hadoop.security.group.mapping, but did not provide a Windows implementation
of the JNI function.  In this case, there was a strong desire to get
HADOOP-8712 into a release, fixing it on Windows required native Windows
API knowledge, and Windows users had a simple workaround available by
changing their configs back to ShellBasedUnixGroupsMapping.  I think this
is the kind of situation where we could allow HADOOP-8712 to commit despite
-1 from Windows Jenkins, with fairly quick follow-up from an engineer with
the Windows expertise to fix it.

To summarize, I don't think it needs to differ greatly from our current
development process.  We're all responsible for breadth of understanding
and maintenance of the whole codebase, but we also rely on specific
individuals with deep expertise in particular areas for certain issues.
 Sometimes we commit despite a -1 from Jenkins, based on the community's
judgment.

Virtualization greatly simplifies cross-platform development.  I use
VirtualBox on a Mac host and run VMs for Windows and Ubuntu with a shared
drive so that they can all see the same copy of the source code.  There are
plenty of variations on this depending on your preference, such as
offloading the VMs to a separate server or cloud service to free up local
RAM.  I'm planning on submitting BUILDING.txt changes later today that
fully describe how to build on Windows.  After some initial setup, it's
nearly identical to the mvn commands that you already use today.

Hope this helps,
--Chris


On Thu, Feb 28, 2013 at 3:25 AM, John Gordon <Jo...@microsoft.com>wrote:

> +1 (non-binding)
>
> I want to share my vote of confidence in this community.  If motivated to
> do so, this community can keep this project cross-platform and continue to
> rapidly innovate without breaking a sweat.
>
> The day we started working on this, I saw the foundations of greatness in
> the quality and volume of dev tests, the code itself, and the Apache values
> themselves.
>
> 1.) Hadoop's unit tests and their frameworks are very well thought out and
> the consideration and energy that went into their design is worthy of
> praise.  The MiniCluster abstractions utilize very few resources and put
> all the processes into one JVM for easy debugging.  It is very easy to
> select specific tests from the full suite to reproduce an issue reported in
> another environment - like the Jenkins build server or another
> contributor's environment.
> 2.) This community has done an excellent job of incorporating well-placed
> log messages to make it easy to post mortem troubleshoot most failures.
>  The logs are very useful, and it is extremely rare that troubleshooting a
> failure requires debugging a live repro.
> 3.) Hadoop is written primarily in Java, a cross-platform language that
> provides its own platform in the form of the JVM to insulate most of the
> code from the specifics of the OS layer.
> 4.) CoPDoC - The right priorities, and well stated.
>
>
> Thank you,
>
> John
>
> -----Original Message-----
> From: Ivan Mitic [mailto:ivanmi@microsoft.com]
> Sent: Wednesday, February 27, 2013 6:32 PM
> To: mapreduce-dev@hadoop.apache.org; common-dev@hadoop.apache.org
> Cc: yarn-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org
> Subject: RE: [Vote] Merge branch-trunk-win to trunk
>
> +1 (non-binding)
>
> I am really glad to see this happening! As people already mentioned, this
> has been a great engineering effort involving many people!
>
>
> Folks raised some valid concerns below and I thought it would be good to
> share my 2 cents. In my opinion, we don't have to solve all these problems
> right now. As we move forward with two platforms, we can start addressing
> one problem at a time and incrementally improve. In the first iteration,
> maintaining Hadoop on Windows could be just everyone trying to do their
> best effort (make sure Jenkins build succeeds at least). We already have
> people who are building/running trunk on Windows daily, so they would jump
> in and fix problems as needed (we've been doing this in branch-trunk-win
> for a while now). Although I see that the problems could arise with
> platform specific features/optimizations, I don't think these are frequent,
> so in most cases everything will just work. Merging the two branches sooner
> rather than later does seems like the right thing to do if the ultimate
> goal is to have Hadoop on both platforms. Now that the port has completed,
> we will have people in Microsoft (and elsewhere) wanting to contribute
> features/improvements to the trunk branch. A separate branch would just
> make things more difficult and confusing for everyone :) Hope this makes
> sense.
>
> -----Original Message-----
> From: Todd Lipcon [mailto:todd@cloudera.com]
> Sent: Wednesday, February 27, 2013 3:43 PM
> To: common-dev@hadoop.apache.org
> Cc: yarn-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org;
> mapreduce-dev@hadoop.apache.org
> Subject: Re: [Vote] Merge branch-trunk-win to trunk
>
> On Wed, Feb 27, 2013 at 2:54 PM, Suresh Srinivas <suresh@hortonworks.com
> >wrote:
>
> > With that we need to decide how our precommit process looks.
> > My inclination is to wait for +1 from precommit builds on both the
> > platforms to ensure no issues are introduced.
> > Thoughts?
> >
> > 2. Feature development impact
> > Some questions have been raised about would new features need to be
> > supported on both the platforms. Yes. I do not see a reason why
> > features cannot work on both the platforms, with the exception of
> > platform specific optimizations. This what Java gives us.
> >
> >
> I'm concerned about the above. Personally, I don't have access to any
> Windows boxes with development tools, and I know nothing about developing
> on Windows. The only Windows I run is an 8GB VM with 1 GB RAM allocated,
> for powerpoint :)
>
> If I submit a patch and it gets -1 "tests failed" on the Windows slave,
> how am I supposed to proceed?
>
> I think a reasonable compromise would be that the tests should always
> *build* on Windows before commit, and contributors should do their best to
> look at the test logs for any Windows-specific failures. But, beyond
> looking at the logs, a "-1 Tests failed on windows" should not block a
> commit.
>
> Those contributors who are interested in Windows being a first-class
> platform should be responsible for watching the Windows builds and
> debugging/fixing any regressions that might be Windows-specific.
>
> I also think the KDE model that Harsh pointed out is an interesting one --
> ie the idea that we would not merge windows support to trunk, but rather
> treat is as a "parallel code line" which lives in the ASF and has its own
> builds and releases. The windows team would periodically merge trunk->win
> to pick up any new changes, and do a separate test/release process. I'm not
> convinced this is the best idea, but worth discussion of pros and cons.
>
> -Todd
>
>
> >
> > On Wed, Feb 27, 2013 at 11:56 AM, Eli Collins <el...@cloudera.com> wrote:
> >
> > > Bobby raises some good questions.  A related one, since most current
> > > developers won't add Windows support for new features that are
> > > platform specific is it assumed that Windows development will either
> > > lag or will people actively work on keeping Windows up with the
> > > latest?  And vice versa in case Windows support is implemented first.
> > >
> > > Is there a jira for resolving the outstanding TODOs in the code base
> > > (similar to HDFS-2148)?  Looks like this merge doesn't introduce
> > > many which is great (just did a quick diff and grep).
> > >
> > > Thanks,
> > > Eli
> > >
> > > On Wed, Feb 27, 2013 at 8:17 AM, Robert Evans <ev...@yahoo-inc.com>
> > wrote:
> > > > After this is merged in is Windows still going to be a second
> > > > class citizen but happens to work for more than just development
> > > > or is it a fully supported platform where if something breaks it
> > > > can block a
> > > release?
> > > >  How do we as a community intend to keep Windows support from
> breaking?
> > > > We don't have any Jenkins slaves to be able to run nightly tests
> > > > to validate everything still compiles/runs.  This is not a blocker
> > > > for me because we often rely on individuals and groups to test
> > > > Hadoop, but I
> > do
> > > > think we need to have this discussion before we put it in.
> > > >
> > > > --Bobby
> > > >
> > > > On 2/26/13 4:55 PM, "Suresh Srinivas" <su...@hortonworks.com>
> wrote:
> > > >
> > > >>I had posted heads up about merging branch-trunk-win to trunk on
> > > >>Feb
> > 8th.
> > > >>I
> > > >>am happy to announce that we are ready for the merge.
> > > >>
> > > >>Here is a brief recap on the highlights of the work done:
> > > >>- Command-line scripts for the Hadoop surface area
> > > >>- Mapping the HDFS permissions model to Windows
> > > >>- Abstracted and reconciled mismatches around differences in Path
> > > >>semantics in Java and Windows
> > > >>- Native Task Controller for Windows
> > > >>- Implementation of a Block Placement Policy to support cloud
> > > >>environments, more specifically Azure.
> > > >>- Implementation of Hadoop native libraries for Windows
> > > >>(compression codecs, native I/O)
> > > >>- Several reliability issues, including race-conditions,
> > > >>intermittent
> > > test
> > > >>failures, resource leaks.
> > > >>- Several new unit test cases written for the above changes
> > > >>
> > > >>Please find the details of the work in
> > > >>CHANGES.branch-trunk-win.txt - Common
> > > >>changes<http://bit.ly/Xe7Ynv>, HDFS changes<
> > http://bit.ly/13QOSo9
> > > >,
> > > >>and YARN and MapReduce changes <http://bit.ly/128zzMt>. This is
> > > >>the
> > work
> > > >>ported from branch-1-win to a branch based on trunk.
> > > >>
> > > >>For details of the testing done, please see the thread -
> > > >>http://bit.ly/WpavJ4. Merge patch for this is available on
> > HADOOP-8562<
> > > >>https://issues.apache.org/jira/browse/HADOOP-8562>.
> > > >>
> > > >>This was a large undertaking that involved developing code,
> > > >>testing the entire Hadoop stack, including scale tests. This is
> > > >>made possible only with the contribution from many many folks in
> > > >>the community. Following
> > people
> > > >>contributed to this work: Ivan Mitic, Chuan Liu, Ramya Sunil,
> > > >>Bikas
> > Saha,
> > > >>Kanna Karanam, John Gordon, Brandon Li, Chris Nauroth, David Lao,
> > > Sumadhur
> > > >>Reddy Bolli, Arpit Agarwal, Ahmed El Baz, Mike Liddell, Jing Zhao,
> > Thejas
> > > >>Nair, Steve Maine, Ganeshan Iyer, Raja Aluri, Giridharan Kesavan,
> > > >>Ramya Bharathi Nimmagadda, Daryn Sharp, Arun Murthy, Tsz-Wo
> > > >>Nicholas Sze,
> > > Suresh
> > > >>Srinivas and Sanjay Radia. There are many others who contributed
> > > >>as
> > well
> > > >>providing feedback and comments on numerous jiras.
> > > >>
> > > >>The vote will run for seven days and will end on March 5, 6:00PM PST.
> > > >>
> > > >>Regards,
> > > >>Suresh
> > > >>
> > > >>
> > > >>
> > > >>
> > > >>On Thu, Feb 7, 2013 at 6:41 PM, Mahadevan Venkatraman
> > > >><ma...@microsoft.com>wrote:
> > > >>
> > > >>> It is super exciting to look at the prospect of these changes
> > > >>>being merged  to trunk. Having Windows as one of the supported
> > > >>>Hadoop platforms is
> > a
> > > >>> fantastic opportunity both for the Hadoop project and Microsoft
> > > >>>customers.
> > > >>>
> > > >>> This work began around a year back when a few of us started with
> > > >>> a
> > > basic
> > > >>> port of Hadoop on Windows. Ever since, the Hadoop team in
> > > >>> Microsoft
> > > have
> > > >>> made significant progress in the following areas:
> > > >>> (PS: Some of these items are already included in Suresh's email,
> > > >>> but including again for completeness)
> > > >>>
> > > >>> - Command-line scripts for the Hadoop surface area
> > > >>> - Mapping the HDFS permissions model to Windows
> > > >>> - Abstracted and reconciled mismatches around differences in
> > > >>> Path semantics in Java and Windows
> > > >>> - Native Task Controller for Windows
> > > >>> - Implementation of a Block Placement Policy to support cloud
> > > >>> environments, more specifically Azure.
> > > >>> - Implementation of Hadoop native libraries for Windows
> > > >>> (compression codecs, native I/O) - Several reliability issues,
> > > >>> including race-conditions, intermittent test failures, resource
> leaks.
> > > >>> - Several new unit test cases written for the above changes
> > > >>>
> > > >>> In the process, we have closely engaged with the Apache open
> > > >>> source community and have got great support and assistance from
> > > >>> the
> > community
> > > >>>in
> > > >>> terms of contributing fixes, code review comments and commits.
> > > >>>
> > > >>> In addition, the Hadoop team at Microsoft has also made good
> > > >>> progress
> > > in
> > > >>> other projects including Hive, Pig, Sqoop, Oozie, HCat and HBase.
> > Many
> > > >>>of
> > > >>> these changes have already been committed to the respective
> > > >>>trunks
> > with
> > > >>> help from various committers and contributors. It is great to
> > > >>> see the commitment of the community to support multiple
> > > >>> platforms, and we
> > look
> > > >>> forward to the day when a developer/customer is able to
> > > >>>successfully deploy  a complete solution stack based on Apache
> > > >>>Hadoop releases.
> > > >>>
> > > >>> Next Steps:
> > > >>>
> > > >>> All of the above changes are part of the Windows Azure HDInsight
> > > >>>and  HDInsight Server products from Microsoft. We have
> > > >>>successfully on-boarded  several internal customers and have been
> > > >>>running production workloads
> > > on
> > > >>> Windows Azure HDInsight. Our vision is to create a big data
> > > >>>platform based  on Hadoop, and we are committed to helping make
> > > >>>Hadoop a world-class  solution that anyone can use to solve their
> > > >>>biggest data challenges.
> > > >>>
> > > >>> As an immediate next step, we would like to have a discussion
> > > >>> around
> > > how
> > > >>> we can ensure that the quality of the mainline Hadoop branches
> > > >>>on Windows  is maintained. To this end, we would like to get to
> > > >>>the state where
> > we
> > > >>>have
> > > >>> pre-checkin validation gates and nightly test runs enabled on
> > Windows.
> > > >>>If
> > > >>> you have any suggestions around this, please do send an email.
> > > >>>We
> > are
> > > >>> committed to helping sustain the long-term quality of Hadoop on
> > > >>>both Linux  and Windows.
> > > >>>
> > > >>> We sincerely thank the community for their contribution and
> > > >>> support
> > so
> > > >>> far. And hope to continue having a close engagement in the future.
> > > >>>
> > > >>> -Microsoft HDInsight Team
> > > >>>
> > > >>>
> > > >>> -----Original Message-----
> > > >>> From: Suresh Srinivas [mailto:suresh@hortonworks.com]
> > > >>> Sent: Thursday, February 7, 2013 5:42 PM
> > > >>> To: common-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org;
> > > >>> hdfs-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
> > > >>> Subject: Heads up - merge branch-trunk-win to trunk
> > > >>>
> > > >>> The support for Hadoop on Windows was proposed in HADOOP-8079<
> > > >>> https://issues.apache.org/jira/browse/HADOOP-8079> almost a year
> > ago.
> > > >>>The
> > > >>> goal was to make Hadoop natively integrated, full-featured, and
> > > >>>performance  and scalability tuned on Windows Server or Windows
> > > >>>Azure.
> > > >>> We are happy to announce that a lot of progress has been made in
> > > >>>this  regard.
> > > >>>
> > > >>> Initial work started in a feature branch, branch-1-win, based on
> > > >>>branch-1.
> > > >>> The details related to the work done in the branch can be seen
> > > >>>in  CHANGES.txt<
> > > >>>
> > > >>>
> > > http://svn.apache.org/viewvc/hadoop/common/branches/branch-1-win/CHA
> > > NGES
> > .
> > > >>>branch-1-win.txt?view=markup
> > > >>> >.
> > > >>> This work has been ported to a branch, branch-trunk-win, based
> > > >>> on
> > > trunk.
> > > >>> Merge patch for this is available on
> > > >>> HADOOP-8562<https://issues.apache.org/jira/browse/HADOOP-8562>
> > > >>> .
> > > >>>
> > > >>> Highlights of the work done so far:
> > > >>> 1. Necessary changes in Hadoop to run natively on Windows. These
> > > changes
> > > >>> handle differences in platforms related to path names,
> > > >>>process/task  management etc.
> > > >>> 2. Addition of winutils tools for managing file permissions and
> > > >>>ownership,  user group mapping, hardlinks, symbolic links, chmod,
> > > >>>disk
> > utilization,
> > > >>>and
> > > >>> process/task management.
> > > >>> 3. Added cmd scripts equivalent to existing shell scripts
> > > >>>hadoop-daemon.sh, start and stop scripts.
> > > >>> 4. Addition of block placement policy implemnation to support
> > > >>>cloud  enviroment, more specifically Azure.
> > > >>>
> > > >>> We are very close to wrapping up the work in branch-trunk-win
> > > >>>and getting  ready for a merge. Currently the merge patch is
> > > >>>passing close to 100%
> > > of
> > > >>> unit tests on Linux. Soon I will call for a vote to merge this
> > > >>>branch into  trunk.
> > > >>>
> > > >>> Next steps:
> > > >>> 1. Call for vote to merge branch-trunk-win to trunk, when the
> > > >>> work completes and precommit build is clean.
> > > >>> 2. Start a discussion on adding Jenkins precommit builds on
> > > >>> windows
> > and
> > > >>> how to integrate that with the existing commit process.
> > > >>>
> > > >>> Let me know if you have any questions.
> > > >>>
> > > >>> Regards,
> > > >>> Suresh
> > > >>>
> > > >>>
> > > >>
> > > >>
> > > >>--
> > > >>http://hortonworks.com/download/
> > > >
> > >
> >
> >
> >
> > --
> > http://hortonworks.com/download/
> >
>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
>
>
>
>
>

RE: [Vote] Merge branch-trunk-win to trunk

[John Gordon <Jo...@microsoft.com>]

+1 (non-binding)

I want to share my vote of confidence in this community.  If motivated to do so, this community can keep this project cross-platform and continue to rapidly innovate without breaking a sweat.

The day we started working on this, I saw the foundations of greatness in the quality and volume of dev tests, the code itself, and the Apache values themselves.

1.) Hadoop's unit tests and their frameworks are very well thought out and the consideration and energy that went into their design is worthy of praise.  The MiniCluster abstractions utilize very few resources and put all the processes into one JVM for easy debugging.  It is very easy to select specific tests from the full suite to reproduce an issue reported in another environment - like the Jenkins build server or another contributor's environment.  
2.) This community has done an excellent job of incorporating well-placed log messages to make it easy to post mortem troubleshoot most failures.  The logs are very useful, and it is extremely rare that troubleshooting a failure requires debugging a live repro.
3.) Hadoop is written primarily in Java, a cross-platform language that provides its own platform in the form of the JVM to insulate most of the code from the specifics of the OS layer.
4.) CoPDoC - The right priorities, and well stated.


Thank you,

John

-----Original Message-----
From: Ivan Mitic [mailto:ivanmi@microsoft.com] 
Sent: Wednesday, February 27, 2013 6:32 PM
To: mapreduce-dev@hadoop.apache.org; common-dev@hadoop.apache.org
Cc: yarn-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org
Subject: RE: [Vote] Merge branch-trunk-win to trunk

+1 (non-binding)

I am really glad to see this happening! As people already mentioned, this has been a great engineering effort involving many people!


Folks raised some valid concerns below and I thought it would be good to share my 2 cents. In my opinion, we don't have to solve all these problems right now. As we move forward with two platforms, we can start addressing one problem at a time and incrementally improve. In the first iteration, maintaining Hadoop on Windows could be just everyone trying to do their best effort (make sure Jenkins build succeeds at least). We already have people who are building/running trunk on Windows daily, so they would jump in and fix problems as needed (we've been doing this in branch-trunk-win for a while now). Although I see that the problems could arise with platform specific features/optimizations, I don't think these are frequent, so in most cases everything will just work. Merging the two branches sooner rather than later does seems like the right thing to do if the ultimate goal is to have Hadoop on both platforms. Now that the port has completed, we will have people in Microsoft (and elsewhere) wanting to contribute features/improvements to the trunk branch. A separate branch would just make things more difficult and confusing for everyone :) Hope this makes sense.

-----Original Message-----
From: Todd Lipcon [mailto:todd@cloudera.com]
Sent: Wednesday, February 27, 2013 3:43 PM
To: common-dev@hadoop.apache.org
Cc: yarn-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
Subject: Re: [Vote] Merge branch-trunk-win to trunk

On Wed, Feb 27, 2013 at 2:54 PM, Suresh Srinivas <su...@hortonworks.com>wrote:

> With that we need to decide how our precommit process looks.
> My inclination is to wait for +1 from precommit builds on both the 
> platforms to ensure no issues are introduced.
> Thoughts?
>
> 2. Feature development impact
> Some questions have been raised about would new features need to be 
> supported on both the platforms. Yes. I do not see a reason why 
> features cannot work on both the platforms, with the exception of 
> platform specific optimizations. This what Java gives us.
>
>
I'm concerned about the above. Personally, I don't have access to any Windows boxes with development tools, and I know nothing about developing on Windows. The only Windows I run is an 8GB VM with 1 GB RAM allocated, for powerpoint :)

If I submit a patch and it gets -1 "tests failed" on the Windows slave, how am I supposed to proceed?

I think a reasonable compromise would be that the tests should always
*build* on Windows before commit, and contributors should do their best to look at the test logs for any Windows-specific failures. But, beyond looking at the logs, a "-1 Tests failed on windows" should not block a commit.

Those contributors who are interested in Windows being a first-class platform should be responsible for watching the Windows builds and debugging/fixing any regressions that might be Windows-specific.

I also think the KDE model that Harsh pointed out is an interesting one -- ie the idea that we would not merge windows support to trunk, but rather treat is as a "parallel code line" which lives in the ASF and has its own builds and releases. The windows team would periodically merge trunk->win to pick up any new changes, and do a separate test/release process. I'm not convinced this is the best idea, but worth discussion of pros and cons.

-Todd


>
> On Wed, Feb 27, 2013 at 11:56 AM, Eli Collins <el...@cloudera.com> wrote:
>
> > Bobby raises some good questions.  A related one, since most current 
> > developers won't add Windows support for new features that are 
> > platform specific is it assumed that Windows development will either 
> > lag or will people actively work on keeping Windows up with the 
> > latest?  And vice versa in case Windows support is implemented first.
> >
> > Is there a jira for resolving the outstanding TODOs in the code base 
> > (similar to HDFS-2148)?  Looks like this merge doesn't introduce 
> > many which is great (just did a quick diff and grep).
> >
> > Thanks,
> > Eli
> >
> > On Wed, Feb 27, 2013 at 8:17 AM, Robert Evans <ev...@yahoo-inc.com>
> wrote:
> > > After this is merged in is Windows still going to be a second 
> > > class citizen but happens to work for more than just development 
> > > or is it a fully supported platform where if something breaks it 
> > > can block a
> > release?
> > >  How do we as a community intend to keep Windows support from breaking?
> > > We don't have any Jenkins slaves to be able to run nightly tests 
> > > to validate everything still compiles/runs.  This is not a blocker 
> > > for me because we often rely on individuals and groups to test 
> > > Hadoop, but I
> do
> > > think we need to have this discussion before we put it in.
> > >
> > > --Bobby
> > >
> > > On 2/26/13 4:55 PM, "Suresh Srinivas" <su...@hortonworks.com> wrote:
> > >
> > >>I had posted heads up about merging branch-trunk-win to trunk on 
> > >>Feb
> 8th.
> > >>I
> > >>am happy to announce that we are ready for the merge.
> > >>
> > >>Here is a brief recap on the highlights of the work done:
> > >>- Command-line scripts for the Hadoop surface area
> > >>- Mapping the HDFS permissions model to Windows
> > >>- Abstracted and reconciled mismatches around differences in Path 
> > >>semantics in Java and Windows
> > >>- Native Task Controller for Windows
> > >>- Implementation of a Block Placement Policy to support cloud 
> > >>environments, more specifically Azure.
> > >>- Implementation of Hadoop native libraries for Windows 
> > >>(compression codecs, native I/O)
> > >>- Several reliability issues, including race-conditions, 
> > >>intermittent
> > test
> > >>failures, resource leaks.
> > >>- Several new unit test cases written for the above changes
> > >>
> > >>Please find the details of the work in 
> > >>CHANGES.branch-trunk-win.txt - Common 
> > >>changes<http://bit.ly/Xe7Ynv>, HDFS changes<
> http://bit.ly/13QOSo9
> > >,
> > >>and YARN and MapReduce changes <http://bit.ly/128zzMt>. This is 
> > >>the
> work
> > >>ported from branch-1-win to a branch based on trunk.
> > >>
> > >>For details of the testing done, please see the thread - 
> > >>http://bit.ly/WpavJ4. Merge patch for this is available on
> HADOOP-8562<
> > >>https://issues.apache.org/jira/browse/HADOOP-8562>.
> > >>
> > >>This was a large undertaking that involved developing code, 
> > >>testing the entire Hadoop stack, including scale tests. This is 
> > >>made possible only with the contribution from many many folks in 
> > >>the community. Following
> people
> > >>contributed to this work: Ivan Mitic, Chuan Liu, Ramya Sunil, 
> > >>Bikas
> Saha,
> > >>Kanna Karanam, John Gordon, Brandon Li, Chris Nauroth, David Lao,
> > Sumadhur
> > >>Reddy Bolli, Arpit Agarwal, Ahmed El Baz, Mike Liddell, Jing Zhao,
> Thejas
> > >>Nair, Steve Maine, Ganeshan Iyer, Raja Aluri, Giridharan Kesavan, 
> > >>Ramya Bharathi Nimmagadda, Daryn Sharp, Arun Murthy, Tsz-Wo 
> > >>Nicholas Sze,
> > Suresh
> > >>Srinivas and Sanjay Radia. There are many others who contributed 
> > >>as
> well
> > >>providing feedback and comments on numerous jiras.
> > >>
> > >>The vote will run for seven days and will end on March 5, 6:00PM PST.
> > >>
> > >>Regards,
> > >>Suresh
> > >>
> > >>
> > >>
> > >>
> > >>On Thu, Feb 7, 2013 at 6:41 PM, Mahadevan Venkatraman
> > >><ma...@microsoft.com>wrote:
> > >>
> > >>> It is super exciting to look at the prospect of these changes 
> > >>>being merged  to trunk. Having Windows as one of the supported 
> > >>>Hadoop platforms is
> a
> > >>> fantastic opportunity both for the Hadoop project and Microsoft 
> > >>>customers.
> > >>>
> > >>> This work began around a year back when a few of us started with 
> > >>> a
> > basic
> > >>> port of Hadoop on Windows. Ever since, the Hadoop team in 
> > >>> Microsoft
> > have
> > >>> made significant progress in the following areas:
> > >>> (PS: Some of these items are already included in Suresh's email, 
> > >>> but including again for completeness)
> > >>>
> > >>> - Command-line scripts for the Hadoop surface area
> > >>> - Mapping the HDFS permissions model to Windows
> > >>> - Abstracted and reconciled mismatches around differences in 
> > >>> Path semantics in Java and Windows
> > >>> - Native Task Controller for Windows
> > >>> - Implementation of a Block Placement Policy to support cloud 
> > >>> environments, more specifically Azure.
> > >>> - Implementation of Hadoop native libraries for Windows 
> > >>> (compression codecs, native I/O) - Several reliability issues, 
> > >>> including race-conditions, intermittent test failures, resource leaks.
> > >>> - Several new unit test cases written for the above changes
> > >>>
> > >>> In the process, we have closely engaged with the Apache open 
> > >>> source community and have got great support and assistance from 
> > >>> the
> community
> > >>>in
> > >>> terms of contributing fixes, code review comments and commits.
> > >>>
> > >>> In addition, the Hadoop team at Microsoft has also made good 
> > >>> progress
> > in
> > >>> other projects including Hive, Pig, Sqoop, Oozie, HCat and HBase.
> Many
> > >>>of
> > >>> these changes have already been committed to the respective 
> > >>>trunks
> with
> > >>> help from various committers and contributors. It is great to 
> > >>> see the commitment of the community to support multiple 
> > >>> platforms, and we
> look
> > >>> forward to the day when a developer/customer is able to 
> > >>>successfully deploy  a complete solution stack based on Apache 
> > >>>Hadoop releases.
> > >>>
> > >>> Next Steps:
> > >>>
> > >>> All of the above changes are part of the Windows Azure HDInsight 
> > >>>and  HDInsight Server products from Microsoft. We have 
> > >>>successfully on-boarded  several internal customers and have been 
> > >>>running production workloads
> > on
> > >>> Windows Azure HDInsight. Our vision is to create a big data 
> > >>>platform based  on Hadoop, and we are committed to helping make 
> > >>>Hadoop a world-class  solution that anyone can use to solve their 
> > >>>biggest data challenges.
> > >>>
> > >>> As an immediate next step, we would like to have a discussion 
> > >>> around
> > how
> > >>> we can ensure that the quality of the mainline Hadoop branches 
> > >>>on Windows  is maintained. To this end, we would like to get to 
> > >>>the state where
> we
> > >>>have
> > >>> pre-checkin validation gates and nightly test runs enabled on
> Windows.
> > >>>If
> > >>> you have any suggestions around this, please do send an email.  
> > >>>We
> are
> > >>> committed to helping sustain the long-term quality of Hadoop on 
> > >>>both Linux  and Windows.
> > >>>
> > >>> We sincerely thank the community for their contribution and 
> > >>> support
> so
> > >>> far. And hope to continue having a close engagement in the future.
> > >>>
> > >>> -Microsoft HDInsight Team
> > >>>
> > >>>
> > >>> -----Original Message-----
> > >>> From: Suresh Srinivas [mailto:suresh@hortonworks.com]
> > >>> Sent: Thursday, February 7, 2013 5:42 PM
> > >>> To: common-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org; 
> > >>> hdfs-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
> > >>> Subject: Heads up - merge branch-trunk-win to trunk
> > >>>
> > >>> The support for Hadoop on Windows was proposed in HADOOP-8079< 
> > >>> https://issues.apache.org/jira/browse/HADOOP-8079> almost a year
> ago.
> > >>>The
> > >>> goal was to make Hadoop natively integrated, full-featured, and 
> > >>>performance  and scalability tuned on Windows Server or Windows 
> > >>>Azure.
> > >>> We are happy to announce that a lot of progress has been made in 
> > >>>this  regard.
> > >>>
> > >>> Initial work started in a feature branch, branch-1-win, based on 
> > >>>branch-1.
> > >>> The details related to the work done in the branch can be seen 
> > >>>in  CHANGES.txt<
> > >>>
> > >>>
> > http://svn.apache.org/viewvc/hadoop/common/branches/branch-1-win/CHA
> > NGES
> .
> > >>>branch-1-win.txt?view=markup
> > >>> >.
> > >>> This work has been ported to a branch, branch-trunk-win, based 
> > >>> on
> > trunk.
> > >>> Merge patch for this is available on 
> > >>> HADOOP-8562<https://issues.apache.org/jira/browse/HADOOP-8562>
> > >>> .
> > >>>
> > >>> Highlights of the work done so far:
> > >>> 1. Necessary changes in Hadoop to run natively on Windows. These
> > changes
> > >>> handle differences in platforms related to path names, 
> > >>>process/task  management etc.
> > >>> 2. Addition of winutils tools for managing file permissions and 
> > >>>ownership,  user group mapping, hardlinks, symbolic links, chmod, 
> > >>>disk
> utilization,
> > >>>and
> > >>> process/task management.
> > >>> 3. Added cmd scripts equivalent to existing shell scripts 
> > >>>hadoop-daemon.sh, start and stop scripts.
> > >>> 4. Addition of block placement policy implemnation to support 
> > >>>cloud  enviroment, more specifically Azure.
> > >>>
> > >>> We are very close to wrapping up the work in branch-trunk-win 
> > >>>and getting  ready for a merge. Currently the merge patch is 
> > >>>passing close to 100%
> > of
> > >>> unit tests on Linux. Soon I will call for a vote to merge this 
> > >>>branch into  trunk.
> > >>>
> > >>> Next steps:
> > >>> 1. Call for vote to merge branch-trunk-win to trunk, when the 
> > >>> work completes and precommit build is clean.
> > >>> 2. Start a discussion on adding Jenkins precommit builds on 
> > >>> windows
> and
> > >>> how to integrate that with the existing commit process.
> > >>>
> > >>> Let me know if you have any questions.
> > >>>
> > >>> Regards,
> > >>> Suresh
> > >>>
> > >>>
> > >>
> > >>
> > >>--
> > >>http://hortonworks.com/download/
> > >
> >
>
>
>
> --
> http://hortonworks.com/download/
>



--
Todd Lipcon
Software Engineer, Cloudera

RE: [Vote] Merge branch-trunk-win to trunk

[John Gordon <Jo...@microsoft.com>]

+1 (non-binding)

I want to share my vote of confidence in this community.  If motivated to do so, this community can keep this project cross-platform and continue to rapidly innovate without breaking a sweat.

The day we started working on this, I saw the foundations of greatness in the quality and volume of dev tests, the code itself, and the Apache values themselves.

1.) Hadoop's unit tests and their frameworks are very well thought out and the consideration and energy that went into their design is worthy of praise.  The MiniCluster abstractions utilize very few resources and put all the processes into one JVM for easy debugging.  It is very easy to select specific tests from the full suite to reproduce an issue reported in another environment - like the Jenkins build server or another contributor's environment.  
2.) This community has done an excellent job of incorporating well-placed log messages to make it easy to post mortem troubleshoot most failures.  The logs are very useful, and it is extremely rare that troubleshooting a failure requires debugging a live repro.
3.) Hadoop is written primarily in Java, a cross-platform language that provides its own platform in the form of the JVM to insulate most of the code from the specifics of the OS layer.
4.) CoPDoC - The right priorities, and well stated.


Thank you,

John

-----Original Message-----
From: Ivan Mitic [mailto:ivanmi@microsoft.com] 
Sent: Wednesday, February 27, 2013 6:32 PM
To: mapreduce-dev@hadoop.apache.org; common-dev@hadoop.apache.org
Cc: yarn-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org
Subject: RE: [Vote] Merge branch-trunk-win to trunk

+1 (non-binding)

I am really glad to see this happening! As people already mentioned, this has been a great engineering effort involving many people!


Folks raised some valid concerns below and I thought it would be good to share my 2 cents. In my opinion, we don't have to solve all these problems right now. As we move forward with two platforms, we can start addressing one problem at a time and incrementally improve. In the first iteration, maintaining Hadoop on Windows could be just everyone trying to do their best effort (make sure Jenkins build succeeds at least). We already have people who are building/running trunk on Windows daily, so they would jump in and fix problems as needed (we've been doing this in branch-trunk-win for a while now). Although I see that the problems could arise with platform specific features/optimizations, I don't think these are frequent, so in most cases everything will just work. Merging the two branches sooner rather than later does seems like the right thing to do if the ultimate goal is to have Hadoop on both platforms. Now that the port has completed, we will have people in Microsoft (and elsewhere) wanting to contribute features/improvements to the trunk branch. A separate branch would just make things more difficult and confusing for everyone :) Hope this makes sense.

-----Original Message-----
From: Todd Lipcon [mailto:todd@cloudera.com]
Sent: Wednesday, February 27, 2013 3:43 PM
To: common-dev@hadoop.apache.org
Cc: yarn-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
Subject: Re: [Vote] Merge branch-trunk-win to trunk

On Wed, Feb 27, 2013 at 2:54 PM, Suresh Srinivas <su...@hortonworks.com>wrote:

> With that we need to decide how our precommit process looks.
> My inclination is to wait for +1 from precommit builds on both the 
> platforms to ensure no issues are introduced.
> Thoughts?
>
> 2. Feature development impact
> Some questions have been raised about would new features need to be 
> supported on both the platforms. Yes. I do not see a reason why 
> features cannot work on both the platforms, with the exception of 
> platform specific optimizations. This what Java gives us.
>
>
I'm concerned about the above. Personally, I don't have access to any Windows boxes with development tools, and I know nothing about developing on Windows. The only Windows I run is an 8GB VM with 1 GB RAM allocated, for powerpoint :)

If I submit a patch and it gets -1 "tests failed" on the Windows slave, how am I supposed to proceed?

I think a reasonable compromise would be that the tests should always
*build* on Windows before commit, and contributors should do their best to look at the test logs for any Windows-specific failures. But, beyond looking at the logs, a "-1 Tests failed on windows" should not block a commit.

Those contributors who are interested in Windows being a first-class platform should be responsible for watching the Windows builds and debugging/fixing any regressions that might be Windows-specific.

I also think the KDE model that Harsh pointed out is an interesting one -- ie the idea that we would not merge windows support to trunk, but rather treat is as a "parallel code line" which lives in the ASF and has its own builds and releases. The windows team would periodically merge trunk->win to pick up any new changes, and do a separate test/release process. I'm not convinced this is the best idea, but worth discussion of pros and cons.

-Todd


>
> On Wed, Feb 27, 2013 at 11:56 AM, Eli Collins <el...@cloudera.com> wrote:
>
> > Bobby raises some good questions.  A related one, since most current 
> > developers won't add Windows support for new features that are 
> > platform specific is it assumed that Windows development will either 
> > lag or will people actively work on keeping Windows up with the 
> > latest?  And vice versa in case Windows support is implemented first.
> >
> > Is there a jira for resolving the outstanding TODOs in the code base 
> > (similar to HDFS-2148)?  Looks like this merge doesn't introduce 
> > many which is great (just did a quick diff and grep).
> >
> > Thanks,
> > Eli
> >
> > On Wed, Feb 27, 2013 at 8:17 AM, Robert Evans <ev...@yahoo-inc.com>
> wrote:
> > > After this is merged in is Windows still going to be a second 
> > > class citizen but happens to work for more than just development 
> > > or is it a fully supported platform where if something breaks it 
> > > can block a
> > release?
> > >  How do we as a community intend to keep Windows support from breaking?
> > > We don't have any Jenkins slaves to be able to run nightly tests 
> > > to validate everything still compiles/runs.  This is not a blocker 
> > > for me because we often rely on individuals and groups to test 
> > > Hadoop, but I
> do
> > > think we need to have this discussion before we put it in.
> > >
> > > --Bobby
> > >
> > > On 2/26/13 4:55 PM, "Suresh Srinivas" <su...@hortonworks.com> wrote:
> > >
> > >>I had posted heads up about merging branch-trunk-win to trunk on 
> > >>Feb
> 8th.
> > >>I
> > >>am happy to announce that we are ready for the merge.
> > >>
> > >>Here is a brief recap on the highlights of the work done:
> > >>- Command-line scripts for the Hadoop surface area
> > >>- Mapping the HDFS permissions model to Windows
> > >>- Abstracted and reconciled mismatches around differences in Path 
> > >>semantics in Java and Windows
> > >>- Native Task Controller for Windows
> > >>- Implementation of a Block Placement Policy to support cloud 
> > >>environments, more specifically Azure.
> > >>- Implementation of Hadoop native libraries for Windows 
> > >>(compression codecs, native I/O)
> > >>- Several reliability issues, including race-conditions, 
> > >>intermittent
> > test
> > >>failures, resource leaks.
> > >>- Several new unit test cases written for the above changes
> > >>
> > >>Please find the details of the work in 
> > >>CHANGES.branch-trunk-win.txt - Common 
> > >>changes<http://bit.ly/Xe7Ynv>, HDFS changes<
> http://bit.ly/13QOSo9
> > >,
> > >>and YARN and MapReduce changes <http://bit.ly/128zzMt>. This is 
> > >>the
> work
> > >>ported from branch-1-win to a branch based on trunk.
> > >>
> > >>For details of the testing done, please see the thread - 
> > >>http://bit.ly/WpavJ4. Merge patch for this is available on
> HADOOP-8562<
> > >>https://issues.apache.org/jira/browse/HADOOP-8562>.
> > >>
> > >>This was a large undertaking that involved developing code, 
> > >>testing the entire Hadoop stack, including scale tests. This is 
> > >>made possible only with the contribution from many many folks in 
> > >>the community. Following
> people
> > >>contributed to this work: Ivan Mitic, Chuan Liu, Ramya Sunil, 
> > >>Bikas
> Saha,
> > >>Kanna Karanam, John Gordon, Brandon Li, Chris Nauroth, David Lao,
> > Sumadhur
> > >>Reddy Bolli, Arpit Agarwal, Ahmed El Baz, Mike Liddell, Jing Zhao,
> Thejas
> > >>Nair, Steve Maine, Ganeshan Iyer, Raja Aluri, Giridharan Kesavan, 
> > >>Ramya Bharathi Nimmagadda, Daryn Sharp, Arun Murthy, Tsz-Wo 
> > >>Nicholas Sze,
> > Suresh
> > >>Srinivas and Sanjay Radia. There are many others who contributed 
> > >>as
> well
> > >>providing feedback and comments on numerous jiras.
> > >>
> > >>The vote will run for seven days and will end on March 5, 6:00PM PST.
> > >>
> > >>Regards,
> > >>Suresh
> > >>
> > >>
> > >>
> > >>
> > >>On Thu, Feb 7, 2013 at 6:41 PM, Mahadevan Venkatraman
> > >><ma...@microsoft.com>wrote:
> > >>
> > >>> It is super exciting to look at the prospect of these changes 
> > >>>being merged  to trunk. Having Windows as one of the supported 
> > >>>Hadoop platforms is
> a
> > >>> fantastic opportunity both for the Hadoop project and Microsoft 
> > >>>customers.
> > >>>
> > >>> This work began around a year back when a few of us started with 
> > >>> a
> > basic
> > >>> port of Hadoop on Windows. Ever since, the Hadoop team in 
> > >>> Microsoft
> > have
> > >>> made significant progress in the following areas:
> > >>> (PS: Some of these items are already included in Suresh's email, 
> > >>> but including again for completeness)
> > >>>
> > >>> - Command-line scripts for the Hadoop surface area
> > >>> - Mapping the HDFS permissions model to Windows
> > >>> - Abstracted and reconciled mismatches around differences in 
> > >>> Path semantics in Java and Windows
> > >>> - Native Task Controller for Windows
> > >>> - Implementation of a Block Placement Policy to support cloud 
> > >>> environments, more specifically Azure.
> > >>> - Implementation of Hadoop native libraries for Windows 
> > >>> (compression codecs, native I/O) - Several reliability issues, 
> > >>> including race-conditions, intermittent test failures, resource leaks.
> > >>> - Several new unit test cases written for the above changes
> > >>>
> > >>> In the process, we have closely engaged with the Apache open 
> > >>> source community and have got great support and assistance from 
> > >>> the
> community
> > >>>in
> > >>> terms of contributing fixes, code review comments and commits.
> > >>>
> > >>> In addition, the Hadoop team at Microsoft has also made good 
> > >>> progress
> > in
> > >>> other projects including Hive, Pig, Sqoop, Oozie, HCat and HBase.
> Many
> > >>>of
> > >>> these changes have already been committed to the respective 
> > >>>trunks
> with
> > >>> help from various committers and contributors. It is great to 
> > >>> see the commitment of the community to support multiple 
> > >>> platforms, and we
> look
> > >>> forward to the day when a developer/customer is able to 
> > >>>successfully deploy  a complete solution stack based on Apache 
> > >>>Hadoop releases.
> > >>>
> > >>> Next Steps:
> > >>>
> > >>> All of the above changes are part of the Windows Azure HDInsight 
> > >>>and  HDInsight Server products from Microsoft. We have 
> > >>>successfully on-boarded  several internal customers and have been 
> > >>>running production workloads
> > on
> > >>> Windows Azure HDInsight. Our vision is to create a big data 
> > >>>platform based  on Hadoop, and we are committed to helping make 
> > >>>Hadoop a world-class  solution that anyone can use to solve their 
> > >>>biggest data challenges.
> > >>>
> > >>> As an immediate next step, we would like to have a discussion 
> > >>> around
> > how
> > >>> we can ensure that the quality of the mainline Hadoop branches 
> > >>>on Windows  is maintained. To this end, we would like to get to 
> > >>>the state where
> we
> > >>>have
> > >>> pre-checkin validation gates and nightly test runs enabled on
> Windows.
> > >>>If
> > >>> you have any suggestions around this, please do send an email.  
> > >>>We
> are
> > >>> committed to helping sustain the long-term quality of Hadoop on 
> > >>>both Linux  and Windows.
> > >>>
> > >>> We sincerely thank the community for their contribution and 
> > >>> support
> so
> > >>> far. And hope to continue having a close engagement in the future.
> > >>>
> > >>> -Microsoft HDInsight Team
> > >>>
> > >>>
> > >>> -----Original Message-----
> > >>> From: Suresh Srinivas [mailto:suresh@hortonworks.com]
> > >>> Sent: Thursday, February 7, 2013 5:42 PM
> > >>> To: common-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org; 
> > >>> hdfs-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
> > >>> Subject: Heads up - merge branch-trunk-win to trunk
> > >>>
> > >>> The support for Hadoop on Windows was proposed in HADOOP-8079< 
> > >>> https://issues.apache.org/jira/browse/HADOOP-8079> almost a year
> ago.
> > >>>The
> > >>> goal was to make Hadoop natively integrated, full-featured, and 
> > >>>performance  and scalability tuned on Windows Server or Windows 
> > >>>Azure.
> > >>> We are happy to announce that a lot of progress has been made in 
> > >>>this  regard.
> > >>>
> > >>> Initial work started in a feature branch, branch-1-win, based on 
> > >>>branch-1.
> > >>> The details related to the work done in the branch can be seen 
> > >>>in  CHANGES.txt<
> > >>>
> > >>>
> > http://svn.apache.org/viewvc/hadoop/common/branches/branch-1-win/CHA
> > NGES
> .
> > >>>branch-1-win.txt?view=markup
> > >>> >.
> > >>> This work has been ported to a branch, branch-trunk-win, based 
> > >>> on
> > trunk.
> > >>> Merge patch for this is available on 
> > >>> HADOOP-8562<https://issues.apache.org/jira/browse/HADOOP-8562>
> > >>> .
> > >>>
> > >>> Highlights of the work done so far:
> > >>> 1. Necessary changes in Hadoop to run natively on Windows. These
> > changes
> > >>> handle differences in platforms related to path names, 
> > >>>process/task  management etc.
> > >>> 2. Addition of winutils tools for managing file permissions and 
> > >>>ownership,  user group mapping, hardlinks, symbolic links, chmod, 
> > >>>disk
> utilization,
> > >>>and
> > >>> process/task management.
> > >>> 3. Added cmd scripts equivalent to existing shell scripts 
> > >>>hadoop-daemon.sh, start and stop scripts.
> > >>> 4. Addition of block placement policy implemnation to support 
> > >>>cloud  enviroment, more specifically Azure.
> > >>>
> > >>> We are very close to wrapping up the work in branch-trunk-win 
> > >>>and getting  ready for a merge. Currently the merge patch is 
> > >>>passing close to 100%
> > of
> > >>> unit tests on Linux. Soon I will call for a vote to merge this 
> > >>>branch into  trunk.
> > >>>
> > >>> Next steps:
> > >>> 1. Call for vote to merge branch-trunk-win to trunk, when the 
> > >>> work completes and precommit build is clean.
> > >>> 2. Start a discussion on adding Jenkins precommit builds on 
> > >>> windows
> and
> > >>> how to integrate that with the existing commit process.
> > >>>
> > >>> Let me know if you have any questions.
> > >>>
> > >>> Regards,
> > >>> Suresh
> > >>>
> > >>>
> > >>
> > >>
> > >>--
> > >>http://hortonworks.com/download/
> > >
> >
>
>
>
> --
> http://hortonworks.com/download/
>



--
Todd Lipcon
Software Engineer, Cloudera

RE: [Vote] Merge branch-trunk-win to trunk

[John Gordon <Jo...@microsoft.com>]

+1 (non-binding)

I want to share my vote of confidence in this community.  If motivated to do so, this community can keep this project cross-platform and continue to rapidly innovate without breaking a sweat.

The day we started working on this, I saw the foundations of greatness in the quality and volume of dev tests, the code itself, and the Apache values themselves.

1.) Hadoop's unit tests and their frameworks are very well thought out and the consideration and energy that went into their design is worthy of praise.  The MiniCluster abstractions utilize very few resources and put all the processes into one JVM for easy debugging.  It is very easy to select specific tests from the full suite to reproduce an issue reported in another environment - like the Jenkins build server or another contributor's environment.  
2.) This community has done an excellent job of incorporating well-placed log messages to make it easy to post mortem troubleshoot most failures.  The logs are very useful, and it is extremely rare that troubleshooting a failure requires debugging a live repro.
3.) Hadoop is written primarily in Java, a cross-platform language that provides its own platform in the form of the JVM to insulate most of the code from the specifics of the OS layer.
4.) CoPDoC - The right priorities, and well stated.


Thank you,

John

-----Original Message-----
From: Ivan Mitic [mailto:ivanmi@microsoft.com] 
Sent: Wednesday, February 27, 2013 6:32 PM
To: mapreduce-dev@hadoop.apache.org; common-dev@hadoop.apache.org
Cc: yarn-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org
Subject: RE: [Vote] Merge branch-trunk-win to trunk

+1 (non-binding)

I am really glad to see this happening! As people already mentioned, this has been a great engineering effort involving many people!


Folks raised some valid concerns below and I thought it would be good to share my 2 cents. In my opinion, we don't have to solve all these problems right now. As we move forward with two platforms, we can start addressing one problem at a time and incrementally improve. In the first iteration, maintaining Hadoop on Windows could be just everyone trying to do their best effort (make sure Jenkins build succeeds at least). We already have people who are building/running trunk on Windows daily, so they would jump in and fix problems as needed (we've been doing this in branch-trunk-win for a while now). Although I see that the problems could arise with platform specific features/optimizations, I don't think these are frequent, so in most cases everything will just work. Merging the two branches sooner rather than later does seems like the right thing to do if the ultimate goal is to have Hadoop on both platforms. Now that the port has completed, we will have people in Microsoft (and elsewhere) wanting to contribute features/improvements to the trunk branch. A separate branch would just make things more difficult and confusing for everyone :) Hope this makes sense.

-----Original Message-----
From: Todd Lipcon [mailto:todd@cloudera.com]
Sent: Wednesday, February 27, 2013 3:43 PM
To: common-dev@hadoop.apache.org
Cc: yarn-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
Subject: Re: [Vote] Merge branch-trunk-win to trunk

On Wed, Feb 27, 2013 at 2:54 PM, Suresh Srinivas <su...@hortonworks.com>wrote:

> With that we need to decide how our precommit process looks.
> My inclination is to wait for +1 from precommit builds on both the 
> platforms to ensure no issues are introduced.
> Thoughts?
>
> 2. Feature development impact
> Some questions have been raised about would new features need to be 
> supported on both the platforms. Yes. I do not see a reason why 
> features cannot work on both the platforms, with the exception of 
> platform specific optimizations. This what Java gives us.
>
>
I'm concerned about the above. Personally, I don't have access to any Windows boxes with development tools, and I know nothing about developing on Windows. The only Windows I run is an 8GB VM with 1 GB RAM allocated, for powerpoint :)

If I submit a patch and it gets -1 "tests failed" on the Windows slave, how am I supposed to proceed?

I think a reasonable compromise would be that the tests should always
*build* on Windows before commit, and contributors should do their best to look at the test logs for any Windows-specific failures. But, beyond looking at the logs, a "-1 Tests failed on windows" should not block a commit.

Those contributors who are interested in Windows being a first-class platform should be responsible for watching the Windows builds and debugging/fixing any regressions that might be Windows-specific.

I also think the KDE model that Harsh pointed out is an interesting one -- ie the idea that we would not merge windows support to trunk, but rather treat is as a "parallel code line" which lives in the ASF and has its own builds and releases. The windows team would periodically merge trunk->win to pick up any new changes, and do a separate test/release process. I'm not convinced this is the best idea, but worth discussion of pros and cons.

-Todd


>
> On Wed, Feb 27, 2013 at 11:56 AM, Eli Collins <el...@cloudera.com> wrote:
>
> > Bobby raises some good questions.  A related one, since most current 
> > developers won't add Windows support for new features that are 
> > platform specific is it assumed that Windows development will either 
> > lag or will people actively work on keeping Windows up with the 
> > latest?  And vice versa in case Windows support is implemented first.
> >
> > Is there a jira for resolving the outstanding TODOs in the code base 
> > (similar to HDFS-2148)?  Looks like this merge doesn't introduce 
> > many which is great (just did a quick diff and grep).
> >
> > Thanks,
> > Eli
> >
> > On Wed, Feb 27, 2013 at 8:17 AM, Robert Evans <ev...@yahoo-inc.com>
> wrote:
> > > After this is merged in is Windows still going to be a second 
> > > class citizen but happens to work for more than just development 
> > > or is it a fully supported platform where if something breaks it 
> > > can block a
> > release?
> > >  How do we as a community intend to keep Windows support from breaking?
> > > We don't have any Jenkins slaves to be able to run nightly tests 
> > > to validate everything still compiles/runs.  This is not a blocker 
> > > for me because we often rely on individuals and groups to test 
> > > Hadoop, but I
> do
> > > think we need to have this discussion before we put it in.
> > >
> > > --Bobby
> > >
> > > On 2/26/13 4:55 PM, "Suresh Srinivas" <su...@hortonworks.com> wrote:
> > >
> > >>I had posted heads up about merging branch-trunk-win to trunk on 
> > >>Feb
> 8th.
> > >>I
> > >>am happy to announce that we are ready for the merge.
> > >>
> > >>Here is a brief recap on the highlights of the work done:
> > >>- Command-line scripts for the Hadoop surface area
> > >>- Mapping the HDFS permissions model to Windows
> > >>- Abstracted and reconciled mismatches around differences in Path 
> > >>semantics in Java and Windows
> > >>- Native Task Controller for Windows
> > >>- Implementation of a Block Placement Policy to support cloud 
> > >>environments, more specifically Azure.
> > >>- Implementation of Hadoop native libraries for Windows 
> > >>(compression codecs, native I/O)
> > >>- Several reliability issues, including race-conditions, 
> > >>intermittent
> > test
> > >>failures, resource leaks.
> > >>- Several new unit test cases written for the above changes
> > >>
> > >>Please find the details of the work in 
> > >>CHANGES.branch-trunk-win.txt - Common 
> > >>changes<http://bit.ly/Xe7Ynv>, HDFS changes<
> http://bit.ly/13QOSo9
> > >,
> > >>and YARN and MapReduce changes <http://bit.ly/128zzMt>. This is 
> > >>the
> work
> > >>ported from branch-1-win to a branch based on trunk.
> > >>
> > >>For details of the testing done, please see the thread - 
> > >>http://bit.ly/WpavJ4. Merge patch for this is available on
> HADOOP-8562<
> > >>https://issues.apache.org/jira/browse/HADOOP-8562>.
> > >>
> > >>This was a large undertaking that involved developing code, 
> > >>testing the entire Hadoop stack, including scale tests. This is 
> > >>made possible only with the contribution from many many folks in 
> > >>the community. Following
> people
> > >>contributed to this work: Ivan Mitic, Chuan Liu, Ramya Sunil, 
> > >>Bikas
> Saha,
> > >>Kanna Karanam, John Gordon, Brandon Li, Chris Nauroth, David Lao,
> > Sumadhur
> > >>Reddy Bolli, Arpit Agarwal, Ahmed El Baz, Mike Liddell, Jing Zhao,
> Thejas
> > >>Nair, Steve Maine, Ganeshan Iyer, Raja Aluri, Giridharan Kesavan, 
> > >>Ramya Bharathi Nimmagadda, Daryn Sharp, Arun Murthy, Tsz-Wo 
> > >>Nicholas Sze,
> > Suresh
> > >>Srinivas and Sanjay Radia. There are many others who contributed 
> > >>as
> well
> > >>providing feedback and comments on numerous jiras.
> > >>
> > >>The vote will run for seven days and will end on March 5, 6:00PM PST.
> > >>
> > >>Regards,
> > >>Suresh
> > >>
> > >>
> > >>
> > >>
> > >>On Thu, Feb 7, 2013 at 6:41 PM, Mahadevan Venkatraman
> > >><ma...@microsoft.com>wrote:
> > >>
> > >>> It is super exciting to look at the prospect of these changes 
> > >>>being merged  to trunk. Having Windows as one of the supported 
> > >>>Hadoop platforms is
> a
> > >>> fantastic opportunity both for the Hadoop project and Microsoft 
> > >>>customers.
> > >>>
> > >>> This work began around a year back when a few of us started with 
> > >>> a
> > basic
> > >>> port of Hadoop on Windows. Ever since, the Hadoop team in 
> > >>> Microsoft
> > have
> > >>> made significant progress in the following areas:
> > >>> (PS: Some of these items are already included in Suresh's email, 
> > >>> but including again for completeness)
> > >>>
> > >>> - Command-line scripts for the Hadoop surface area
> > >>> - Mapping the HDFS permissions model to Windows
> > >>> - Abstracted and reconciled mismatches around differences in 
> > >>> Path semantics in Java and Windows
> > >>> - Native Task Controller for Windows
> > >>> - Implementation of a Block Placement Policy to support cloud 
> > >>> environments, more specifically Azure.
> > >>> - Implementation of Hadoop native libraries for Windows 
> > >>> (compression codecs, native I/O) - Several reliability issues, 
> > >>> including race-conditions, intermittent test failures, resource leaks.
> > >>> - Several new unit test cases written for the above changes
> > >>>
> > >>> In the process, we have closely engaged with the Apache open 
> > >>> source community and have got great support and assistance from 
> > >>> the
> community
> > >>>in
> > >>> terms of contributing fixes, code review comments and commits.
> > >>>
> > >>> In addition, the Hadoop team at Microsoft has also made good 
> > >>> progress
> > in
> > >>> other projects including Hive, Pig, Sqoop, Oozie, HCat and HBase.
> Many
> > >>>of
> > >>> these changes have already been committed to the respective 
> > >>>trunks
> with
> > >>> help from various committers and contributors. It is great to 
> > >>> see the commitment of the community to support multiple 
> > >>> platforms, and we
> look
> > >>> forward to the day when a developer/customer is able to 
> > >>>successfully deploy  a complete solution stack based on Apache 
> > >>>Hadoop releases.
> > >>>
> > >>> Next Steps:
> > >>>
> > >>> All of the above changes are part of the Windows Azure HDInsight 
> > >>>and  HDInsight Server products from Microsoft. We have 
> > >>>successfully on-boarded  several internal customers and have been 
> > >>>running production workloads
> > on
> > >>> Windows Azure HDInsight. Our vision is to create a big data 
> > >>>platform based  on Hadoop, and we are committed to helping make 
> > >>>Hadoop a world-class  solution that anyone can use to solve their 
> > >>>biggest data challenges.
> > >>>
> > >>> As an immediate next step, we would like to have a discussion 
> > >>> around
> > how
> > >>> we can ensure that the quality of the mainline Hadoop branches 
> > >>>on Windows  is maintained. To this end, we would like to get to 
> > >>>the state where
> we
> > >>>have
> > >>> pre-checkin validation gates and nightly test runs enabled on
> Windows.
> > >>>If
> > >>> you have any suggestions around this, please do send an email.  
> > >>>We
> are
> > >>> committed to helping sustain the long-term quality of Hadoop on 
> > >>>both Linux  and Windows.
> > >>>
> > >>> We sincerely thank the community for their contribution and 
> > >>> support
> so
> > >>> far. And hope to continue having a close engagement in the future.
> > >>>
> > >>> -Microsoft HDInsight Team
> > >>>
> > >>>
> > >>> -----Original Message-----
> > >>> From: Suresh Srinivas [mailto:suresh@hortonworks.com]
> > >>> Sent: Thursday, February 7, 2013 5:42 PM
> > >>> To: common-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org; 
> > >>> hdfs-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
> > >>> Subject: Heads up - merge branch-trunk-win to trunk
> > >>>
> > >>> The support for Hadoop on Windows was proposed in HADOOP-8079< 
> > >>> https://issues.apache.org/jira/browse/HADOOP-8079> almost a year
> ago.
> > >>>The
> > >>> goal was to make Hadoop natively integrated, full-featured, and 
> > >>>performance  and scalability tuned on Windows Server or Windows 
> > >>>Azure.
> > >>> We are happy to announce that a lot of progress has been made in 
> > >>>this  regard.
> > >>>
> > >>> Initial work started in a feature branch, branch-1-win, based on 
> > >>>branch-1.
> > >>> The details related to the work done in the branch can be seen 
> > >>>in  CHANGES.txt<
> > >>>
> > >>>
> > http://svn.apache.org/viewvc/hadoop/common/branches/branch-1-win/CHA
> > NGES
> .
> > >>>branch-1-win.txt?view=markup
> > >>> >.
> > >>> This work has been ported to a branch, branch-trunk-win, based 
> > >>> on
> > trunk.
> > >>> Merge patch for this is available on 
> > >>> HADOOP-8562<https://issues.apache.org/jira/browse/HADOOP-8562>
> > >>> .
> > >>>
> > >>> Highlights of the work done so far:
> > >>> 1. Necessary changes in Hadoop to run natively on Windows. These
> > changes
> > >>> handle differences in platforms related to path names, 
> > >>>process/task  management etc.
> > >>> 2. Addition of winutils tools for managing file permissions and 
> > >>>ownership,  user group mapping, hardlinks, symbolic links, chmod, 
> > >>>disk
> utilization,
> > >>>and
> > >>> process/task management.
> > >>> 3. Added cmd scripts equivalent to existing shell scripts 
> > >>>hadoop-daemon.sh, start and stop scripts.
> > >>> 4. Addition of block placement policy implemnation to support 
> > >>>cloud  enviroment, more specifically Azure.
> > >>>
> > >>> We are very close to wrapping up the work in branch-trunk-win 
> > >>>and getting  ready for a merge. Currently the merge patch is 
> > >>>passing close to 100%
> > of
> > >>> unit tests on Linux. Soon I will call for a vote to merge this 
> > >>>branch into  trunk.
> > >>>
> > >>> Next steps:
> > >>> 1. Call for vote to merge branch-trunk-win to trunk, when the 
> > >>> work completes and precommit build is clean.
> > >>> 2. Start a discussion on adding Jenkins precommit builds on 
> > >>> windows
> and
> > >>> how to integrate that with the existing commit process.
> > >>>
> > >>> Let me know if you have any questions.
> > >>>
> > >>> Regards,
> > >>> Suresh
> > >>>
> > >>>
> > >>
> > >>
> > >>--
> > >>http://hortonworks.com/download/
> > >
> >
>
>
>
> --
> http://hortonworks.com/download/
>



--
Todd Lipcon
Software Engineer, Cloudera

RE: [Vote] Merge branch-trunk-win to trunk

[John Gordon <Jo...@microsoft.com>]

+1 (non-binding)

I want to share my vote of confidence in this community.  If motivated to do so, this community can keep this project cross-platform and continue to rapidly innovate without breaking a sweat.

The day we started working on this, I saw the foundations of greatness in the quality and volume of dev tests, the code itself, and the Apache values themselves.

1.) Hadoop's unit tests and their frameworks are very well thought out and the consideration and energy that went into their design is worthy of praise.  The MiniCluster abstractions utilize very few resources and put all the processes into one JVM for easy debugging.  It is very easy to select specific tests from the full suite to reproduce an issue reported in another environment - like the Jenkins build server or another contributor's environment.  
2.) This community has done an excellent job of incorporating well-placed log messages to make it easy to post mortem troubleshoot most failures.  The logs are very useful, and it is extremely rare that troubleshooting a failure requires debugging a live repro.
3.) Hadoop is written primarily in Java, a cross-platform language that provides its own platform in the form of the JVM to insulate most of the code from the specifics of the OS layer.
4.) CoPDoC - The right priorities, and well stated.


Thank you,

John

-----Original Message-----
From: Ivan Mitic [mailto:ivanmi@microsoft.com] 
Sent: Wednesday, February 27, 2013 6:32 PM
To: mapreduce-dev@hadoop.apache.org; common-dev@hadoop.apache.org
Cc: yarn-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org
Subject: RE: [Vote] Merge branch-trunk-win to trunk

+1 (non-binding)

I am really glad to see this happening! As people already mentioned, this has been a great engineering effort involving many people!


Folks raised some valid concerns below and I thought it would be good to share my 2 cents. In my opinion, we don't have to solve all these problems right now. As we move forward with two platforms, we can start addressing one problem at a time and incrementally improve. In the first iteration, maintaining Hadoop on Windows could be just everyone trying to do their best effort (make sure Jenkins build succeeds at least). We already have people who are building/running trunk on Windows daily, so they would jump in and fix problems as needed (we've been doing this in branch-trunk-win for a while now). Although I see that the problems could arise with platform specific features/optimizations, I don't think these are frequent, so in most cases everything will just work. Merging the two branches sooner rather than later does seems like the right thing to do if the ultimate goal is to have Hadoop on both platforms. Now that the port has completed, we will have people in Microsoft (and elsewhere) wanting to contribute features/improvements to the trunk branch. A separate branch would just make things more difficult and confusing for everyone :) Hope this makes sense.

-----Original Message-----
From: Todd Lipcon [mailto:todd@cloudera.com]
Sent: Wednesday, February 27, 2013 3:43 PM
To: common-dev@hadoop.apache.org
Cc: yarn-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
Subject: Re: [Vote] Merge branch-trunk-win to trunk

On Wed, Feb 27, 2013 at 2:54 PM, Suresh Srinivas <su...@hortonworks.com>wrote:

> With that we need to decide how our precommit process looks.
> My inclination is to wait for +1 from precommit builds on both the 
> platforms to ensure no issues are introduced.
> Thoughts?
>
> 2. Feature development impact
> Some questions have been raised about would new features need to be 
> supported on both the platforms. Yes. I do not see a reason why 
> features cannot work on both the platforms, with the exception of 
> platform specific optimizations. This what Java gives us.
>
>
I'm concerned about the above. Personally, I don't have access to any Windows boxes with development tools, and I know nothing about developing on Windows. The only Windows I run is an 8GB VM with 1 GB RAM allocated, for powerpoint :)

If I submit a patch and it gets -1 "tests failed" on the Windows slave, how am I supposed to proceed?

I think a reasonable compromise would be that the tests should always
*build* on Windows before commit, and contributors should do their best to look at the test logs for any Windows-specific failures. But, beyond looking at the logs, a "-1 Tests failed on windows" should not block a commit.

Those contributors who are interested in Windows being a first-class platform should be responsible for watching the Windows builds and debugging/fixing any regressions that might be Windows-specific.

I also think the KDE model that Harsh pointed out is an interesting one -- ie the idea that we would not merge windows support to trunk, but rather treat is as a "parallel code line" which lives in the ASF and has its own builds and releases. The windows team would periodically merge trunk->win to pick up any new changes, and do a separate test/release process. I'm not convinced this is the best idea, but worth discussion of pros and cons.

-Todd


>
> On Wed, Feb 27, 2013 at 11:56 AM, Eli Collins <el...@cloudera.com> wrote:
>
> > Bobby raises some good questions.  A related one, since most current 
> > developers won't add Windows support for new features that are 
> > platform specific is it assumed that Windows development will either 
> > lag or will people actively work on keeping Windows up with the 
> > latest?  And vice versa in case Windows support is implemented first.
> >
> > Is there a jira for resolving the outstanding TODOs in the code base 
> > (similar to HDFS-2148)?  Looks like this merge doesn't introduce 
> > many which is great (just did a quick diff and grep).
> >
> > Thanks,
> > Eli
> >
> > On Wed, Feb 27, 2013 at 8:17 AM, Robert Evans <ev...@yahoo-inc.com>
> wrote:
> > > After this is merged in is Windows still going to be a second 
> > > class citizen but happens to work for more than just development 
> > > or is it a fully supported platform where if something breaks it 
> > > can block a
> > release?
> > >  How do we as a community intend to keep Windows support from breaking?
> > > We don't have any Jenkins slaves to be able to run nightly tests 
> > > to validate everything still compiles/runs.  This is not a blocker 
> > > for me because we often rely on individuals and groups to test 
> > > Hadoop, but I
> do
> > > think we need to have this discussion before we put it in.
> > >
> > > --Bobby
> > >
> > > On 2/26/13 4:55 PM, "Suresh Srinivas" <su...@hortonworks.com> wrote:
> > >
> > >>I had posted heads up about merging branch-trunk-win to trunk on 
> > >>Feb
> 8th.
> > >>I
> > >>am happy to announce that we are ready for the merge.
> > >>
> > >>Here is a brief recap on the highlights of the work done:
> > >>- Command-line scripts for the Hadoop surface area
> > >>- Mapping the HDFS permissions model to Windows
> > >>- Abstracted and reconciled mismatches around differences in Path 
> > >>semantics in Java and Windows
> > >>- Native Task Controller for Windows
> > >>- Implementation of a Block Placement Policy to support cloud 
> > >>environments, more specifically Azure.
> > >>- Implementation of Hadoop native libraries for Windows 
> > >>(compression codecs, native I/O)
> > >>- Several reliability issues, including race-conditions, 
> > >>intermittent
> > test
> > >>failures, resource leaks.
> > >>- Several new unit test cases written for the above changes
> > >>
> > >>Please find the details of the work in 
> > >>CHANGES.branch-trunk-win.txt - Common 
> > >>changes<http://bit.ly/Xe7Ynv>, HDFS changes<
> http://bit.ly/13QOSo9
> > >,
> > >>and YARN and MapReduce changes <http://bit.ly/128zzMt>. This is 
> > >>the
> work
> > >>ported from branch-1-win to a branch based on trunk.
> > >>
> > >>For details of the testing done, please see the thread - 
> > >>http://bit.ly/WpavJ4. Merge patch for this is available on
> HADOOP-8562<
> > >>https://issues.apache.org/jira/browse/HADOOP-8562>.
> > >>
> > >>This was a large undertaking that involved developing code, 
> > >>testing the entire Hadoop stack, including scale tests. This is 
> > >>made possible only with the contribution from many many folks in 
> > >>the community. Following
> people
> > >>contributed to this work: Ivan Mitic, Chuan Liu, Ramya Sunil, 
> > >>Bikas
> Saha,
> > >>Kanna Karanam, John Gordon, Brandon Li, Chris Nauroth, David Lao,
> > Sumadhur
> > >>Reddy Bolli, Arpit Agarwal, Ahmed El Baz, Mike Liddell, Jing Zhao,
> Thejas
> > >>Nair, Steve Maine, Ganeshan Iyer, Raja Aluri, Giridharan Kesavan, 
> > >>Ramya Bharathi Nimmagadda, Daryn Sharp, Arun Murthy, Tsz-Wo 
> > >>Nicholas Sze,
> > Suresh
> > >>Srinivas and Sanjay Radia. There are many others who contributed 
> > >>as
> well
> > >>providing feedback and comments on numerous jiras.
> > >>
> > >>The vote will run for seven days and will end on March 5, 6:00PM PST.
> > >>
> > >>Regards,
> > >>Suresh
> > >>
> > >>
> > >>
> > >>
> > >>On Thu, Feb 7, 2013 at 6:41 PM, Mahadevan Venkatraman
> > >><ma...@microsoft.com>wrote:
> > >>
> > >>> It is super exciting to look at the prospect of these changes 
> > >>>being merged  to trunk. Having Windows as one of the supported 
> > >>>Hadoop platforms is
> a
> > >>> fantastic opportunity both for the Hadoop project and Microsoft 
> > >>>customers.
> > >>>
> > >>> This work began around a year back when a few of us started with 
> > >>> a
> > basic
> > >>> port of Hadoop on Windows. Ever since, the Hadoop team in 
> > >>> Microsoft
> > have
> > >>> made significant progress in the following areas:
> > >>> (PS: Some of these items are already included in Suresh's email, 
> > >>> but including again for completeness)
> > >>>
> > >>> - Command-line scripts for the Hadoop surface area
> > >>> - Mapping the HDFS permissions model to Windows
> > >>> - Abstracted and reconciled mismatches around differences in 
> > >>> Path semantics in Java and Windows
> > >>> - Native Task Controller for Windows
> > >>> - Implementation of a Block Placement Policy to support cloud 
> > >>> environments, more specifically Azure.
> > >>> - Implementation of Hadoop native libraries for Windows 
> > >>> (compression codecs, native I/O) - Several reliability issues, 
> > >>> including race-conditions, intermittent test failures, resource leaks.
> > >>> - Several new unit test cases written for the above changes
> > >>>
> > >>> In the process, we have closely engaged with the Apache open 
> > >>> source community and have got great support and assistance from 
> > >>> the
> community
> > >>>in
> > >>> terms of contributing fixes, code review comments and commits.
> > >>>
> > >>> In addition, the Hadoop team at Microsoft has also made good 
> > >>> progress
> > in
> > >>> other projects including Hive, Pig, Sqoop, Oozie, HCat and HBase.
> Many
> > >>>of
> > >>> these changes have already been committed to the respective 
> > >>>trunks
> with
> > >>> help from various committers and contributors. It is great to 
> > >>> see the commitment of the community to support multiple 
> > >>> platforms, and we
> look
> > >>> forward to the day when a developer/customer is able to 
> > >>>successfully deploy  a complete solution stack based on Apache 
> > >>>Hadoop releases.
> > >>>
> > >>> Next Steps:
> > >>>
> > >>> All of the above changes are part of the Windows Azure HDInsight 
> > >>>and  HDInsight Server products from Microsoft. We have 
> > >>>successfully on-boarded  several internal customers and have been 
> > >>>running production workloads
> > on
> > >>> Windows Azure HDInsight. Our vision is to create a big data 
> > >>>platform based  on Hadoop, and we are committed to helping make 
> > >>>Hadoop a world-class  solution that anyone can use to solve their 
> > >>>biggest data challenges.
> > >>>
> > >>> As an immediate next step, we would like to have a discussion 
> > >>> around
> > how
> > >>> we can ensure that the quality of the mainline Hadoop branches 
> > >>>on Windows  is maintained. To this end, we would like to get to 
> > >>>the state where
> we
> > >>>have
> > >>> pre-checkin validation gates and nightly test runs enabled on
> Windows.
> > >>>If
> > >>> you have any suggestions around this, please do send an email.  
> > >>>We
> are
> > >>> committed to helping sustain the long-term quality of Hadoop on 
> > >>>both Linux  and Windows.
> > >>>
> > >>> We sincerely thank the community for their contribution and 
> > >>> support
> so
> > >>> far. And hope to continue having a close engagement in the future.
> > >>>
> > >>> -Microsoft HDInsight Team
> > >>>
> > >>>
> > >>> -----Original Message-----
> > >>> From: Suresh Srinivas [mailto:suresh@hortonworks.com]
> > >>> Sent: Thursday, February 7, 2013 5:42 PM
> > >>> To: common-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org; 
> > >>> hdfs-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
> > >>> Subject: Heads up - merge branch-trunk-win to trunk
> > >>>
> > >>> The support for Hadoop on Windows was proposed in HADOOP-8079< 
> > >>> https://issues.apache.org/jira/browse/HADOOP-8079> almost a year
> ago.
> > >>>The
> > >>> goal was to make Hadoop natively integrated, full-featured, and 
> > >>>performance  and scalability tuned on Windows Server or Windows 
> > >>>Azure.
> > >>> We are happy to announce that a lot of progress has been made in 
> > >>>this  regard.
> > >>>
> > >>> Initial work started in a feature branch, branch-1-win, based on 
> > >>>branch-1.
> > >>> The details related to the work done in the branch can be seen 
> > >>>in  CHANGES.txt<
> > >>>
> > >>>
> > http://svn.apache.org/viewvc/hadoop/common/branches/branch-1-win/CHA
> > NGES
> .
> > >>>branch-1-win.txt?view=markup
> > >>> >.
> > >>> This work has been ported to a branch, branch-trunk-win, based 
> > >>> on
> > trunk.
> > >>> Merge patch for this is available on 
> > >>> HADOOP-8562<https://issues.apache.org/jira/browse/HADOOP-8562>
> > >>> .
> > >>>
> > >>> Highlights of the work done so far:
> > >>> 1. Necessary changes in Hadoop to run natively on Windows. These
> > changes
> > >>> handle differences in platforms related to path names, 
> > >>>process/task  management etc.
> > >>> 2. Addition of winutils tools for managing file permissions and 
> > >>>ownership,  user group mapping, hardlinks, symbolic links, chmod, 
> > >>>disk
> utilization,
> > >>>and
> > >>> process/task management.
> > >>> 3. Added cmd scripts equivalent to existing shell scripts 
> > >>>hadoop-daemon.sh, start and stop scripts.
> > >>> 4. Addition of block placement policy implemnation to support 
> > >>>cloud  enviroment, more specifically Azure.
> > >>>
> > >>> We are very close to wrapping up the work in branch-trunk-win 
> > >>>and getting  ready for a merge. Currently the merge patch is 
> > >>>passing close to 100%
> > of
> > >>> unit tests on Linux. Soon I will call for a vote to merge this 
> > >>>branch into  trunk.
> > >>>
> > >>> Next steps:
> > >>> 1. Call for vote to merge branch-trunk-win to trunk, when the 
> > >>> work completes and precommit build is clean.
> > >>> 2. Start a discussion on adding Jenkins precommit builds on 
> > >>> windows
> and
> > >>> how to integrate that with the existing commit process.
> > >>>
> > >>> Let me know if you have any questions.
> > >>>
> > >>> Regards,
> > >>> Suresh
> > >>>
> > >>>
> > >>
> > >>
> > >>--
> > >>http://hortonworks.com/download/
> > >
> >
>
>
>
> --
> http://hortonworks.com/download/
>



--
Todd Lipcon
Software Engineer, Cloudera

RE: [Vote] Merge branch-trunk-win to trunk

[Ivan Mitic <iv...@microsoft.com>]

+1 (non-binding)

I am really glad to see this happening! As people already mentioned, this has been a great engineering effort involving many people!


Folks raised some valid concerns below and I thought it would be good to share my 2 cents. In my opinion, we don't have to solve all these problems right now. As we move forward with two platforms, we can start addressing one problem at a time and incrementally improve. In the first iteration, maintaining Hadoop on Windows could be just everyone trying to do their best effort (make sure Jenkins build succeeds at least). We already have people who are building/running trunk on Windows daily, so they would jump in and fix problems as needed (we've been doing this in branch-trunk-win for a while now). Although I see that the problems could arise with platform specific features/optimizations, I don't think these are frequent, so in most cases everything will just work. Merging the two branches sooner rather than later does seems like the right thing to do if the ultimate goal is to have Hadoop on both platforms. Now that the port has completed, we will have people in Microsoft (and elsewhere) wanting to contribute features/improvements to the trunk branch. A separate branch would just make things more difficult and confusing for everyone :) Hope this makes sense.

-----Original Message-----
From: Todd Lipcon [mailto:todd@cloudera.com] 
Sent: Wednesday, February 27, 2013 3:43 PM
To: common-dev@hadoop.apache.org
Cc: yarn-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
Subject: Re: [Vote] Merge branch-trunk-win to trunk

On Wed, Feb 27, 2013 at 2:54 PM, Suresh Srinivas <su...@hortonworks.com>wrote:

> With that we need to decide how our precommit process looks.
> My inclination is to wait for +1 from precommit builds on both the 
> platforms to ensure no issues are introduced.
> Thoughts?
>
> 2. Feature development impact
> Some questions have been raised about would new features need to be 
> supported on both the platforms. Yes. I do not see a reason why 
> features cannot work on both the platforms, with the exception of 
> platform specific optimizations. This what Java gives us.
>
>
I'm concerned about the above. Personally, I don't have access to any Windows boxes with development tools, and I know nothing about developing on Windows. The only Windows I run is an 8GB VM with 1 GB RAM allocated, for powerpoint :)

If I submit a patch and it gets -1 "tests failed" on the Windows slave, how am I supposed to proceed?

I think a reasonable compromise would be that the tests should always
*build* on Windows before commit, and contributors should do their best to look at the test logs for any Windows-specific failures. But, beyond looking at the logs, a "-1 Tests failed on windows" should not block a commit.

Those contributors who are interested in Windows being a first-class platform should be responsible for watching the Windows builds and debugging/fixing any regressions that might be Windows-specific.

I also think the KDE model that Harsh pointed out is an interesting one -- ie the idea that we would not merge windows support to trunk, but rather treat is as a "parallel code line" which lives in the ASF and has its own builds and releases. The windows team would periodically merge trunk->win to pick up any new changes, and do a separate test/release process. I'm not convinced this is the best idea, but worth discussion of pros and cons.

-Todd


>
> On Wed, Feb 27, 2013 at 11:56 AM, Eli Collins <el...@cloudera.com> wrote:
>
> > Bobby raises some good questions.  A related one, since most current 
> > developers won't add Windows support for new features that are 
> > platform specific is it assumed that Windows development will either 
> > lag or will people actively work on keeping Windows up with the 
> > latest?  And vice versa in case Windows support is implemented first.
> >
> > Is there a jira for resolving the outstanding TODOs in the code base 
> > (similar to HDFS-2148)?  Looks like this merge doesn't introduce 
> > many which is great (just did a quick diff and grep).
> >
> > Thanks,
> > Eli
> >
> > On Wed, Feb 27, 2013 at 8:17 AM, Robert Evans <ev...@yahoo-inc.com>
> wrote:
> > > After this is merged in is Windows still going to be a second 
> > > class citizen but happens to work for more than just development 
> > > or is it a fully supported platform where if something breaks it 
> > > can block a
> > release?
> > >  How do we as a community intend to keep Windows support from breaking?
> > > We don't have any Jenkins slaves to be able to run nightly tests 
> > > to validate everything still compiles/runs.  This is not a blocker 
> > > for me because we often rely on individuals and groups to test 
> > > Hadoop, but I
> do
> > > think we need to have this discussion before we put it in.
> > >
> > > --Bobby
> > >
> > > On 2/26/13 4:55 PM, "Suresh Srinivas" <su...@hortonworks.com> wrote:
> > >
> > >>I had posted heads up about merging branch-trunk-win to trunk on 
> > >>Feb
> 8th.
> > >>I
> > >>am happy to announce that we are ready for the merge.
> > >>
> > >>Here is a brief recap on the highlights of the work done:
> > >>- Command-line scripts for the Hadoop surface area
> > >>- Mapping the HDFS permissions model to Windows
> > >>- Abstracted and reconciled mismatches around differences in Path 
> > >>semantics in Java and Windows
> > >>- Native Task Controller for Windows
> > >>- Implementation of a Block Placement Policy to support cloud 
> > >>environments, more specifically Azure.
> > >>- Implementation of Hadoop native libraries for Windows 
> > >>(compression codecs, native I/O)
> > >>- Several reliability issues, including race-conditions, 
> > >>intermittent
> > test
> > >>failures, resource leaks.
> > >>- Several new unit test cases written for the above changes
> > >>
> > >>Please find the details of the work in 
> > >>CHANGES.branch-trunk-win.txt - Common 
> > >>changes<http://bit.ly/Xe7Ynv>, HDFS changes<
> http://bit.ly/13QOSo9
> > >,
> > >>and YARN and MapReduce changes <http://bit.ly/128zzMt>. This is 
> > >>the
> work
> > >>ported from branch-1-win to a branch based on trunk.
> > >>
> > >>For details of the testing done, please see the thread - 
> > >>http://bit.ly/WpavJ4. Merge patch for this is available on
> HADOOP-8562<
> > >>https://issues.apache.org/jira/browse/HADOOP-8562>.
> > >>
> > >>This was a large undertaking that involved developing code, 
> > >>testing the entire Hadoop stack, including scale tests. This is 
> > >>made possible only with the contribution from many many folks in 
> > >>the community. Following
> people
> > >>contributed to this work: Ivan Mitic, Chuan Liu, Ramya Sunil, 
> > >>Bikas
> Saha,
> > >>Kanna Karanam, John Gordon, Brandon Li, Chris Nauroth, David Lao,
> > Sumadhur
> > >>Reddy Bolli, Arpit Agarwal, Ahmed El Baz, Mike Liddell, Jing Zhao,
> Thejas
> > >>Nair, Steve Maine, Ganeshan Iyer, Raja Aluri, Giridharan Kesavan, 
> > >>Ramya Bharathi Nimmagadda, Daryn Sharp, Arun Murthy, Tsz-Wo 
> > >>Nicholas Sze,
> > Suresh
> > >>Srinivas and Sanjay Radia. There are many others who contributed 
> > >>as
> well
> > >>providing feedback and comments on numerous jiras.
> > >>
> > >>The vote will run for seven days and will end on March 5, 6:00PM PST.
> > >>
> > >>Regards,
> > >>Suresh
> > >>
> > >>
> > >>
> > >>
> > >>On Thu, Feb 7, 2013 at 6:41 PM, Mahadevan Venkatraman
> > >><ma...@microsoft.com>wrote:
> > >>
> > >>> It is super exciting to look at the prospect of these changes 
> > >>>being merged  to trunk. Having Windows as one of the supported 
> > >>>Hadoop platforms is
> a
> > >>> fantastic opportunity both for the Hadoop project and Microsoft 
> > >>>customers.
> > >>>
> > >>> This work began around a year back when a few of us started with 
> > >>> a
> > basic
> > >>> port of Hadoop on Windows. Ever since, the Hadoop team in 
> > >>> Microsoft
> > have
> > >>> made significant progress in the following areas:
> > >>> (PS: Some of these items are already included in Suresh's email, 
> > >>> but including again for completeness)
> > >>>
> > >>> - Command-line scripts for the Hadoop surface area
> > >>> - Mapping the HDFS permissions model to Windows
> > >>> - Abstracted and reconciled mismatches around differences in 
> > >>> Path semantics in Java and Windows
> > >>> - Native Task Controller for Windows
> > >>> - Implementation of a Block Placement Policy to support cloud 
> > >>> environments, more specifically Azure.
> > >>> - Implementation of Hadoop native libraries for Windows 
> > >>> (compression codecs, native I/O) - Several reliability issues, 
> > >>> including race-conditions, intermittent test failures, resource leaks.
> > >>> - Several new unit test cases written for the above changes
> > >>>
> > >>> In the process, we have closely engaged with the Apache open 
> > >>> source community and have got great support and assistance from 
> > >>> the
> community
> > >>>in
> > >>> terms of contributing fixes, code review comments and commits.
> > >>>
> > >>> In addition, the Hadoop team at Microsoft has also made good 
> > >>> progress
> > in
> > >>> other projects including Hive, Pig, Sqoop, Oozie, HCat and HBase.
> Many
> > >>>of
> > >>> these changes have already been committed to the respective 
> > >>>trunks
> with
> > >>> help from various committers and contributors. It is great to 
> > >>> see the commitment of the community to support multiple 
> > >>> platforms, and we
> look
> > >>> forward to the day when a developer/customer is able to 
> > >>>successfully deploy  a complete solution stack based on Apache 
> > >>>Hadoop releases.
> > >>>
> > >>> Next Steps:
> > >>>
> > >>> All of the above changes are part of the Windows Azure HDInsight 
> > >>>and  HDInsight Server products from Microsoft. We have 
> > >>>successfully on-boarded  several internal customers and have been 
> > >>>running production workloads
> > on
> > >>> Windows Azure HDInsight. Our vision is to create a big data 
> > >>>platform based  on Hadoop, and we are committed to helping make 
> > >>>Hadoop a world-class  solution that anyone can use to solve their 
> > >>>biggest data challenges.
> > >>>
> > >>> As an immediate next step, we would like to have a discussion 
> > >>> around
> > how
> > >>> we can ensure that the quality of the mainline Hadoop branches 
> > >>>on Windows  is maintained. To this end, we would like to get to 
> > >>>the state where
> we
> > >>>have
> > >>> pre-checkin validation gates and nightly test runs enabled on
> Windows.
> > >>>If
> > >>> you have any suggestions around this, please do send an email.  
> > >>>We
> are
> > >>> committed to helping sustain the long-term quality of Hadoop on 
> > >>>both Linux  and Windows.
> > >>>
> > >>> We sincerely thank the community for their contribution and 
> > >>> support
> so
> > >>> far. And hope to continue having a close engagement in the future.
> > >>>
> > >>> -Microsoft HDInsight Team
> > >>>
> > >>>
> > >>> -----Original Message-----
> > >>> From: Suresh Srinivas [mailto:suresh@hortonworks.com]
> > >>> Sent: Thursday, February 7, 2013 5:42 PM
> > >>> To: common-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org; 
> > >>> hdfs-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
> > >>> Subject: Heads up - merge branch-trunk-win to trunk
> > >>>
> > >>> The support for Hadoop on Windows was proposed in HADOOP-8079< 
> > >>> https://issues.apache.org/jira/browse/HADOOP-8079> almost a year
> ago.
> > >>>The
> > >>> goal was to make Hadoop natively integrated, full-featured, and 
> > >>>performance  and scalability tuned on Windows Server or Windows 
> > >>>Azure.
> > >>> We are happy to announce that a lot of progress has been made in 
> > >>>this  regard.
> > >>>
> > >>> Initial work started in a feature branch, branch-1-win, based on 
> > >>>branch-1.
> > >>> The details related to the work done in the branch can be seen 
> > >>>in  CHANGES.txt<
> > >>>
> > >>>
> > http://svn.apache.org/viewvc/hadoop/common/branches/branch-1-win/CHA
> > NGES
> .
> > >>>branch-1-win.txt?view=markup
> > >>> >.
> > >>> This work has been ported to a branch, branch-trunk-win, based 
> > >>> on
> > trunk.
> > >>> Merge patch for this is available on 
> > >>> HADOOP-8562<https://issues.apache.org/jira/browse/HADOOP-8562>
> > >>> .
> > >>>
> > >>> Highlights of the work done so far:
> > >>> 1. Necessary changes in Hadoop to run natively on Windows. These
> > changes
> > >>> handle differences in platforms related to path names, 
> > >>>process/task  management etc.
> > >>> 2. Addition of winutils tools for managing file permissions and 
> > >>>ownership,  user group mapping, hardlinks, symbolic links, chmod, 
> > >>>disk
> utilization,
> > >>>and
> > >>> process/task management.
> > >>> 3. Added cmd scripts equivalent to existing shell scripts  
> > >>>hadoop-daemon.sh, start and stop scripts.
> > >>> 4. Addition of block placement policy implemnation to support 
> > >>>cloud  enviroment, more specifically Azure.
> > >>>
> > >>> We are very close to wrapping up the work in branch-trunk-win 
> > >>>and getting  ready for a merge. Currently the merge patch is 
> > >>>passing close to 100%
> > of
> > >>> unit tests on Linux. Soon I will call for a vote to merge this 
> > >>>branch into  trunk.
> > >>>
> > >>> Next steps:
> > >>> 1. Call for vote to merge branch-trunk-win to trunk, when the 
> > >>> work completes and precommit build is clean.
> > >>> 2. Start a discussion on adding Jenkins precommit builds on 
> > >>> windows
> and
> > >>> how to integrate that with the existing commit process.
> > >>>
> > >>> Let me know if you have any questions.
> > >>>
> > >>> Regards,
> > >>> Suresh
> > >>>
> > >>>
> > >>
> > >>
> > >>--
> > >>http://hortonworks.com/download/
> > >
> >
>
>
>
> --
> http://hortonworks.com/download/
>



--
Todd Lipcon
Software Engineer, Cloudera

RE: [Vote] Merge branch-trunk-win to trunk

[Ivan Mitic <iv...@microsoft.com>]

+1 (non-binding)

I am really glad to see this happening! As people already mentioned, this has been a great engineering effort involving many people!


Folks raised some valid concerns below and I thought it would be good to share my 2 cents. In my opinion, we don't have to solve all these problems right now. As we move forward with two platforms, we can start addressing one problem at a time and incrementally improve. In the first iteration, maintaining Hadoop on Windows could be just everyone trying to do their best effort (make sure Jenkins build succeeds at least). We already have people who are building/running trunk on Windows daily, so they would jump in and fix problems as needed (we've been doing this in branch-trunk-win for a while now). Although I see that the problems could arise with platform specific features/optimizations, I don't think these are frequent, so in most cases everything will just work. Merging the two branches sooner rather than later does seems like the right thing to do if the ultimate goal is to have Hadoop on both platforms. Now that the port has completed, we will have people in Microsoft (and elsewhere) wanting to contribute features/improvements to the trunk branch. A separate branch would just make things more difficult and confusing for everyone :) Hope this makes sense.

-----Original Message-----
From: Todd Lipcon [mailto:todd@cloudera.com] 
Sent: Wednesday, February 27, 2013 3:43 PM
To: common-dev@hadoop.apache.org
Cc: yarn-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
Subject: Re: [Vote] Merge branch-trunk-win to trunk

On Wed, Feb 27, 2013 at 2:54 PM, Suresh Srinivas <su...@hortonworks.com>wrote:

> With that we need to decide how our precommit process looks.
> My inclination is to wait for +1 from precommit builds on both the 
> platforms to ensure no issues are introduced.
> Thoughts?
>
> 2. Feature development impact
> Some questions have been raised about would new features need to be 
> supported on both the platforms. Yes. I do not see a reason why 
> features cannot work on both the platforms, with the exception of 
> platform specific optimizations. This what Java gives us.
>
>
I'm concerned about the above. Personally, I don't have access to any Windows boxes with development tools, and I know nothing about developing on Windows. The only Windows I run is an 8GB VM with 1 GB RAM allocated, for powerpoint :)

If I submit a patch and it gets -1 "tests failed" on the Windows slave, how am I supposed to proceed?

I think a reasonable compromise would be that the tests should always
*build* on Windows before commit, and contributors should do their best to look at the test logs for any Windows-specific failures. But, beyond looking at the logs, a "-1 Tests failed on windows" should not block a commit.

Those contributors who are interested in Windows being a first-class platform should be responsible for watching the Windows builds and debugging/fixing any regressions that might be Windows-specific.

I also think the KDE model that Harsh pointed out is an interesting one -- ie the idea that we would not merge windows support to trunk, but rather treat is as a "parallel code line" which lives in the ASF and has its own builds and releases. The windows team would periodically merge trunk->win to pick up any new changes, and do a separate test/release process. I'm not convinced this is the best idea, but worth discussion of pros and cons.

-Todd


>
> On Wed, Feb 27, 2013 at 11:56 AM, Eli Collins <el...@cloudera.com> wrote:
>
> > Bobby raises some good questions.  A related one, since most current 
> > developers won't add Windows support for new features that are 
> > platform specific is it assumed that Windows development will either 
> > lag or will people actively work on keeping Windows up with the 
> > latest?  And vice versa in case Windows support is implemented first.
> >
> > Is there a jira for resolving the outstanding TODOs in the code base 
> > (similar to HDFS-2148)?  Looks like this merge doesn't introduce 
> > many which is great (just did a quick diff and grep).
> >
> > Thanks,
> > Eli
> >
> > On Wed, Feb 27, 2013 at 8:17 AM, Robert Evans <ev...@yahoo-inc.com>
> wrote:
> > > After this is merged in is Windows still going to be a second 
> > > class citizen but happens to work for more than just development 
> > > or is it a fully supported platform where if something breaks it 
> > > can block a
> > release?
> > >  How do we as a community intend to keep Windows support from breaking?
> > > We don't have any Jenkins slaves to be able to run nightly tests 
> > > to validate everything still compiles/runs.  This is not a blocker 
> > > for me because we often rely on individuals and groups to test 
> > > Hadoop, but I
> do
> > > think we need to have this discussion before we put it in.
> > >
> > > --Bobby
> > >
> > > On 2/26/13 4:55 PM, "Suresh Srinivas" <su...@hortonworks.com> wrote:
> > >
> > >>I had posted heads up about merging branch-trunk-win to trunk on 
> > >>Feb
> 8th.
> > >>I
> > >>am happy to announce that we are ready for the merge.
> > >>
> > >>Here is a brief recap on the highlights of the work done:
> > >>- Command-line scripts for the Hadoop surface area
> > >>- Mapping the HDFS permissions model to Windows
> > >>- Abstracted and reconciled mismatches around differences in Path 
> > >>semantics in Java and Windows
> > >>- Native Task Controller for Windows
> > >>- Implementation of a Block Placement Policy to support cloud 
> > >>environments, more specifically Azure.
> > >>- Implementation of Hadoop native libraries for Windows 
> > >>(compression codecs, native I/O)
> > >>- Several reliability issues, including race-conditions, 
> > >>intermittent
> > test
> > >>failures, resource leaks.
> > >>- Several new unit test cases written for the above changes
> > >>
> > >>Please find the details of the work in 
> > >>CHANGES.branch-trunk-win.txt - Common 
> > >>changes<http://bit.ly/Xe7Ynv>, HDFS changes<
> http://bit.ly/13QOSo9
> > >,
> > >>and YARN and MapReduce changes <http://bit.ly/128zzMt>. This is 
> > >>the
> work
> > >>ported from branch-1-win to a branch based on trunk.
> > >>
> > >>For details of the testing done, please see the thread - 
> > >>http://bit.ly/WpavJ4. Merge patch for this is available on
> HADOOP-8562<
> > >>https://issues.apache.org/jira/browse/HADOOP-8562>.
> > >>
> > >>This was a large undertaking that involved developing code, 
> > >>testing the entire Hadoop stack, including scale tests. This is 
> > >>made possible only with the contribution from many many folks in 
> > >>the community. Following
> people
> > >>contributed to this work: Ivan Mitic, Chuan Liu, Ramya Sunil, 
> > >>Bikas
> Saha,
> > >>Kanna Karanam, John Gordon, Brandon Li, Chris Nauroth, David Lao,
> > Sumadhur
> > >>Reddy Bolli, Arpit Agarwal, Ahmed El Baz, Mike Liddell, Jing Zhao,
> Thejas
> > >>Nair, Steve Maine, Ganeshan Iyer, Raja Aluri, Giridharan Kesavan, 
> > >>Ramya Bharathi Nimmagadda, Daryn Sharp, Arun Murthy, Tsz-Wo 
> > >>Nicholas Sze,
> > Suresh
> > >>Srinivas and Sanjay Radia. There are many others who contributed 
> > >>as
> well
> > >>providing feedback and comments on numerous jiras.
> > >>
> > >>The vote will run for seven days and will end on March 5, 6:00PM PST.
> > >>
> > >>Regards,
> > >>Suresh
> > >>
> > >>
> > >>
> > >>
> > >>On Thu, Feb 7, 2013 at 6:41 PM, Mahadevan Venkatraman
> > >><ma...@microsoft.com>wrote:
> > >>
> > >>> It is super exciting to look at the prospect of these changes 
> > >>>being merged  to trunk. Having Windows as one of the supported 
> > >>>Hadoop platforms is
> a
> > >>> fantastic opportunity both for the Hadoop project and Microsoft 
> > >>>customers.
> > >>>
> > >>> This work began around a year back when a few of us started with 
> > >>> a
> > basic
> > >>> port of Hadoop on Windows. Ever since, the Hadoop team in 
> > >>> Microsoft
> > have
> > >>> made significant progress in the following areas:
> > >>> (PS: Some of these items are already included in Suresh's email, 
> > >>> but including again for completeness)
> > >>>
> > >>> - Command-line scripts for the Hadoop surface area
> > >>> - Mapping the HDFS permissions model to Windows
> > >>> - Abstracted and reconciled mismatches around differences in 
> > >>> Path semantics in Java and Windows
> > >>> - Native Task Controller for Windows
> > >>> - Implementation of a Block Placement Policy to support cloud 
> > >>> environments, more specifically Azure.
> > >>> - Implementation of Hadoop native libraries for Windows 
> > >>> (compression codecs, native I/O) - Several reliability issues, 
> > >>> including race-conditions, intermittent test failures, resource leaks.
> > >>> - Several new unit test cases written for the above changes
> > >>>
> > >>> In the process, we have closely engaged with the Apache open 
> > >>> source community and have got great support and assistance from 
> > >>> the
> community
> > >>>in
> > >>> terms of contributing fixes, code review comments and commits.
> > >>>
> > >>> In addition, the Hadoop team at Microsoft has also made good 
> > >>> progress
> > in
> > >>> other projects including Hive, Pig, Sqoop, Oozie, HCat and HBase.
> Many
> > >>>of
> > >>> these changes have already been committed to the respective 
> > >>>trunks
> with
> > >>> help from various committers and contributors. It is great to 
> > >>> see the commitment of the community to support multiple 
> > >>> platforms, and we
> look
> > >>> forward to the day when a developer/customer is able to 
> > >>>successfully deploy  a complete solution stack based on Apache 
> > >>>Hadoop releases.
> > >>>
> > >>> Next Steps:
> > >>>
> > >>> All of the above changes are part of the Windows Azure HDInsight 
> > >>>and  HDInsight Server products from Microsoft. We have 
> > >>>successfully on-boarded  several internal customers and have been 
> > >>>running production workloads
> > on
> > >>> Windows Azure HDInsight. Our vision is to create a big data 
> > >>>platform based  on Hadoop, and we are committed to helping make 
> > >>>Hadoop a world-class  solution that anyone can use to solve their 
> > >>>biggest data challenges.
> > >>>
> > >>> As an immediate next step, we would like to have a discussion 
> > >>> around
> > how
> > >>> we can ensure that the quality of the mainline Hadoop branches 
> > >>>on Windows  is maintained. To this end, we would like to get to 
> > >>>the state where
> we
> > >>>have
> > >>> pre-checkin validation gates and nightly test runs enabled on
> Windows.
> > >>>If
> > >>> you have any suggestions around this, please do send an email.  
> > >>>We
> are
> > >>> committed to helping sustain the long-term quality of Hadoop on 
> > >>>both Linux  and Windows.
> > >>>
> > >>> We sincerely thank the community for their contribution and 
> > >>> support
> so
> > >>> far. And hope to continue having a close engagement in the future.
> > >>>
> > >>> -Microsoft HDInsight Team
> > >>>
> > >>>
> > >>> -----Original Message-----
> > >>> From: Suresh Srinivas [mailto:suresh@hortonworks.com]
> > >>> Sent: Thursday, February 7, 2013 5:42 PM
> > >>> To: common-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org; 
> > >>> hdfs-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
> > >>> Subject: Heads up - merge branch-trunk-win to trunk
> > >>>
> > >>> The support for Hadoop on Windows was proposed in HADOOP-8079< 
> > >>> https://issues.apache.org/jira/browse/HADOOP-8079> almost a year
> ago.
> > >>>The
> > >>> goal was to make Hadoop natively integrated, full-featured, and 
> > >>>performance  and scalability tuned on Windows Server or Windows 
> > >>>Azure.
> > >>> We are happy to announce that a lot of progress has been made in 
> > >>>this  regard.
> > >>>
> > >>> Initial work started in a feature branch, branch-1-win, based on 
> > >>>branch-1.
> > >>> The details related to the work done in the branch can be seen 
> > >>>in  CHANGES.txt<
> > >>>
> > >>>
> > http://svn.apache.org/viewvc/hadoop/common/branches/branch-1-win/CHA
> > NGES
> .
> > >>>branch-1-win.txt?view=markup
> > >>> >.
> > >>> This work has been ported to a branch, branch-trunk-win, based 
> > >>> on
> > trunk.
> > >>> Merge patch for this is available on 
> > >>> HADOOP-8562<https://issues.apache.org/jira/browse/HADOOP-8562>
> > >>> .
> > >>>
> > >>> Highlights of the work done so far:
> > >>> 1. Necessary changes in Hadoop to run natively on Windows. These
> > changes
> > >>> handle differences in platforms related to path names, 
> > >>>process/task  management etc.
> > >>> 2. Addition of winutils tools for managing file permissions and 
> > >>>ownership,  user group mapping, hardlinks, symbolic links, chmod, 
> > >>>disk
> utilization,
> > >>>and
> > >>> process/task management.
> > >>> 3. Added cmd scripts equivalent to existing shell scripts  
> > >>>hadoop-daemon.sh, start and stop scripts.
> > >>> 4. Addition of block placement policy implemnation to support 
> > >>>cloud  enviroment, more specifically Azure.
> > >>>
> > >>> We are very close to wrapping up the work in branch-trunk-win 
> > >>>and getting  ready for a merge. Currently the merge patch is 
> > >>>passing close to 100%
> > of
> > >>> unit tests on Linux. Soon I will call for a vote to merge this 
> > >>>branch into  trunk.
> > >>>
> > >>> Next steps:
> > >>> 1. Call for vote to merge branch-trunk-win to trunk, when the 
> > >>> work completes and precommit build is clean.
> > >>> 2. Start a discussion on adding Jenkins precommit builds on 
> > >>> windows
> and
> > >>> how to integrate that with the existing commit process.
> > >>>
> > >>> Let me know if you have any questions.
> > >>>
> > >>> Regards,
> > >>> Suresh
> > >>>
> > >>>
> > >>
> > >>
> > >>--
> > >>http://hortonworks.com/download/
> > >
> >
>
>
>
> --
> http://hortonworks.com/download/
>



--
Todd Lipcon
Software Engineer, Cloudera

RE: [Vote] Merge branch-trunk-win to trunk

[Ivan Mitic <iv...@microsoft.com>]

+1 (non-binding)

I am really glad to see this happening! As people already mentioned, this has been a great engineering effort involving many people!


Folks raised some valid concerns below and I thought it would be good to share my 2 cents. In my opinion, we don't have to solve all these problems right now. As we move forward with two platforms, we can start addressing one problem at a time and incrementally improve. In the first iteration, maintaining Hadoop on Windows could be just everyone trying to do their best effort (make sure Jenkins build succeeds at least). We already have people who are building/running trunk on Windows daily, so they would jump in and fix problems as needed (we've been doing this in branch-trunk-win for a while now). Although I see that the problems could arise with platform specific features/optimizations, I don't think these are frequent, so in most cases everything will just work. Merging the two branches sooner rather than later does seems like the right thing to do if the ultimate goal is to have Hadoop on both platforms. Now that the port has completed, we will have people in Microsoft (and elsewhere) wanting to contribute features/improvements to the trunk branch. A separate branch would just make things more difficult and confusing for everyone :) Hope this makes sense.

-----Original Message-----
From: Todd Lipcon [mailto:todd@cloudera.com] 
Sent: Wednesday, February 27, 2013 3:43 PM
To: common-dev@hadoop.apache.org
Cc: yarn-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
Subject: Re: [Vote] Merge branch-trunk-win to trunk

On Wed, Feb 27, 2013 at 2:54 PM, Suresh Srinivas <su...@hortonworks.com>wrote:

> With that we need to decide how our precommit process looks.
> My inclination is to wait for +1 from precommit builds on both the 
> platforms to ensure no issues are introduced.
> Thoughts?
>
> 2. Feature development impact
> Some questions have been raised about would new features need to be 
> supported on both the platforms. Yes. I do not see a reason why 
> features cannot work on both the platforms, with the exception of 
> platform specific optimizations. This what Java gives us.
>
>
I'm concerned about the above. Personally, I don't have access to any Windows boxes with development tools, and I know nothing about developing on Windows. The only Windows I run is an 8GB VM with 1 GB RAM allocated, for powerpoint :)

If I submit a patch and it gets -1 "tests failed" on the Windows slave, how am I supposed to proceed?

I think a reasonable compromise would be that the tests should always
*build* on Windows before commit, and contributors should do their best to look at the test logs for any Windows-specific failures. But, beyond looking at the logs, a "-1 Tests failed on windows" should not block a commit.

Those contributors who are interested in Windows being a first-class platform should be responsible for watching the Windows builds and debugging/fixing any regressions that might be Windows-specific.

I also think the KDE model that Harsh pointed out is an interesting one -- ie the idea that we would not merge windows support to trunk, but rather treat is as a "parallel code line" which lives in the ASF and has its own builds and releases. The windows team would periodically merge trunk->win to pick up any new changes, and do a separate test/release process. I'm not convinced this is the best idea, but worth discussion of pros and cons.

-Todd


>
> On Wed, Feb 27, 2013 at 11:56 AM, Eli Collins <el...@cloudera.com> wrote:
>
> > Bobby raises some good questions.  A related one, since most current 
> > developers won't add Windows support for new features that are 
> > platform specific is it assumed that Windows development will either 
> > lag or will people actively work on keeping Windows up with the 
> > latest?  And vice versa in case Windows support is implemented first.
> >
> > Is there a jira for resolving the outstanding TODOs in the code base 
> > (similar to HDFS-2148)?  Looks like this merge doesn't introduce 
> > many which is great (just did a quick diff and grep).
> >
> > Thanks,
> > Eli
> >
> > On Wed, Feb 27, 2013 at 8:17 AM, Robert Evans <ev...@yahoo-inc.com>
> wrote:
> > > After this is merged in is Windows still going to be a second 
> > > class citizen but happens to work for more than just development 
> > > or is it a fully supported platform where if something breaks it 
> > > can block a
> > release?
> > >  How do we as a community intend to keep Windows support from breaking?
> > > We don't have any Jenkins slaves to be able to run nightly tests 
> > > to validate everything still compiles/runs.  This is not a blocker 
> > > for me because we often rely on individuals and groups to test 
> > > Hadoop, but I
> do
> > > think we need to have this discussion before we put it in.
> > >
> > > --Bobby
> > >
> > > On 2/26/13 4:55 PM, "Suresh Srinivas" <su...@hortonworks.com> wrote:
> > >
> > >>I had posted heads up about merging branch-trunk-win to trunk on 
> > >>Feb
> 8th.
> > >>I
> > >>am happy to announce that we are ready for the merge.
> > >>
> > >>Here is a brief recap on the highlights of the work done:
> > >>- Command-line scripts for the Hadoop surface area
> > >>- Mapping the HDFS permissions model to Windows
> > >>- Abstracted and reconciled mismatches around differences in Path 
> > >>semantics in Java and Windows
> > >>- Native Task Controller for Windows
> > >>- Implementation of a Block Placement Policy to support cloud 
> > >>environments, more specifically Azure.
> > >>- Implementation of Hadoop native libraries for Windows 
> > >>(compression codecs, native I/O)
> > >>- Several reliability issues, including race-conditions, 
> > >>intermittent
> > test
> > >>failures, resource leaks.
> > >>- Several new unit test cases written for the above changes
> > >>
> > >>Please find the details of the work in 
> > >>CHANGES.branch-trunk-win.txt - Common 
> > >>changes<http://bit.ly/Xe7Ynv>, HDFS changes<
> http://bit.ly/13QOSo9
> > >,
> > >>and YARN and MapReduce changes <http://bit.ly/128zzMt>. This is 
> > >>the
> work
> > >>ported from branch-1-win to a branch based on trunk.
> > >>
> > >>For details of the testing done, please see the thread - 
> > >>http://bit.ly/WpavJ4. Merge patch for this is available on
> HADOOP-8562<
> > >>https://issues.apache.org/jira/browse/HADOOP-8562>.
> > >>
> > >>This was a large undertaking that involved developing code, 
> > >>testing the entire Hadoop stack, including scale tests. This is 
> > >>made possible only with the contribution from many many folks in 
> > >>the community. Following
> people
> > >>contributed to this work: Ivan Mitic, Chuan Liu, Ramya Sunil, 
> > >>Bikas
> Saha,
> > >>Kanna Karanam, John Gordon, Brandon Li, Chris Nauroth, David Lao,
> > Sumadhur
> > >>Reddy Bolli, Arpit Agarwal, Ahmed El Baz, Mike Liddell, Jing Zhao,
> Thejas
> > >>Nair, Steve Maine, Ganeshan Iyer, Raja Aluri, Giridharan Kesavan, 
> > >>Ramya Bharathi Nimmagadda, Daryn Sharp, Arun Murthy, Tsz-Wo 
> > >>Nicholas Sze,
> > Suresh
> > >>Srinivas and Sanjay Radia. There are many others who contributed 
> > >>as
> well
> > >>providing feedback and comments on numerous jiras.
> > >>
> > >>The vote will run for seven days and will end on March 5, 6:00PM PST.
> > >>
> > >>Regards,
> > >>Suresh
> > >>
> > >>
> > >>
> > >>
> > >>On Thu, Feb 7, 2013 at 6:41 PM, Mahadevan Venkatraman
> > >><ma...@microsoft.com>wrote:
> > >>
> > >>> It is super exciting to look at the prospect of these changes 
> > >>>being merged  to trunk. Having Windows as one of the supported 
> > >>>Hadoop platforms is
> a
> > >>> fantastic opportunity both for the Hadoop project and Microsoft 
> > >>>customers.
> > >>>
> > >>> This work began around a year back when a few of us started with 
> > >>> a
> > basic
> > >>> port of Hadoop on Windows. Ever since, the Hadoop team in 
> > >>> Microsoft
> > have
> > >>> made significant progress in the following areas:
> > >>> (PS: Some of these items are already included in Suresh's email, 
> > >>> but including again for completeness)
> > >>>
> > >>> - Command-line scripts for the Hadoop surface area
> > >>> - Mapping the HDFS permissions model to Windows
> > >>> - Abstracted and reconciled mismatches around differences in 
> > >>> Path semantics in Java and Windows
> > >>> - Native Task Controller for Windows
> > >>> - Implementation of a Block Placement Policy to support cloud 
> > >>> environments, more specifically Azure.
> > >>> - Implementation of Hadoop native libraries for Windows 
> > >>> (compression codecs, native I/O) - Several reliability issues, 
> > >>> including race-conditions, intermittent test failures, resource leaks.
> > >>> - Several new unit test cases written for the above changes
> > >>>
> > >>> In the process, we have closely engaged with the Apache open 
> > >>> source community and have got great support and assistance from 
> > >>> the
> community
> > >>>in
> > >>> terms of contributing fixes, code review comments and commits.
> > >>>
> > >>> In addition, the Hadoop team at Microsoft has also made good 
> > >>> progress
> > in
> > >>> other projects including Hive, Pig, Sqoop, Oozie, HCat and HBase.
> Many
> > >>>of
> > >>> these changes have already been committed to the respective 
> > >>>trunks
> with
> > >>> help from various committers and contributors. It is great to 
> > >>> see the commitment of the community to support multiple 
> > >>> platforms, and we
> look
> > >>> forward to the day when a developer/customer is able to 
> > >>>successfully deploy  a complete solution stack based on Apache 
> > >>>Hadoop releases.
> > >>>
> > >>> Next Steps:
> > >>>
> > >>> All of the above changes are part of the Windows Azure HDInsight 
> > >>>and  HDInsight Server products from Microsoft. We have 
> > >>>successfully on-boarded  several internal customers and have been 
> > >>>running production workloads
> > on
> > >>> Windows Azure HDInsight. Our vision is to create a big data 
> > >>>platform based  on Hadoop, and we are committed to helping make 
> > >>>Hadoop a world-class  solution that anyone can use to solve their 
> > >>>biggest data challenges.
> > >>>
> > >>> As an immediate next step, we would like to have a discussion 
> > >>> around
> > how
> > >>> we can ensure that the quality of the mainline Hadoop branches 
> > >>>on Windows  is maintained. To this end, we would like to get to 
> > >>>the state where
> we
> > >>>have
> > >>> pre-checkin validation gates and nightly test runs enabled on
> Windows.
> > >>>If
> > >>> you have any suggestions around this, please do send an email.  
> > >>>We
> are
> > >>> committed to helping sustain the long-term quality of Hadoop on 
> > >>>both Linux  and Windows.
> > >>>
> > >>> We sincerely thank the community for their contribution and 
> > >>> support
> so
> > >>> far. And hope to continue having a close engagement in the future.
> > >>>
> > >>> -Microsoft HDInsight Team
> > >>>
> > >>>
> > >>> -----Original Message-----
> > >>> From: Suresh Srinivas [mailto:suresh@hortonworks.com]
> > >>> Sent: Thursday, February 7, 2013 5:42 PM
> > >>> To: common-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org; 
> > >>> hdfs-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
> > >>> Subject: Heads up - merge branch-trunk-win to trunk
> > >>>
> > >>> The support for Hadoop on Windows was proposed in HADOOP-8079< 
> > >>> https://issues.apache.org/jira/browse/HADOOP-8079> almost a year
> ago.
> > >>>The
> > >>> goal was to make Hadoop natively integrated, full-featured, and 
> > >>>performance  and scalability tuned on Windows Server or Windows 
> > >>>Azure.
> > >>> We are happy to announce that a lot of progress has been made in 
> > >>>this  regard.
> > >>>
> > >>> Initial work started in a feature branch, branch-1-win, based on 
> > >>>branch-1.
> > >>> The details related to the work done in the branch can be seen 
> > >>>in  CHANGES.txt<
> > >>>
> > >>>
> > http://svn.apache.org/viewvc/hadoop/common/branches/branch-1-win/CHA
> > NGES
> .
> > >>>branch-1-win.txt?view=markup
> > >>> >.
> > >>> This work has been ported to a branch, branch-trunk-win, based 
> > >>> on
> > trunk.
> > >>> Merge patch for this is available on 
> > >>> HADOOP-8562<https://issues.apache.org/jira/browse/HADOOP-8562>
> > >>> .
> > >>>
> > >>> Highlights of the work done so far:
> > >>> 1. Necessary changes in Hadoop to run natively on Windows. These
> > changes
> > >>> handle differences in platforms related to path names, 
> > >>>process/task  management etc.
> > >>> 2. Addition of winutils tools for managing file permissions and 
> > >>>ownership,  user group mapping, hardlinks, symbolic links, chmod, 
> > >>>disk
> utilization,
> > >>>and
> > >>> process/task management.
> > >>> 3. Added cmd scripts equivalent to existing shell scripts  
> > >>>hadoop-daemon.sh, start and stop scripts.
> > >>> 4. Addition of block placement policy implemnation to support 
> > >>>cloud  enviroment, more specifically Azure.
> > >>>
> > >>> We are very close to wrapping up the work in branch-trunk-win 
> > >>>and getting  ready for a merge. Currently the merge patch is 
> > >>>passing close to 100%
> > of
> > >>> unit tests on Linux. Soon I will call for a vote to merge this 
> > >>>branch into  trunk.
> > >>>
> > >>> Next steps:
> > >>> 1. Call for vote to merge branch-trunk-win to trunk, when the 
> > >>> work completes and precommit build is clean.
> > >>> 2. Start a discussion on adding Jenkins precommit builds on 
> > >>> windows
> and
> > >>> how to integrate that with the existing commit process.
> > >>>
> > >>> Let me know if you have any questions.
> > >>>
> > >>> Regards,
> > >>> Suresh
> > >>>
> > >>>
> > >>
> > >>
> > >>--
> > >>http://hortonworks.com/download/
> > >
> >
>
>
>
> --
> http://hortonworks.com/download/
>



--
Todd Lipcon
Software Engineer, Cloudera

Re: [Vote] Merge branch-trunk-win to trunk

[Konstantin Shvachko <sh...@gmail.com>]

> I have attached a patch to the jira:
> https://issues.apache.org/jira/secure/attachment/12571338/branch-trunk-win-min.patch
> The number of lines goes to 1537 lines from the original patch with 15958
> lines.

Suresh, this might be a confusing statement as your patch includes
only Yarn changes.
Could you please update it.

Thanks,
--Konst

On Wed, Feb 27, 2013 at 6:17 PM, Suresh Srinivas <su...@hortonworks.com> wrote:
>>
>> I'm concerned about the above. Personally, I don't have access to any
>> Windows boxes with development tools, and I know nothing about developing
>> on Windows. The only Windows I run is an 8GB VM with 1 GB RAM allocated,
>> for powerpoint :)
>
>
>> If I submit a patch and it gets -1 "tests failed" on the Windows slave, how
>> am I supposed to proceed?
>>
> I think a reasonable compromise would be that the tests should always
>> *build* on Windows before commit, and contributors should do their best to
>> look at the test logs for any Windows-specific failures. But, beyond
>> looking at the logs, a "-1 Tests failed on windows" should not block a
>> commit.
>>
>
> I think these concerns may be exaggerated.
>
> I have attached a patch to the jira:
> https://issues.apache.org/jira/secure/attachment/12571338/branch-trunk-win-min.patch
> The number of lines goes to 1537 lines from the original patch with 15958
> lines.
>
> So the choice of Java in Hadoop meant that the work required was minimal.
> Most
> of the changes are related platform specific behavior such File separator
> and line
> separator which are fairly trivial to debug and fix.

Re: [Vote] Merge branch-trunk-win to trunk

[Suresh Srinivas <su...@hortonworks.com>]

>
> I'm concerned about the above. Personally, I don't have access to any
> Windows boxes with development tools, and I know nothing about developing
> on Windows. The only Windows I run is an 8GB VM with 1 GB RAM allocated,
> for powerpoint :)


> If I submit a patch and it gets -1 "tests failed" on the Windows slave, how
> am I supposed to proceed?
>
I think a reasonable compromise would be that the tests should always
> *build* on Windows before commit, and contributors should do their best to
> look at the test logs for any Windows-specific failures. But, beyond
> looking at the logs, a "-1 Tests failed on windows" should not block a
> commit.
>

I think these concerns may be exaggerated.

I have attached a patch to the jira:
https://issues.apache.org/jira/secure/attachment/12571338/branch-trunk-win-min.patch
The number of lines goes to 1537 lines from the original patch with 15958
lines.

So the choice of Java in Hadoop meant that the work required was minimal.
Most
of the changes are related platform specific behavior such File separator
and line
separator which are fairly trivial to debug and fix.

RE: [Vote] Merge branch-trunk-win to trunk

[Ivan Mitic <iv...@microsoft.com>]

+1 (non-binding)

I am really glad to see this happening! As people already mentioned, this has been a great engineering effort involving many people!


Folks raised some valid concerns below and I thought it would be good to share my 2 cents. In my opinion, we don't have to solve all these problems right now. As we move forward with two platforms, we can start addressing one problem at a time and incrementally improve. In the first iteration, maintaining Hadoop on Windows could be just everyone trying to do their best effort (make sure Jenkins build succeeds at least). We already have people who are building/running trunk on Windows daily, so they would jump in and fix problems as needed (we've been doing this in branch-trunk-win for a while now). Although I see that the problems could arise with platform specific features/optimizations, I don't think these are frequent, so in most cases everything will just work. Merging the two branches sooner rather than later does seems like the right thing to do if the ultimate goal is to have Hadoop on both platforms. Now that the port has completed, we will have people in Microsoft (and elsewhere) wanting to contribute features/improvements to the trunk branch. A separate branch would just make things more difficult and confusing for everyone :) Hope this makes sense.

-----Original Message-----
From: Todd Lipcon [mailto:todd@cloudera.com] 
Sent: Wednesday, February 27, 2013 3:43 PM
To: common-dev@hadoop.apache.org
Cc: yarn-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
Subject: Re: [Vote] Merge branch-trunk-win to trunk

On Wed, Feb 27, 2013 at 2:54 PM, Suresh Srinivas <su...@hortonworks.com>wrote:

> With that we need to decide how our precommit process looks.
> My inclination is to wait for +1 from precommit builds on both the 
> platforms to ensure no issues are introduced.
> Thoughts?
>
> 2. Feature development impact
> Some questions have been raised about would new features need to be 
> supported on both the platforms. Yes. I do not see a reason why 
> features cannot work on both the platforms, with the exception of 
> platform specific optimizations. This what Java gives us.
>
>
I'm concerned about the above. Personally, I don't have access to any Windows boxes with development tools, and I know nothing about developing on Windows. The only Windows I run is an 8GB VM with 1 GB RAM allocated, for powerpoint :)

If I submit a patch and it gets -1 "tests failed" on the Windows slave, how am I supposed to proceed?

I think a reasonable compromise would be that the tests should always
*build* on Windows before commit, and contributors should do their best to look at the test logs for any Windows-specific failures. But, beyond looking at the logs, a "-1 Tests failed on windows" should not block a commit.

Those contributors who are interested in Windows being a first-class platform should be responsible for watching the Windows builds and debugging/fixing any regressions that might be Windows-specific.

I also think the KDE model that Harsh pointed out is an interesting one -- ie the idea that we would not merge windows support to trunk, but rather treat is as a "parallel code line" which lives in the ASF and has its own builds and releases. The windows team would periodically merge trunk->win to pick up any new changes, and do a separate test/release process. I'm not convinced this is the best idea, but worth discussion of pros and cons.

-Todd


>
> On Wed, Feb 27, 2013 at 11:56 AM, Eli Collins <el...@cloudera.com> wrote:
>
> > Bobby raises some good questions.  A related one, since most current 
> > developers won't add Windows support for new features that are 
> > platform specific is it assumed that Windows development will either 
> > lag or will people actively work on keeping Windows up with the 
> > latest?  And vice versa in case Windows support is implemented first.
> >
> > Is there a jira for resolving the outstanding TODOs in the code base 
> > (similar to HDFS-2148)?  Looks like this merge doesn't introduce 
> > many which is great (just did a quick diff and grep).
> >
> > Thanks,
> > Eli
> >
> > On Wed, Feb 27, 2013 at 8:17 AM, Robert Evans <ev...@yahoo-inc.com>
> wrote:
> > > After this is merged in is Windows still going to be a second 
> > > class citizen but happens to work for more than just development 
> > > or is it a fully supported platform where if something breaks it 
> > > can block a
> > release?
> > >  How do we as a community intend to keep Windows support from breaking?
> > > We don't have any Jenkins slaves to be able to run nightly tests 
> > > to validate everything still compiles/runs.  This is not a blocker 
> > > for me because we often rely on individuals and groups to test 
> > > Hadoop, but I
> do
> > > think we need to have this discussion before we put it in.
> > >
> > > --Bobby
> > >
> > > On 2/26/13 4:55 PM, "Suresh Srinivas" <su...@hortonworks.com> wrote:
> > >
> > >>I had posted heads up about merging branch-trunk-win to trunk on 
> > >>Feb
> 8th.
> > >>I
> > >>am happy to announce that we are ready for the merge.
> > >>
> > >>Here is a brief recap on the highlights of the work done:
> > >>- Command-line scripts for the Hadoop surface area
> > >>- Mapping the HDFS permissions model to Windows
> > >>- Abstracted and reconciled mismatches around differences in Path 
> > >>semantics in Java and Windows
> > >>- Native Task Controller for Windows
> > >>- Implementation of a Block Placement Policy to support cloud 
> > >>environments, more specifically Azure.
> > >>- Implementation of Hadoop native libraries for Windows 
> > >>(compression codecs, native I/O)
> > >>- Several reliability issues, including race-conditions, 
> > >>intermittent
> > test
> > >>failures, resource leaks.
> > >>- Several new unit test cases written for the above changes
> > >>
> > >>Please find the details of the work in 
> > >>CHANGES.branch-trunk-win.txt - Common 
> > >>changes<http://bit.ly/Xe7Ynv>, HDFS changes<
> http://bit.ly/13QOSo9
> > >,
> > >>and YARN and MapReduce changes <http://bit.ly/128zzMt>. This is 
> > >>the
> work
> > >>ported from branch-1-win to a branch based on trunk.
> > >>
> > >>For details of the testing done, please see the thread - 
> > >>http://bit.ly/WpavJ4. Merge patch for this is available on
> HADOOP-8562<
> > >>https://issues.apache.org/jira/browse/HADOOP-8562>.
> > >>
> > >>This was a large undertaking that involved developing code, 
> > >>testing the entire Hadoop stack, including scale tests. This is 
> > >>made possible only with the contribution from many many folks in 
> > >>the community. Following
> people
> > >>contributed to this work: Ivan Mitic, Chuan Liu, Ramya Sunil, 
> > >>Bikas
> Saha,
> > >>Kanna Karanam, John Gordon, Brandon Li, Chris Nauroth, David Lao,
> > Sumadhur
> > >>Reddy Bolli, Arpit Agarwal, Ahmed El Baz, Mike Liddell, Jing Zhao,
> Thejas
> > >>Nair, Steve Maine, Ganeshan Iyer, Raja Aluri, Giridharan Kesavan, 
> > >>Ramya Bharathi Nimmagadda, Daryn Sharp, Arun Murthy, Tsz-Wo 
> > >>Nicholas Sze,
> > Suresh
> > >>Srinivas and Sanjay Radia. There are many others who contributed 
> > >>as
> well
> > >>providing feedback and comments on numerous jiras.
> > >>
> > >>The vote will run for seven days and will end on March 5, 6:00PM PST.
> > >>
> > >>Regards,
> > >>Suresh
> > >>
> > >>
> > >>
> > >>
> > >>On Thu, Feb 7, 2013 at 6:41 PM, Mahadevan Venkatraman
> > >><ma...@microsoft.com>wrote:
> > >>
> > >>> It is super exciting to look at the prospect of these changes 
> > >>>being merged  to trunk. Having Windows as one of the supported 
> > >>>Hadoop platforms is
> a
> > >>> fantastic opportunity both for the Hadoop project and Microsoft 
> > >>>customers.
> > >>>
> > >>> This work began around a year back when a few of us started with 
> > >>> a
> > basic
> > >>> port of Hadoop on Windows. Ever since, the Hadoop team in 
> > >>> Microsoft
> > have
> > >>> made significant progress in the following areas:
> > >>> (PS: Some of these items are already included in Suresh's email, 
> > >>> but including again for completeness)
> > >>>
> > >>> - Command-line scripts for the Hadoop surface area
> > >>> - Mapping the HDFS permissions model to Windows
> > >>> - Abstracted and reconciled mismatches around differences in 
> > >>> Path semantics in Java and Windows
> > >>> - Native Task Controller for Windows
> > >>> - Implementation of a Block Placement Policy to support cloud 
> > >>> environments, more specifically Azure.
> > >>> - Implementation of Hadoop native libraries for Windows 
> > >>> (compression codecs, native I/O) - Several reliability issues, 
> > >>> including race-conditions, intermittent test failures, resource leaks.
> > >>> - Several new unit test cases written for the above changes
> > >>>
> > >>> In the process, we have closely engaged with the Apache open 
> > >>> source community and have got great support and assistance from 
> > >>> the
> community
> > >>>in
> > >>> terms of contributing fixes, code review comments and commits.
> > >>>
> > >>> In addition, the Hadoop team at Microsoft has also made good 
> > >>> progress
> > in
> > >>> other projects including Hive, Pig, Sqoop, Oozie, HCat and HBase.
> Many
> > >>>of
> > >>> these changes have already been committed to the respective 
> > >>>trunks
> with
> > >>> help from various committers and contributors. It is great to 
> > >>> see the commitment of the community to support multiple 
> > >>> platforms, and we
> look
> > >>> forward to the day when a developer/customer is able to 
> > >>>successfully deploy  a complete solution stack based on Apache 
> > >>>Hadoop releases.
> > >>>
> > >>> Next Steps:
> > >>>
> > >>> All of the above changes are part of the Windows Azure HDInsight 
> > >>>and  HDInsight Server products from Microsoft. We have 
> > >>>successfully on-boarded  several internal customers and have been 
> > >>>running production workloads
> > on
> > >>> Windows Azure HDInsight. Our vision is to create a big data 
> > >>>platform based  on Hadoop, and we are committed to helping make 
> > >>>Hadoop a world-class  solution that anyone can use to solve their 
> > >>>biggest data challenges.
> > >>>
> > >>> As an immediate next step, we would like to have a discussion 
> > >>> around
> > how
> > >>> we can ensure that the quality of the mainline Hadoop branches 
> > >>>on Windows  is maintained. To this end, we would like to get to 
> > >>>the state where
> we
> > >>>have
> > >>> pre-checkin validation gates and nightly test runs enabled on
> Windows.
> > >>>If
> > >>> you have any suggestions around this, please do send an email.  
> > >>>We
> are
> > >>> committed to helping sustain the long-term quality of Hadoop on 
> > >>>both Linux  and Windows.
> > >>>
> > >>> We sincerely thank the community for their contribution and 
> > >>> support
> so
> > >>> far. And hope to continue having a close engagement in the future.
> > >>>
> > >>> -Microsoft HDInsight Team
> > >>>
> > >>>
> > >>> -----Original Message-----
> > >>> From: Suresh Srinivas [mailto:suresh@hortonworks.com]
> > >>> Sent: Thursday, February 7, 2013 5:42 PM
> > >>> To: common-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org; 
> > >>> hdfs-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
> > >>> Subject: Heads up - merge branch-trunk-win to trunk
> > >>>
> > >>> The support for Hadoop on Windows was proposed in HADOOP-8079< 
> > >>> https://issues.apache.org/jira/browse/HADOOP-8079> almost a year
> ago.
> > >>>The
> > >>> goal was to make Hadoop natively integrated, full-featured, and 
> > >>>performance  and scalability tuned on Windows Server or Windows 
> > >>>Azure.
> > >>> We are happy to announce that a lot of progress has been made in 
> > >>>this  regard.
> > >>>
> > >>> Initial work started in a feature branch, branch-1-win, based on 
> > >>>branch-1.
> > >>> The details related to the work done in the branch can be seen 
> > >>>in  CHANGES.txt<
> > >>>
> > >>>
> > http://svn.apache.org/viewvc/hadoop/common/branches/branch-1-win/CHA
> > NGES
> .
> > >>>branch-1-win.txt?view=markup
> > >>> >.
> > >>> This work has been ported to a branch, branch-trunk-win, based 
> > >>> on
> > trunk.
> > >>> Merge patch for this is available on 
> > >>> HADOOP-8562<https://issues.apache.org/jira/browse/HADOOP-8562>
> > >>> .
> > >>>
> > >>> Highlights of the work done so far:
> > >>> 1. Necessary changes in Hadoop to run natively on Windows. These
> > changes
> > >>> handle differences in platforms related to path names, 
> > >>>process/task  management etc.
> > >>> 2. Addition of winutils tools for managing file permissions and 
> > >>>ownership,  user group mapping, hardlinks, symbolic links, chmod, 
> > >>>disk
> utilization,
> > >>>and
> > >>> process/task management.
> > >>> 3. Added cmd scripts equivalent to existing shell scripts  
> > >>>hadoop-daemon.sh, start and stop scripts.
> > >>> 4. Addition of block placement policy implemnation to support 
> > >>>cloud  enviroment, more specifically Azure.
> > >>>
> > >>> We are very close to wrapping up the work in branch-trunk-win 
> > >>>and getting  ready for a merge. Currently the merge patch is 
> > >>>passing close to 100%
> > of
> > >>> unit tests on Linux. Soon I will call for a vote to merge this 
> > >>>branch into  trunk.
> > >>>
> > >>> Next steps:
> > >>> 1. Call for vote to merge branch-trunk-win to trunk, when the 
> > >>> work completes and precommit build is clean.
> > >>> 2. Start a discussion on adding Jenkins precommit builds on 
> > >>> windows
> and
> > >>> how to integrate that with the existing commit process.
> > >>>
> > >>> Let me know if you have any questions.
> > >>>
> > >>> Regards,
> > >>> Suresh
> > >>>
> > >>>
> > >>
> > >>
> > >>--
> > >>http://hortonworks.com/download/
> > >
> >
>
>
>
> --
> http://hortonworks.com/download/
>



--
Todd Lipcon
Software Engineer, Cloudera

Re: [Vote] Merge branch-trunk-win to trunk

[Todd Lipcon <to...@cloudera.com>]

On Wed, Feb 27, 2013 at 2:54 PM, Suresh Srinivas <su...@hortonworks.com>wrote:

> With that we need to decide how our precommit process looks.
> My inclination is to wait for +1 from precommit builds on
> both the platforms to ensure no issues are introduced.
> Thoughts?
>
> 2. Feature development impact
> Some questions have been raised about would new features
> need to be supported on both the platforms. Yes. I do not see a
> reason why features cannot work on both the platforms, with
> the exception of platform specific optimizations. This what Java
> gives us.
>
>
I'm concerned about the above. Personally, I don't have access to any
Windows boxes with development tools, and I know nothing about developing
on Windows. The only Windows I run is an 8GB VM with 1 GB RAM allocated,
for powerpoint :)

If I submit a patch and it gets -1 "tests failed" on the Windows slave, how
am I supposed to proceed?

I think a reasonable compromise would be that the tests should always
*build* on Windows before commit, and contributors should do their best to
look at the test logs for any Windows-specific failures. But, beyond
looking at the logs, a "-1 Tests failed on windows" should not block a
commit.

Those contributors who are interested in Windows being a first-class
platform should be responsible for watching the Windows builds and
debugging/fixing any regressions that might be Windows-specific.

I also think the KDE model that Harsh pointed out is an interesting one --
ie the idea that we would not merge windows support to trunk, but rather
treat is as a "parallel code line" which lives in the ASF and has its own
builds and releases. The windows team would periodically merge trunk->win
to pick up any new changes, and do a separate test/release process. I'm not
convinced this is the best idea, but worth discussion of pros and cons.

-Todd


>
> On Wed, Feb 27, 2013 at 11:56 AM, Eli Collins <el...@cloudera.com> wrote:
>
> > Bobby raises some good questions.  A related one, since most current
> > developers won't add Windows support for new features that are
> > platform specific is it assumed that Windows development will either
> > lag or will people actively work on keeping Windows up with the
> > latest?  And vice versa in case Windows support is implemented first.
> >
> > Is there a jira for resolving the outstanding TODOs in the code base
> > (similar to HDFS-2148)?  Looks like this merge doesn't introduce many
> > which is great (just did a quick diff and grep).
> >
> > Thanks,
> > Eli
> >
> > On Wed, Feb 27, 2013 at 8:17 AM, Robert Evans <ev...@yahoo-inc.com>
> wrote:
> > > After this is merged in is Windows still going to be a second class
> > > citizen but happens to work for more than just development or is it a
> > > fully supported platform where if something breaks it can block a
> > release?
> > >  How do we as a community intend to keep Windows support from breaking?
> > > We don't have any Jenkins slaves to be able to run nightly tests to
> > > validate everything still compiles/runs.  This is not a blocker for me
> > > because we often rely on individuals and groups to test Hadoop, but I
> do
> > > think we need to have this discussion before we put it in.
> > >
> > > --Bobby
> > >
> > > On 2/26/13 4:55 PM, "Suresh Srinivas" <su...@hortonworks.com> wrote:
> > >
> > >>I had posted heads up about merging branch-trunk-win to trunk on Feb
> 8th.
> > >>I
> > >>am happy to announce that we are ready for the merge.
> > >>
> > >>Here is a brief recap on the highlights of the work done:
> > >>- Command-line scripts for the Hadoop surface area
> > >>- Mapping the HDFS permissions model to Windows
> > >>- Abstracted and reconciled mismatches around differences in Path
> > >>semantics
> > >>in Java and Windows
> > >>- Native Task Controller for Windows
> > >>- Implementation of a Block Placement Policy to support cloud
> > >>environments,
> > >>more specifically Azure.
> > >>- Implementation of Hadoop native libraries for Windows (compression
> > >>codecs, native I/O)
> > >>- Several reliability issues, including race-conditions, intermittent
> > test
> > >>failures, resource leaks.
> > >>- Several new unit test cases written for the above changes
> > >>
> > >>Please find the details of the work in CHANGES.branch-trunk-win.txt -
> > >>Common changes<http://bit.ly/Xe7Ynv>, HDFS changes<
> http://bit.ly/13QOSo9
> > >,
> > >>and YARN and MapReduce changes <http://bit.ly/128zzMt>. This is the
> work
> > >>ported from branch-1-win to a branch based on trunk.
> > >>
> > >>For details of the testing done, please see the thread -
> > >>http://bit.ly/WpavJ4. Merge patch for this is available on
> HADOOP-8562<
> > >>https://issues.apache.org/jira/browse/HADOOP-8562>.
> > >>
> > >>This was a large undertaking that involved developing code, testing the
> > >>entire Hadoop stack, including scale tests. This is made possible only
> > >>with
> > >>the contribution from many many folks in the community. Following
> people
> > >>contributed to this work: Ivan Mitic, Chuan Liu, Ramya Sunil, Bikas
> Saha,
> > >>Kanna Karanam, John Gordon, Brandon Li, Chris Nauroth, David Lao,
> > Sumadhur
> > >>Reddy Bolli, Arpit Agarwal, Ahmed El Baz, Mike Liddell, Jing Zhao,
> Thejas
> > >>Nair, Steve Maine, Ganeshan Iyer, Raja Aluri, Giridharan Kesavan, Ramya
> > >>Bharathi Nimmagadda, Daryn Sharp, Arun Murthy, Tsz-Wo Nicholas Sze,
> > Suresh
> > >>Srinivas and Sanjay Radia. There are many others who contributed as
> well
> > >>providing feedback and comments on numerous jiras.
> > >>
> > >>The vote will run for seven days and will end on March 5, 6:00PM PST.
> > >>
> > >>Regards,
> > >>Suresh
> > >>
> > >>
> > >>
> > >>
> > >>On Thu, Feb 7, 2013 at 6:41 PM, Mahadevan Venkatraman
> > >><ma...@microsoft.com>wrote:
> > >>
> > >>> It is super exciting to look at the prospect of these changes being
> > >>>merged
> > >>> to trunk. Having Windows as one of the supported Hadoop platforms is
> a
> > >>> fantastic opportunity both for the Hadoop project and Microsoft
> > >>>customers.
> > >>>
> > >>> This work began around a year back when a few of us started with a
> > basic
> > >>> port of Hadoop on Windows. Ever since, the Hadoop team in Microsoft
> > have
> > >>> made significant progress in the following areas:
> > >>> (PS: Some of these items are already included in Suresh's email, but
> > >>> including again for completeness)
> > >>>
> > >>> - Command-line scripts for the Hadoop surface area
> > >>> - Mapping the HDFS permissions model to Windows
> > >>> - Abstracted and reconciled mismatches around differences in Path
> > >>> semantics in Java and Windows
> > >>> - Native Task Controller for Windows
> > >>> - Implementation of a Block Placement Policy to support cloud
> > >>> environments, more specifically Azure.
> > >>> - Implementation of Hadoop native libraries for Windows (compression
> > >>> codecs, native I/O) - Several reliability issues, including
> > >>> race-conditions, intermittent test failures, resource leaks.
> > >>> - Several new unit test cases written for the above changes
> > >>>
> > >>> In the process, we have closely engaged with the Apache open source
> > >>> community and have got great support and assistance from the
> community
> > >>>in
> > >>> terms of contributing fixes, code review comments and commits.
> > >>>
> > >>> In addition, the Hadoop team at Microsoft has also made good progress
> > in
> > >>> other projects including Hive, Pig, Sqoop, Oozie, HCat and HBase.
> Many
> > >>>of
> > >>> these changes have already been committed to the respective trunks
> with
> > >>> help from various committers and contributors. It is great to see the
> > >>> commitment of the community to support multiple platforms, and we
> look
> > >>> forward to the day when a developer/customer is able to successfully
> > >>>deploy
> > >>> a complete solution stack based on Apache Hadoop releases.
> > >>>
> > >>> Next Steps:
> > >>>
> > >>> All of the above changes are part of the Windows Azure HDInsight and
> > >>> HDInsight Server products from Microsoft. We have successfully
> > >>>on-boarded
> > >>> several internal customers and have been running production workloads
> > on
> > >>> Windows Azure HDInsight. Our vision is to create a big data platform
> > >>>based
> > >>> on Hadoop, and we are committed to helping make Hadoop a world-class
> > >>> solution that anyone can use to solve their biggest data challenges.
> > >>>
> > >>> As an immediate next step, we would like to have a discussion around
> > how
> > >>> we can ensure that the quality of the mainline Hadoop branches on
> > >>>Windows
> > >>> is maintained. To this end, we would like to get to the state where
> we
> > >>>have
> > >>> pre-checkin validation gates and nightly test runs enabled on
> Windows.
> > >>>If
> > >>> you have any suggestions around this, please do send an email.  We
> are
> > >>> committed to helping sustain the long-term quality of Hadoop on both
> > >>>Linux
> > >>> and Windows.
> > >>>
> > >>> We sincerely thank the community for their contribution and support
> so
> > >>> far. And hope to continue having a close engagement in the future.
> > >>>
> > >>> -Microsoft HDInsight Team
> > >>>
> > >>>
> > >>> -----Original Message-----
> > >>> From: Suresh Srinivas [mailto:suresh@hortonworks.com]
> > >>> Sent: Thursday, February 7, 2013 5:42 PM
> > >>> To: common-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org;
> > >>> hdfs-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
> > >>> Subject: Heads up - merge branch-trunk-win to trunk
> > >>>
> > >>> The support for Hadoop on Windows was proposed in HADOOP-8079<
> > >>> https://issues.apache.org/jira/browse/HADOOP-8079> almost a year
> ago.
> > >>>The
> > >>> goal was to make Hadoop natively integrated, full-featured, and
> > >>>performance
> > >>> and scalability tuned on Windows Server or Windows Azure.
> > >>> We are happy to announce that a lot of progress has been made in this
> > >>> regard.
> > >>>
> > >>> Initial work started in a feature branch, branch-1-win, based on
> > >>>branch-1.
> > >>> The details related to the work done in the branch can be seen in
> > >>> CHANGES.txt<
> > >>>
> > >>>
> > http://svn.apache.org/viewvc/hadoop/common/branches/branch-1-win/CHANGES
> .
> > >>>branch-1-win.txt?view=markup
> > >>> >.
> > >>> This work has been ported to a branch, branch-trunk-win, based on
> > trunk.
> > >>> Merge patch for this is available on
> > >>> HADOOP-8562<https://issues.apache.org/jira/browse/HADOOP-8562>
> > >>> .
> > >>>
> > >>> Highlights of the work done so far:
> > >>> 1. Necessary changes in Hadoop to run natively on Windows. These
> > changes
> > >>> handle differences in platforms related to path names, process/task
> > >>> management etc.
> > >>> 2. Addition of winutils tools for managing file permissions and
> > >>>ownership,
> > >>> user group mapping, hardlinks, symbolic links, chmod, disk
> utilization,
> > >>>and
> > >>> process/task management.
> > >>> 3. Added cmd scripts equivalent to existing shell scripts
> > >>> hadoop-daemon.sh, start and stop scripts.
> > >>> 4. Addition of block placement policy implemnation to support cloud
> > >>> enviroment, more specifically Azure.
> > >>>
> > >>> We are very close to wrapping up the work in branch-trunk-win and
> > >>>getting
> > >>> ready for a merge. Currently the merge patch is passing close to 100%
> > of
> > >>> unit tests on Linux. Soon I will call for a vote to merge this branch
> > >>>into
> > >>> trunk.
> > >>>
> > >>> Next steps:
> > >>> 1. Call for vote to merge branch-trunk-win to trunk, when the work
> > >>> completes and precommit build is clean.
> > >>> 2. Start a discussion on adding Jenkins precommit builds on windows
> and
> > >>> how to integrate that with the existing commit process.
> > >>>
> > >>> Let me know if you have any questions.
> > >>>
> > >>> Regards,
> > >>> Suresh
> > >>>
> > >>>
> > >>
> > >>
> > >>--
> > >>http://hortonworks.com/download/
> > >
> >
>
>
>
> --
> http://hortonworks.com/download/
>



-- 
Todd Lipcon
Software Engineer, Cloudera

Re: [Vote] Merge branch-trunk-win to trunk

[Todd Lipcon <to...@cloudera.com>]

On Wed, Feb 27, 2013 at 2:54 PM, Suresh Srinivas <su...@hortonworks.com>wrote:

> With that we need to decide how our precommit process looks.
> My inclination is to wait for +1 from precommit builds on
> both the platforms to ensure no issues are introduced.
> Thoughts?
>
> 2. Feature development impact
> Some questions have been raised about would new features
> need to be supported on both the platforms. Yes. I do not see a
> reason why features cannot work on both the platforms, with
> the exception of platform specific optimizations. This what Java
> gives us.
>
>
I'm concerned about the above. Personally, I don't have access to any
Windows boxes with development tools, and I know nothing about developing
on Windows. The only Windows I run is an 8GB VM with 1 GB RAM allocated,
for powerpoint :)

If I submit a patch and it gets -1 "tests failed" on the Windows slave, how
am I supposed to proceed?

I think a reasonable compromise would be that the tests should always
*build* on Windows before commit, and contributors should do their best to
look at the test logs for any Windows-specific failures. But, beyond
looking at the logs, a "-1 Tests failed on windows" should not block a
commit.

Those contributors who are interested in Windows being a first-class
platform should be responsible for watching the Windows builds and
debugging/fixing any regressions that might be Windows-specific.

I also think the KDE model that Harsh pointed out is an interesting one --
ie the idea that we would not merge windows support to trunk, but rather
treat is as a "parallel code line" which lives in the ASF and has its own
builds and releases. The windows team would periodically merge trunk->win
to pick up any new changes, and do a separate test/release process. I'm not
convinced this is the best idea, but worth discussion of pros and cons.

-Todd


>
> On Wed, Feb 27, 2013 at 11:56 AM, Eli Collins <el...@cloudera.com> wrote:
>
> > Bobby raises some good questions.  A related one, since most current
> > developers won't add Windows support for new features that are
> > platform specific is it assumed that Windows development will either
> > lag or will people actively work on keeping Windows up with the
> > latest?  And vice versa in case Windows support is implemented first.
> >
> > Is there a jira for resolving the outstanding TODOs in the code base
> > (similar to HDFS-2148)?  Looks like this merge doesn't introduce many
> > which is great (just did a quick diff and grep).
> >
> > Thanks,
> > Eli
> >
> > On Wed, Feb 27, 2013 at 8:17 AM, Robert Evans <ev...@yahoo-inc.com>
> wrote:
> > > After this is merged in is Windows still going to be a second class
> > > citizen but happens to work for more than just development or is it a
> > > fully supported platform where if something breaks it can block a
> > release?
> > >  How do we as a community intend to keep Windows support from breaking?
> > > We don't have any Jenkins slaves to be able to run nightly tests to
> > > validate everything still compiles/runs.  This is not a blocker for me
> > > because we often rely on individuals and groups to test Hadoop, but I
> do
> > > think we need to have this discussion before we put it in.
> > >
> > > --Bobby
> > >
> > > On 2/26/13 4:55 PM, "Suresh Srinivas" <su...@hortonworks.com> wrote:
> > >
> > >>I had posted heads up about merging branch-trunk-win to trunk on Feb
> 8th.
> > >>I
> > >>am happy to announce that we are ready for the merge.
> > >>
> > >>Here is a brief recap on the highlights of the work done:
> > >>- Command-line scripts for the Hadoop surface area
> > >>- Mapping the HDFS permissions model to Windows
> > >>- Abstracted and reconciled mismatches around differences in Path
> > >>semantics
> > >>in Java and Windows
> > >>- Native Task Controller for Windows
> > >>- Implementation of a Block Placement Policy to support cloud
> > >>environments,
> > >>more specifically Azure.
> > >>- Implementation of Hadoop native libraries for Windows (compression
> > >>codecs, native I/O)
> > >>- Several reliability issues, including race-conditions, intermittent
> > test
> > >>failures, resource leaks.
> > >>- Several new unit test cases written for the above changes
> > >>
> > >>Please find the details of the work in CHANGES.branch-trunk-win.txt -
> > >>Common changes<http://bit.ly/Xe7Ynv>, HDFS changes<
> http://bit.ly/13QOSo9
> > >,
> > >>and YARN and MapReduce changes <http://bit.ly/128zzMt>. This is the
> work
> > >>ported from branch-1-win to a branch based on trunk.
> > >>
> > >>For details of the testing done, please see the thread -
> > >>http://bit.ly/WpavJ4. Merge patch for this is available on
> HADOOP-8562<
> > >>https://issues.apache.org/jira/browse/HADOOP-8562>.
> > >>
> > >>This was a large undertaking that involved developing code, testing the
> > >>entire Hadoop stack, including scale tests. This is made possible only
> > >>with
> > >>the contribution from many many folks in the community. Following
> people
> > >>contributed to this work: Ivan Mitic, Chuan Liu, Ramya Sunil, Bikas
> Saha,
> > >>Kanna Karanam, John Gordon, Brandon Li, Chris Nauroth, David Lao,
> > Sumadhur
> > >>Reddy Bolli, Arpit Agarwal, Ahmed El Baz, Mike Liddell, Jing Zhao,
> Thejas
> > >>Nair, Steve Maine, Ganeshan Iyer, Raja Aluri, Giridharan Kesavan, Ramya
> > >>Bharathi Nimmagadda, Daryn Sharp, Arun Murthy, Tsz-Wo Nicholas Sze,
> > Suresh
> > >>Srinivas and Sanjay Radia. There are many others who contributed as
> well
> > >>providing feedback and comments on numerous jiras.
> > >>
> > >>The vote will run for seven days and will end on March 5, 6:00PM PST.
> > >>
> > >>Regards,
> > >>Suresh
> > >>
> > >>
> > >>
> > >>
> > >>On Thu, Feb 7, 2013 at 6:41 PM, Mahadevan Venkatraman
> > >><ma...@microsoft.com>wrote:
> > >>
> > >>> It is super exciting to look at the prospect of these changes being
> > >>>merged
> > >>> to trunk. Having Windows as one of the supported Hadoop platforms is
> a
> > >>> fantastic opportunity both for the Hadoop project and Microsoft
> > >>>customers.
> > >>>
> > >>> This work began around a year back when a few of us started with a
> > basic
> > >>> port of Hadoop on Windows. Ever since, the Hadoop team in Microsoft
> > have
> > >>> made significant progress in the following areas:
> > >>> (PS: Some of these items are already included in Suresh's email, but
> > >>> including again for completeness)
> > >>>
> > >>> - Command-line scripts for the Hadoop surface area
> > >>> - Mapping the HDFS permissions model to Windows
> > >>> - Abstracted and reconciled mismatches around differences in Path
> > >>> semantics in Java and Windows
> > >>> - Native Task Controller for Windows
> > >>> - Implementation of a Block Placement Policy to support cloud
> > >>> environments, more specifically Azure.
> > >>> - Implementation of Hadoop native libraries for Windows (compression
> > >>> codecs, native I/O) - Several reliability issues, including
> > >>> race-conditions, intermittent test failures, resource leaks.
> > >>> - Several new unit test cases written for the above changes
> > >>>
> > >>> In the process, we have closely engaged with the Apache open source
> > >>> community and have got great support and assistance from the
> community
> > >>>in
> > >>> terms of contributing fixes, code review comments and commits.
> > >>>
> > >>> In addition, the Hadoop team at Microsoft has also made good progress
> > in
> > >>> other projects including Hive, Pig, Sqoop, Oozie, HCat and HBase.
> Many
> > >>>of
> > >>> these changes have already been committed to the respective trunks
> with
> > >>> help from various committers and contributors. It is great to see the
> > >>> commitment of the community to support multiple platforms, and we
> look
> > >>> forward to the day when a developer/customer is able to successfully
> > >>>deploy
> > >>> a complete solution stack based on Apache Hadoop releases.
> > >>>
> > >>> Next Steps:
> > >>>
> > >>> All of the above changes are part of the Windows Azure HDInsight and
> > >>> HDInsight Server products from Microsoft. We have successfully
> > >>>on-boarded
> > >>> several internal customers and have been running production workloads
> > on
> > >>> Windows Azure HDInsight. Our vision is to create a big data platform
> > >>>based
> > >>> on Hadoop, and we are committed to helping make Hadoop a world-class
> > >>> solution that anyone can use to solve their biggest data challenges.
> > >>>
> > >>> As an immediate next step, we would like to have a discussion around
> > how
> > >>> we can ensure that the quality of the mainline Hadoop branches on
> > >>>Windows
> > >>> is maintained. To this end, we would like to get to the state where
> we
> > >>>have
> > >>> pre-checkin validation gates and nightly test runs enabled on
> Windows.
> > >>>If
> > >>> you have any suggestions around this, please do send an email.  We
> are
> > >>> committed to helping sustain the long-term quality of Hadoop on both
> > >>>Linux
> > >>> and Windows.
> > >>>
> > >>> We sincerely thank the community for their contribution and support
> so
> > >>> far. And hope to continue having a close engagement in the future.
> > >>>
> > >>> -Microsoft HDInsight Team
> > >>>
> > >>>
> > >>> -----Original Message-----
> > >>> From: Suresh Srinivas [mailto:suresh@hortonworks.com]
> > >>> Sent: Thursday, February 7, 2013 5:42 PM
> > >>> To: common-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org;
> > >>> hdfs-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
> > >>> Subject: Heads up - merge branch-trunk-win to trunk
> > >>>
> > >>> The support for Hadoop on Windows was proposed in HADOOP-8079<
> > >>> https://issues.apache.org/jira/browse/HADOOP-8079> almost a year
> ago.
> > >>>The
> > >>> goal was to make Hadoop natively integrated, full-featured, and
> > >>>performance
> > >>> and scalability tuned on Windows Server or Windows Azure.
> > >>> We are happy to announce that a lot of progress has been made in this
> > >>> regard.
> > >>>
> > >>> Initial work started in a feature branch, branch-1-win, based on
> > >>>branch-1.
> > >>> The details related to the work done in the branch can be seen in
> > >>> CHANGES.txt<
> > >>>
> > >>>
> > http://svn.apache.org/viewvc/hadoop/common/branches/branch-1-win/CHANGES
> .
> > >>>branch-1-win.txt?view=markup
> > >>> >.
> > >>> This work has been ported to a branch, branch-trunk-win, based on
> > trunk.
> > >>> Merge patch for this is available on
> > >>> HADOOP-8562<https://issues.apache.org/jira/browse/HADOOP-8562>
> > >>> .
> > >>>
> > >>> Highlights of the work done so far:
> > >>> 1. Necessary changes in Hadoop to run natively on Windows. These
> > changes
> > >>> handle differences in platforms related to path names, process/task
> > >>> management etc.
> > >>> 2. Addition of winutils tools for managing file permissions and
> > >>>ownership,
> > >>> user group mapping, hardlinks, symbolic links, chmod, disk
> utilization,
> > >>>and
> > >>> process/task management.
> > >>> 3. Added cmd scripts equivalent to existing shell scripts
> > >>> hadoop-daemon.sh, start and stop scripts.
> > >>> 4. Addition of block placement policy implemnation to support cloud
> > >>> enviroment, more specifically Azure.
> > >>>
> > >>> We are very close to wrapping up the work in branch-trunk-win and
> > >>>getting
> > >>> ready for a merge. Currently the merge patch is passing close to 100%
> > of
> > >>> unit tests on Linux. Soon I will call for a vote to merge this branch
> > >>>into
> > >>> trunk.
> > >>>
> > >>> Next steps:
> > >>> 1. Call for vote to merge branch-trunk-win to trunk, when the work
> > >>> completes and precommit build is clean.
> > >>> 2. Start a discussion on adding Jenkins precommit builds on windows
> and
> > >>> how to integrate that with the existing commit process.
> > >>>
> > >>> Let me know if you have any questions.
> > >>>
> > >>> Regards,
> > >>> Suresh
> > >>>
> > >>>
> > >>
> > >>
> > >>--
> > >>http://hortonworks.com/download/
> > >
> >
>
>
>
> --
> http://hortonworks.com/download/
>



-- 
Todd Lipcon
Software Engineer, Cloudera

Re: [Vote] Merge branch-trunk-win to trunk

[Todd Lipcon <to...@cloudera.com>]

On Wed, Feb 27, 2013 at 2:54 PM, Suresh Srinivas <su...@hortonworks.com>wrote:

> With that we need to decide how our precommit process looks.
> My inclination is to wait for +1 from precommit builds on
> both the platforms to ensure no issues are introduced.
> Thoughts?
>
> 2. Feature development impact
> Some questions have been raised about would new features
> need to be supported on both the platforms. Yes. I do not see a
> reason why features cannot work on both the platforms, with
> the exception of platform specific optimizations. This what Java
> gives us.
>
>
I'm concerned about the above. Personally, I don't have access to any
Windows boxes with development tools, and I know nothing about developing
on Windows. The only Windows I run is an 8GB VM with 1 GB RAM allocated,
for powerpoint :)

If I submit a patch and it gets -1 "tests failed" on the Windows slave, how
am I supposed to proceed?

I think a reasonable compromise would be that the tests should always
*build* on Windows before commit, and contributors should do their best to
look at the test logs for any Windows-specific failures. But, beyond
looking at the logs, a "-1 Tests failed on windows" should not block a
commit.

Those contributors who are interested in Windows being a first-class
platform should be responsible for watching the Windows builds and
debugging/fixing any regressions that might be Windows-specific.

I also think the KDE model that Harsh pointed out is an interesting one --
ie the idea that we would not merge windows support to trunk, but rather
treat is as a "parallel code line" which lives in the ASF and has its own
builds and releases. The windows team would periodically merge trunk->win
to pick up any new changes, and do a separate test/release process. I'm not
convinced this is the best idea, but worth discussion of pros and cons.

-Todd


>
> On Wed, Feb 27, 2013 at 11:56 AM, Eli Collins <el...@cloudera.com> wrote:
>
> > Bobby raises some good questions.  A related one, since most current
> > developers won't add Windows support for new features that are
> > platform specific is it assumed that Windows development will either
> > lag or will people actively work on keeping Windows up with the
> > latest?  And vice versa in case Windows support is implemented first.
> >
> > Is there a jira for resolving the outstanding TODOs in the code base
> > (similar to HDFS-2148)?  Looks like this merge doesn't introduce many
> > which is great (just did a quick diff and grep).
> >
> > Thanks,
> > Eli
> >
> > On Wed, Feb 27, 2013 at 8:17 AM, Robert Evans <ev...@yahoo-inc.com>
> wrote:
> > > After this is merged in is Windows still going to be a second class
> > > citizen but happens to work for more than just development or is it a
> > > fully supported platform where if something breaks it can block a
> > release?
> > >  How do we as a community intend to keep Windows support from breaking?
> > > We don't have any Jenkins slaves to be able to run nightly tests to
> > > validate everything still compiles/runs.  This is not a blocker for me
> > > because we often rely on individuals and groups to test Hadoop, but I
> do
> > > think we need to have this discussion before we put it in.
> > >
> > > --Bobby
> > >
> > > On 2/26/13 4:55 PM, "Suresh Srinivas" <su...@hortonworks.com> wrote:
> > >
> > >>I had posted heads up about merging branch-trunk-win to trunk on Feb
> 8th.
> > >>I
> > >>am happy to announce that we are ready for the merge.
> > >>
> > >>Here is a brief recap on the highlights of the work done:
> > >>- Command-line scripts for the Hadoop surface area
> > >>- Mapping the HDFS permissions model to Windows
> > >>- Abstracted and reconciled mismatches around differences in Path
> > >>semantics
> > >>in Java and Windows
> > >>- Native Task Controller for Windows
> > >>- Implementation of a Block Placement Policy to support cloud
> > >>environments,
> > >>more specifically Azure.
> > >>- Implementation of Hadoop native libraries for Windows (compression
> > >>codecs, native I/O)
> > >>- Several reliability issues, including race-conditions, intermittent
> > test
> > >>failures, resource leaks.
> > >>- Several new unit test cases written for the above changes
> > >>
> > >>Please find the details of the work in CHANGES.branch-trunk-win.txt -
> > >>Common changes<http://bit.ly/Xe7Ynv>, HDFS changes<
> http://bit.ly/13QOSo9
> > >,
> > >>and YARN and MapReduce changes <http://bit.ly/128zzMt>. This is the
> work
> > >>ported from branch-1-win to a branch based on trunk.
> > >>
> > >>For details of the testing done, please see the thread -
> > >>http://bit.ly/WpavJ4. Merge patch for this is available on
> HADOOP-8562<
> > >>https://issues.apache.org/jira/browse/HADOOP-8562>.
> > >>
> > >>This was a large undertaking that involved developing code, testing the
> > >>entire Hadoop stack, including scale tests. This is made possible only
> > >>with
> > >>the contribution from many many folks in the community. Following
> people
> > >>contributed to this work: Ivan Mitic, Chuan Liu, Ramya Sunil, Bikas
> Saha,
> > >>Kanna Karanam, John Gordon, Brandon Li, Chris Nauroth, David Lao,
> > Sumadhur
> > >>Reddy Bolli, Arpit Agarwal, Ahmed El Baz, Mike Liddell, Jing Zhao,
> Thejas
> > >>Nair, Steve Maine, Ganeshan Iyer, Raja Aluri, Giridharan Kesavan, Ramya
> > >>Bharathi Nimmagadda, Daryn Sharp, Arun Murthy, Tsz-Wo Nicholas Sze,
> > Suresh
> > >>Srinivas and Sanjay Radia. There are many others who contributed as
> well
> > >>providing feedback and comments on numerous jiras.
> > >>
> > >>The vote will run for seven days and will end on March 5, 6:00PM PST.
> > >>
> > >>Regards,
> > >>Suresh
> > >>
> > >>
> > >>
> > >>
> > >>On Thu, Feb 7, 2013 at 6:41 PM, Mahadevan Venkatraman
> > >><ma...@microsoft.com>wrote:
> > >>
> > >>> It is super exciting to look at the prospect of these changes being
> > >>>merged
> > >>> to trunk. Having Windows as one of the supported Hadoop platforms is
> a
> > >>> fantastic opportunity both for the Hadoop project and Microsoft
> > >>>customers.
> > >>>
> > >>> This work began around a year back when a few of us started with a
> > basic
> > >>> port of Hadoop on Windows. Ever since, the Hadoop team in Microsoft
> > have
> > >>> made significant progress in the following areas:
> > >>> (PS: Some of these items are already included in Suresh's email, but
> > >>> including again for completeness)
> > >>>
> > >>> - Command-line scripts for the Hadoop surface area
> > >>> - Mapping the HDFS permissions model to Windows
> > >>> - Abstracted and reconciled mismatches around differences in Path
> > >>> semantics in Java and Windows
> > >>> - Native Task Controller for Windows
> > >>> - Implementation of a Block Placement Policy to support cloud
> > >>> environments, more specifically Azure.
> > >>> - Implementation of Hadoop native libraries for Windows (compression
> > >>> codecs, native I/O) - Several reliability issues, including
> > >>> race-conditions, intermittent test failures, resource leaks.
> > >>> - Several new unit test cases written for the above changes
> > >>>
> > >>> In the process, we have closely engaged with the Apache open source
> > >>> community and have got great support and assistance from the
> community
> > >>>in
> > >>> terms of contributing fixes, code review comments and commits.
> > >>>
> > >>> In addition, the Hadoop team at Microsoft has also made good progress
> > in
> > >>> other projects including Hive, Pig, Sqoop, Oozie, HCat and HBase.
> Many
> > >>>of
> > >>> these changes have already been committed to the respective trunks
> with
> > >>> help from various committers and contributors. It is great to see the
> > >>> commitment of the community to support multiple platforms, and we
> look
> > >>> forward to the day when a developer/customer is able to successfully
> > >>>deploy
> > >>> a complete solution stack based on Apache Hadoop releases.
> > >>>
> > >>> Next Steps:
> > >>>
> > >>> All of the above changes are part of the Windows Azure HDInsight and
> > >>> HDInsight Server products from Microsoft. We have successfully
> > >>>on-boarded
> > >>> several internal customers and have been running production workloads
> > on
> > >>> Windows Azure HDInsight. Our vision is to create a big data platform
> > >>>based
> > >>> on Hadoop, and we are committed to helping make Hadoop a world-class
> > >>> solution that anyone can use to solve their biggest data challenges.
> > >>>
> > >>> As an immediate next step, we would like to have a discussion around
> > how
> > >>> we can ensure that the quality of the mainline Hadoop branches on
> > >>>Windows
> > >>> is maintained. To this end, we would like to get to the state where
> we
> > >>>have
> > >>> pre-checkin validation gates and nightly test runs enabled on
> Windows.
> > >>>If
> > >>> you have any suggestions around this, please do send an email.  We
> are
> > >>> committed to helping sustain the long-term quality of Hadoop on both
> > >>>Linux
> > >>> and Windows.
> > >>>
> > >>> We sincerely thank the community for their contribution and support
> so
> > >>> far. And hope to continue having a close engagement in the future.
> > >>>
> > >>> -Microsoft HDInsight Team
> > >>>
> > >>>
> > >>> -----Original Message-----
> > >>> From: Suresh Srinivas [mailto:suresh@hortonworks.com]
> > >>> Sent: Thursday, February 7, 2013 5:42 PM
> > >>> To: common-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org;
> > >>> hdfs-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
> > >>> Subject: Heads up - merge branch-trunk-win to trunk
> > >>>
> > >>> The support for Hadoop on Windows was proposed in HADOOP-8079<
> > >>> https://issues.apache.org/jira/browse/HADOOP-8079> almost a year
> ago.
> > >>>The
> > >>> goal was to make Hadoop natively integrated, full-featured, and
> > >>>performance
> > >>> and scalability tuned on Windows Server or Windows Azure.
> > >>> We are happy to announce that a lot of progress has been made in this
> > >>> regard.
> > >>>
> > >>> Initial work started in a feature branch, branch-1-win, based on
> > >>>branch-1.
> > >>> The details related to the work done in the branch can be seen in
> > >>> CHANGES.txt<
> > >>>
> > >>>
> > http://svn.apache.org/viewvc/hadoop/common/branches/branch-1-win/CHANGES
> .
> > >>>branch-1-win.txt?view=markup
> > >>> >.
> > >>> This work has been ported to a branch, branch-trunk-win, based on
> > trunk.
> > >>> Merge patch for this is available on
> > >>> HADOOP-8562<https://issues.apache.org/jira/browse/HADOOP-8562>
> > >>> .
> > >>>
> > >>> Highlights of the work done so far:
> > >>> 1. Necessary changes in Hadoop to run natively on Windows. These
> > changes
> > >>> handle differences in platforms related to path names, process/task
> > >>> management etc.
> > >>> 2. Addition of winutils tools for managing file permissions and
> > >>>ownership,
> > >>> user group mapping, hardlinks, symbolic links, chmod, disk
> utilization,
> > >>>and
> > >>> process/task management.
> > >>> 3. Added cmd scripts equivalent to existing shell scripts
> > >>> hadoop-daemon.sh, start and stop scripts.
> > >>> 4. Addition of block placement policy implemnation to support cloud
> > >>> enviroment, more specifically Azure.
> > >>>
> > >>> We are very close to wrapping up the work in branch-trunk-win and
> > >>>getting
> > >>> ready for a merge. Currently the merge patch is passing close to 100%
> > of
> > >>> unit tests on Linux. Soon I will call for a vote to merge this branch
> > >>>into
> > >>> trunk.
> > >>>
> > >>> Next steps:
> > >>> 1. Call for vote to merge branch-trunk-win to trunk, when the work
> > >>> completes and precommit build is clean.
> > >>> 2. Start a discussion on adding Jenkins precommit builds on windows
> and
> > >>> how to integrate that with the existing commit process.
> > >>>
> > >>> Let me know if you have any questions.
> > >>>
> > >>> Regards,
> > >>> Suresh
> > >>>
> > >>>
> > >>
> > >>
> > >>--
> > >>http://hortonworks.com/download/
> > >
> >
>
>
>
> --
> http://hortonworks.com/download/
>



-- 
Todd Lipcon
Software Engineer, Cloudera

Re: [Vote] Merge branch-trunk-win to trunk

[Todd Lipcon <to...@cloudera.com>]

On Wed, Feb 27, 2013 at 2:54 PM, Suresh Srinivas <su...@hortonworks.com>wrote:

> With that we need to decide how our precommit process looks.
> My inclination is to wait for +1 from precommit builds on
> both the platforms to ensure no issues are introduced.
> Thoughts?
>
> 2. Feature development impact
> Some questions have been raised about would new features
> need to be supported on both the platforms. Yes. I do not see a
> reason why features cannot work on both the platforms, with
> the exception of platform specific optimizations. This what Java
> gives us.
>
>
I'm concerned about the above. Personally, I don't have access to any
Windows boxes with development tools, and I know nothing about developing
on Windows. The only Windows I run is an 8GB VM with 1 GB RAM allocated,
for powerpoint :)

If I submit a patch and it gets -1 "tests failed" on the Windows slave, how
am I supposed to proceed?

I think a reasonable compromise would be that the tests should always
*build* on Windows before commit, and contributors should do their best to
look at the test logs for any Windows-specific failures. But, beyond
looking at the logs, a "-1 Tests failed on windows" should not block a
commit.

Those contributors who are interested in Windows being a first-class
platform should be responsible for watching the Windows builds and
debugging/fixing any regressions that might be Windows-specific.

I also think the KDE model that Harsh pointed out is an interesting one --
ie the idea that we would not merge windows support to trunk, but rather
treat is as a "parallel code line" which lives in the ASF and has its own
builds and releases. The windows team would periodically merge trunk->win
to pick up any new changes, and do a separate test/release process. I'm not
convinced this is the best idea, but worth discussion of pros and cons.

-Todd


>
> On Wed, Feb 27, 2013 at 11:56 AM, Eli Collins <el...@cloudera.com> wrote:
>
> > Bobby raises some good questions.  A related one, since most current
> > developers won't add Windows support for new features that are
> > platform specific is it assumed that Windows development will either
> > lag or will people actively work on keeping Windows up with the
> > latest?  And vice versa in case Windows support is implemented first.
> >
> > Is there a jira for resolving the outstanding TODOs in the code base
> > (similar to HDFS-2148)?  Looks like this merge doesn't introduce many
> > which is great (just did a quick diff and grep).
> >
> > Thanks,
> > Eli
> >
> > On Wed, Feb 27, 2013 at 8:17 AM, Robert Evans <ev...@yahoo-inc.com>
> wrote:
> > > After this is merged in is Windows still going to be a second class
> > > citizen but happens to work for more than just development or is it a
> > > fully supported platform where if something breaks it can block a
> > release?
> > >  How do we as a community intend to keep Windows support from breaking?
> > > We don't have any Jenkins slaves to be able to run nightly tests to
> > > validate everything still compiles/runs.  This is not a blocker for me
> > > because we often rely on individuals and groups to test Hadoop, but I
> do
> > > think we need to have this discussion before we put it in.
> > >
> > > --Bobby
> > >
> > > On 2/26/13 4:55 PM, "Suresh Srinivas" <su...@hortonworks.com> wrote:
> > >
> > >>I had posted heads up about merging branch-trunk-win to trunk on Feb
> 8th.
> > >>I
> > >>am happy to announce that we are ready for the merge.
> > >>
> > >>Here is a brief recap on the highlights of the work done:
> > >>- Command-line scripts for the Hadoop surface area
> > >>- Mapping the HDFS permissions model to Windows
> > >>- Abstracted and reconciled mismatches around differences in Path
> > >>semantics
> > >>in Java and Windows
> > >>- Native Task Controller for Windows
> > >>- Implementation of a Block Placement Policy to support cloud
> > >>environments,
> > >>more specifically Azure.
> > >>- Implementation of Hadoop native libraries for Windows (compression
> > >>codecs, native I/O)
> > >>- Several reliability issues, including race-conditions, intermittent
> > test
> > >>failures, resource leaks.
> > >>- Several new unit test cases written for the above changes
> > >>
> > >>Please find the details of the work in CHANGES.branch-trunk-win.txt -
> > >>Common changes<http://bit.ly/Xe7Ynv>, HDFS changes<
> http://bit.ly/13QOSo9
> > >,
> > >>and YARN and MapReduce changes <http://bit.ly/128zzMt>. This is the
> work
> > >>ported from branch-1-win to a branch based on trunk.
> > >>
> > >>For details of the testing done, please see the thread -
> > >>http://bit.ly/WpavJ4. Merge patch for this is available on
> HADOOP-8562<
> > >>https://issues.apache.org/jira/browse/HADOOP-8562>.
> > >>
> > >>This was a large undertaking that involved developing code, testing the
> > >>entire Hadoop stack, including scale tests. This is made possible only
> > >>with
> > >>the contribution from many many folks in the community. Following
> people
> > >>contributed to this work: Ivan Mitic, Chuan Liu, Ramya Sunil, Bikas
> Saha,
> > >>Kanna Karanam, John Gordon, Brandon Li, Chris Nauroth, David Lao,
> > Sumadhur
> > >>Reddy Bolli, Arpit Agarwal, Ahmed El Baz, Mike Liddell, Jing Zhao,
> Thejas
> > >>Nair, Steve Maine, Ganeshan Iyer, Raja Aluri, Giridharan Kesavan, Ramya
> > >>Bharathi Nimmagadda, Daryn Sharp, Arun Murthy, Tsz-Wo Nicholas Sze,
> > Suresh
> > >>Srinivas and Sanjay Radia. There are many others who contributed as
> well
> > >>providing feedback and comments on numerous jiras.
> > >>
> > >>The vote will run for seven days and will end on March 5, 6:00PM PST.
> > >>
> > >>Regards,
> > >>Suresh
> > >>
> > >>
> > >>
> > >>
> > >>On Thu, Feb 7, 2013 at 6:41 PM, Mahadevan Venkatraman
> > >><ma...@microsoft.com>wrote:
> > >>
> > >>> It is super exciting to look at the prospect of these changes being
> > >>>merged
> > >>> to trunk. Having Windows as one of the supported Hadoop platforms is
> a
> > >>> fantastic opportunity both for the Hadoop project and Microsoft
> > >>>customers.
> > >>>
> > >>> This work began around a year back when a few of us started with a
> > basic
> > >>> port of Hadoop on Windows. Ever since, the Hadoop team in Microsoft
> > have
> > >>> made significant progress in the following areas:
> > >>> (PS: Some of these items are already included in Suresh's email, but
> > >>> including again for completeness)
> > >>>
> > >>> - Command-line scripts for the Hadoop surface area
> > >>> - Mapping the HDFS permissions model to Windows
> > >>> - Abstracted and reconciled mismatches around differences in Path
> > >>> semantics in Java and Windows
> > >>> - Native Task Controller for Windows
> > >>> - Implementation of a Block Placement Policy to support cloud
> > >>> environments, more specifically Azure.
> > >>> - Implementation of Hadoop native libraries for Windows (compression
> > >>> codecs, native I/O) - Several reliability issues, including
> > >>> race-conditions, intermittent test failures, resource leaks.
> > >>> - Several new unit test cases written for the above changes
> > >>>
> > >>> In the process, we have closely engaged with the Apache open source
> > >>> community and have got great support and assistance from the
> community
> > >>>in
> > >>> terms of contributing fixes, code review comments and commits.
> > >>>
> > >>> In addition, the Hadoop team at Microsoft has also made good progress
> > in
> > >>> other projects including Hive, Pig, Sqoop, Oozie, HCat and HBase.
> Many
> > >>>of
> > >>> these changes have already been committed to the respective trunks
> with
> > >>> help from various committers and contributors. It is great to see the
> > >>> commitment of the community to support multiple platforms, and we
> look
> > >>> forward to the day when a developer/customer is able to successfully
> > >>>deploy
> > >>> a complete solution stack based on Apache Hadoop releases.
> > >>>
> > >>> Next Steps:
> > >>>
> > >>> All of the above changes are part of the Windows Azure HDInsight and
> > >>> HDInsight Server products from Microsoft. We have successfully
> > >>>on-boarded
> > >>> several internal customers and have been running production workloads
> > on
> > >>> Windows Azure HDInsight. Our vision is to create a big data platform
> > >>>based
> > >>> on Hadoop, and we are committed to helping make Hadoop a world-class
> > >>> solution that anyone can use to solve their biggest data challenges.
> > >>>
> > >>> As an immediate next step, we would like to have a discussion around
> > how
> > >>> we can ensure that the quality of the mainline Hadoop branches on
> > >>>Windows
> > >>> is maintained. To this end, we would like to get to the state where
> we
> > >>>have
> > >>> pre-checkin validation gates and nightly test runs enabled on
> Windows.
> > >>>If
> > >>> you have any suggestions around this, please do send an email.  We
> are
> > >>> committed to helping sustain the long-term quality of Hadoop on both
> > >>>Linux
> > >>> and Windows.
> > >>>
> > >>> We sincerely thank the community for their contribution and support
> so
> > >>> far. And hope to continue having a close engagement in the future.
> > >>>
> > >>> -Microsoft HDInsight Team
> > >>>
> > >>>
> > >>> -----Original Message-----
> > >>> From: Suresh Srinivas [mailto:suresh@hortonworks.com]
> > >>> Sent: Thursday, February 7, 2013 5:42 PM
> > >>> To: common-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org;
> > >>> hdfs-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
> > >>> Subject: Heads up - merge branch-trunk-win to trunk
> > >>>
> > >>> The support for Hadoop on Windows was proposed in HADOOP-8079<
> > >>> https://issues.apache.org/jira/browse/HADOOP-8079> almost a year
> ago.
> > >>>The
> > >>> goal was to make Hadoop natively integrated, full-featured, and
> > >>>performance
> > >>> and scalability tuned on Windows Server or Windows Azure.
> > >>> We are happy to announce that a lot of progress has been made in this
> > >>> regard.
> > >>>
> > >>> Initial work started in a feature branch, branch-1-win, based on
> > >>>branch-1.
> > >>> The details related to the work done in the branch can be seen in
> > >>> CHANGES.txt<
> > >>>
> > >>>
> > http://svn.apache.org/viewvc/hadoop/common/branches/branch-1-win/CHANGES
> .
> > >>>branch-1-win.txt?view=markup
> > >>> >.
> > >>> This work has been ported to a branch, branch-trunk-win, based on
> > trunk.
> > >>> Merge patch for this is available on
> > >>> HADOOP-8562<https://issues.apache.org/jira/browse/HADOOP-8562>
> > >>> .
> > >>>
> > >>> Highlights of the work done so far:
> > >>> 1. Necessary changes in Hadoop to run natively on Windows. These
> > changes
> > >>> handle differences in platforms related to path names, process/task
> > >>> management etc.
> > >>> 2. Addition of winutils tools for managing file permissions and
> > >>>ownership,
> > >>> user group mapping, hardlinks, symbolic links, chmod, disk
> utilization,
> > >>>and
> > >>> process/task management.
> > >>> 3. Added cmd scripts equivalent to existing shell scripts
> > >>> hadoop-daemon.sh, start and stop scripts.
> > >>> 4. Addition of block placement policy implemnation to support cloud
> > >>> enviroment, more specifically Azure.
> > >>>
> > >>> We are very close to wrapping up the work in branch-trunk-win and
> > >>>getting
> > >>> ready for a merge. Currently the merge patch is passing close to 100%
> > of
> > >>> unit tests on Linux. Soon I will call for a vote to merge this branch
> > >>>into
> > >>> trunk.
> > >>>
> > >>> Next steps:
> > >>> 1. Call for vote to merge branch-trunk-win to trunk, when the work
> > >>> completes and precommit build is clean.
> > >>> 2. Start a discussion on adding Jenkins precommit builds on windows
> and
> > >>> how to integrate that with the existing commit process.
> > >>>
> > >>> Let me know if you have any questions.
> > >>>
> > >>> Regards,
> > >>> Suresh
> > >>>
> > >>>
> > >>
> > >>
> > >>--
> > >>http://hortonworks.com/download/
> > >
> >
>
>
>
> --
> http://hortonworks.com/download/
>



-- 
Todd Lipcon
Software Engineer, Cloudera

Re: [Vote] Merge branch-trunk-win to trunk

[Suresh Srinivas <su...@hortonworks.com>]

Thanks for raising good questions.

Currently the merge patch passes all the tests on Linux, hence
the proposal for merging the patch to trunk. But as Bobby, Harsh
and Eli pointed out, before declaring support for Windows, we need the
discussion on the following:

1. Precommit and development process
Jenkins infrastructure for Windows build will be made available.
Giri and Microsoft contributors have volunteered to help make
this happen.

With that we need to decide how our precommit process looks.
My inclination is to wait for +1 from precommit builds on
both the platforms to ensure no issues are introduced.
Thoughts?

2. Feature development impact
Some questions have been raised about would new features
need to be supported on both the platforms. Yes. I do not see a
reason why features cannot work on both the platforms, with
the exception of platform specific optimizations. This what Java
gives us.

3. Platform specific features/optimizations
As regards platform specific optimization, each platform can
evolve at its own pace and should not block progress of a
specific platform.

As indicated in my earlier email, there is a sizable number
of contributors to work on issues and support of Hadoop on Windows
platform. I am excited to see Hadoop reach the other large
part of server market.

Eli, as pointed out by you, the TODO items need to be addressed.
Also we realized we still need to add information on how to
build on Windows in BUILDING.txt. We will address this ASAP.
Giri and Matt have some expirience with this and should be able
to provide more information.



On Wed, Feb 27, 2013 at 11:56 AM, Eli Collins <el...@cloudera.com> wrote:

> Bobby raises some good questions.  A related one, since most current
> developers won't add Windows support for new features that are
> platform specific is it assumed that Windows development will either
> lag or will people actively work on keeping Windows up with the
> latest?  And vice versa in case Windows support is implemented first.
>
> Is there a jira for resolving the outstanding TODOs in the code base
> (similar to HDFS-2148)?  Looks like this merge doesn't introduce many
> which is great (just did a quick diff and grep).
>
> Thanks,
> Eli
>
> On Wed, Feb 27, 2013 at 8:17 AM, Robert Evans <ev...@yahoo-inc.com> wrote:
> > After this is merged in is Windows still going to be a second class
> > citizen but happens to work for more than just development or is it a
> > fully supported platform where if something breaks it can block a
> release?
> >  How do we as a community intend to keep Windows support from breaking?
> > We don't have any Jenkins slaves to be able to run nightly tests to
> > validate everything still compiles/runs.  This is not a blocker for me
> > because we often rely on individuals and groups to test Hadoop, but I do
> > think we need to have this discussion before we put it in.
> >
> > --Bobby
> >
> > On 2/26/13 4:55 PM, "Suresh Srinivas" <su...@hortonworks.com> wrote:
> >
> >>I had posted heads up about merging branch-trunk-win to trunk on Feb 8th.
> >>I
> >>am happy to announce that we are ready for the merge.
> >>
> >>Here is a brief recap on the highlights of the work done:
> >>- Command-line scripts for the Hadoop surface area
> >>- Mapping the HDFS permissions model to Windows
> >>- Abstracted and reconciled mismatches around differences in Path
> >>semantics
> >>in Java and Windows
> >>- Native Task Controller for Windows
> >>- Implementation of a Block Placement Policy to support cloud
> >>environments,
> >>more specifically Azure.
> >>- Implementation of Hadoop native libraries for Windows (compression
> >>codecs, native I/O)
> >>- Several reliability issues, including race-conditions, intermittent
> test
> >>failures, resource leaks.
> >>- Several new unit test cases written for the above changes
> >>
> >>Please find the details of the work in CHANGES.branch-trunk-win.txt -
> >>Common changes<http://bit.ly/Xe7Ynv>, HDFS changes<http://bit.ly/13QOSo9
> >,
> >>and YARN and MapReduce changes <http://bit.ly/128zzMt>. This is the work
> >>ported from branch-1-win to a branch based on trunk.
> >>
> >>For details of the testing done, please see the thread -
> >>http://bit.ly/WpavJ4. Merge patch for this is available on HADOOP-8562<
> >>https://issues.apache.org/jira/browse/HADOOP-8562>.
> >>
> >>This was a large undertaking that involved developing code, testing the
> >>entire Hadoop stack, including scale tests. This is made possible only
> >>with
> >>the contribution from many many folks in the community. Following people
> >>contributed to this work: Ivan Mitic, Chuan Liu, Ramya Sunil, Bikas Saha,
> >>Kanna Karanam, John Gordon, Brandon Li, Chris Nauroth, David Lao,
> Sumadhur
> >>Reddy Bolli, Arpit Agarwal, Ahmed El Baz, Mike Liddell, Jing Zhao, Thejas
> >>Nair, Steve Maine, Ganeshan Iyer, Raja Aluri, Giridharan Kesavan, Ramya
> >>Bharathi Nimmagadda, Daryn Sharp, Arun Murthy, Tsz-Wo Nicholas Sze,
> Suresh
> >>Srinivas and Sanjay Radia. There are many others who contributed as well
> >>providing feedback and comments on numerous jiras.
> >>
> >>The vote will run for seven days and will end on March 5, 6:00PM PST.
> >>
> >>Regards,
> >>Suresh
> >>
> >>
> >>
> >>
> >>On Thu, Feb 7, 2013 at 6:41 PM, Mahadevan Venkatraman
> >><ma...@microsoft.com>wrote:
> >>
> >>> It is super exciting to look at the prospect of these changes being
> >>>merged
> >>> to trunk. Having Windows as one of the supported Hadoop platforms is a
> >>> fantastic opportunity both for the Hadoop project and Microsoft
> >>>customers.
> >>>
> >>> This work began around a year back when a few of us started with a
> basic
> >>> port of Hadoop on Windows. Ever since, the Hadoop team in Microsoft
> have
> >>> made significant progress in the following areas:
> >>> (PS: Some of these items are already included in Suresh's email, but
> >>> including again for completeness)
> >>>
> >>> - Command-line scripts for the Hadoop surface area
> >>> - Mapping the HDFS permissions model to Windows
> >>> - Abstracted and reconciled mismatches around differences in Path
> >>> semantics in Java and Windows
> >>> - Native Task Controller for Windows
> >>> - Implementation of a Block Placement Policy to support cloud
> >>> environments, more specifically Azure.
> >>> - Implementation of Hadoop native libraries for Windows (compression
> >>> codecs, native I/O) - Several reliability issues, including
> >>> race-conditions, intermittent test failures, resource leaks.
> >>> - Several new unit test cases written for the above changes
> >>>
> >>> In the process, we have closely engaged with the Apache open source
> >>> community and have got great support and assistance from the community
> >>>in
> >>> terms of contributing fixes, code review comments and commits.
> >>>
> >>> In addition, the Hadoop team at Microsoft has also made good progress
> in
> >>> other projects including Hive, Pig, Sqoop, Oozie, HCat and HBase. Many
> >>>of
> >>> these changes have already been committed to the respective trunks with
> >>> help from various committers and contributors. It is great to see the
> >>> commitment of the community to support multiple platforms, and we look
> >>> forward to the day when a developer/customer is able to successfully
> >>>deploy
> >>> a complete solution stack based on Apache Hadoop releases.
> >>>
> >>> Next Steps:
> >>>
> >>> All of the above changes are part of the Windows Azure HDInsight and
> >>> HDInsight Server products from Microsoft. We have successfully
> >>>on-boarded
> >>> several internal customers and have been running production workloads
> on
> >>> Windows Azure HDInsight. Our vision is to create a big data platform
> >>>based
> >>> on Hadoop, and we are committed to helping make Hadoop a world-class
> >>> solution that anyone can use to solve their biggest data challenges.
> >>>
> >>> As an immediate next step, we would like to have a discussion around
> how
> >>> we can ensure that the quality of the mainline Hadoop branches on
> >>>Windows
> >>> is maintained. To this end, we would like to get to the state where we
> >>>have
> >>> pre-checkin validation gates and nightly test runs enabled on Windows.
> >>>If
> >>> you have any suggestions around this, please do send an email.  We are
> >>> committed to helping sustain the long-term quality of Hadoop on both
> >>>Linux
> >>> and Windows.
> >>>
> >>> We sincerely thank the community for their contribution and support so
> >>> far. And hope to continue having a close engagement in the future.
> >>>
> >>> -Microsoft HDInsight Team
> >>>
> >>>
> >>> -----Original Message-----
> >>> From: Suresh Srinivas [mailto:suresh@hortonworks.com]
> >>> Sent: Thursday, February 7, 2013 5:42 PM
> >>> To: common-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org;
> >>> hdfs-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
> >>> Subject: Heads up - merge branch-trunk-win to trunk
> >>>
> >>> The support for Hadoop on Windows was proposed in HADOOP-8079<
> >>> https://issues.apache.org/jira/browse/HADOOP-8079> almost a year ago.
> >>>The
> >>> goal was to make Hadoop natively integrated, full-featured, and
> >>>performance
> >>> and scalability tuned on Windows Server or Windows Azure.
> >>> We are happy to announce that a lot of progress has been made in this
> >>> regard.
> >>>
> >>> Initial work started in a feature branch, branch-1-win, based on
> >>>branch-1.
> >>> The details related to the work done in the branch can be seen in
> >>> CHANGES.txt<
> >>>
> >>>
> http://svn.apache.org/viewvc/hadoop/common/branches/branch-1-win/CHANGES.
> >>>branch-1-win.txt?view=markup
> >>> >.
> >>> This work has been ported to a branch, branch-trunk-win, based on
> trunk.
> >>> Merge patch for this is available on
> >>> HADOOP-8562<https://issues.apache.org/jira/browse/HADOOP-8562>
> >>> .
> >>>
> >>> Highlights of the work done so far:
> >>> 1. Necessary changes in Hadoop to run natively on Windows. These
> changes
> >>> handle differences in platforms related to path names, process/task
> >>> management etc.
> >>> 2. Addition of winutils tools for managing file permissions and
> >>>ownership,
> >>> user group mapping, hardlinks, symbolic links, chmod, disk utilization,
> >>>and
> >>> process/task management.
> >>> 3. Added cmd scripts equivalent to existing shell scripts
> >>> hadoop-daemon.sh, start and stop scripts.
> >>> 4. Addition of block placement policy implemnation to support cloud
> >>> enviroment, more specifically Azure.
> >>>
> >>> We are very close to wrapping up the work in branch-trunk-win and
> >>>getting
> >>> ready for a merge. Currently the merge patch is passing close to 100%
> of
> >>> unit tests on Linux. Soon I will call for a vote to merge this branch
> >>>into
> >>> trunk.
> >>>
> >>> Next steps:
> >>> 1. Call for vote to merge branch-trunk-win to trunk, when the work
> >>> completes and precommit build is clean.
> >>> 2. Start a discussion on adding Jenkins precommit builds on windows and
> >>> how to integrate that with the existing commit process.
> >>>
> >>> Let me know if you have any questions.
> >>>
> >>> Regards,
> >>> Suresh
> >>>
> >>>
> >>
> >>
> >>--
> >>http://hortonworks.com/download/
> >
>



-- 
http://hortonworks.com/download/

Re: [Vote] Merge branch-trunk-win to trunk

[Arpit Agarwal <aa...@hortonworks.com>]

+1 non-binding.

I have extensively tested this on both Windows and Linux over the last few
months.

Thanks,
-Arpit

On Wed, Feb 27, 2013 at 11:56 AM, Eli Collins <el...@cloudera.com> wrote:

> Bobby raises some good questions.  A related one, since most current
> developers won't add Windows support for new features that are
> platform specific is it assumed that Windows development will either
> lag or will people actively work on keeping Windows up with the
> latest?  And vice versa in case Windows support is implemented first.
>
> Is there a jira for resolving the outstanding TODOs in the code base
> (similar to HDFS-2148)?  Looks like this merge doesn't introduce many
> which is great (just did a quick diff and grep).
>
> Thanks,
> Eli
>
> On Wed, Feb 27, 2013 at 8:17 AM, Robert Evans <ev...@yahoo-inc.com> wrote:
> > After this is merged in is Windows still going to be a second class
> > citizen but happens to work for more than just development or is it a
> > fully supported platform where if something breaks it can block a
> release?
> >  How do we as a community intend to keep Windows support from breaking?
> > We don't have any Jenkins slaves to be able to run nightly tests to
> > validate everything still compiles/runs.  This is not a blocker for me
> > because we often rely on individuals and groups to test Hadoop, but I do
> > think we need to have this discussion before we put it in.
> >
> > --Bobby
> >
> > On 2/26/13 4:55 PM, "Suresh Srinivas" <su...@hortonworks.com> wrote:
> >
> >>I had posted heads up about merging branch-trunk-win to trunk on Feb 8th.
> >>I
> >>am happy to announce that we are ready for the merge.
> >>
> >>Here is a brief recap on the highlights of the work done:
> >>- Command-line scripts for the Hadoop surface area
> >>- Mapping the HDFS permissions model to Windows
> >>- Abstracted and reconciled mismatches around differences in Path
> >>semantics
> >>in Java and Windows
> >>- Native Task Controller for Windows
> >>- Implementation of a Block Placement Policy to support cloud
> >>environments,
> >>more specifically Azure.
> >>- Implementation of Hadoop native libraries for Windows (compression
> >>codecs, native I/O)
> >>- Several reliability issues, including race-conditions, intermittent
> test
> >>failures, resource leaks.
> >>- Several new unit test cases written for the above changes
> >>
> >>Please find the details of the work in CHANGES.branch-trunk-win.txt -
> >>Common changes<http://bit.ly/Xe7Ynv>, HDFS changes<http://bit.ly/13QOSo9
> >,
> >>and YARN and MapReduce changes <http://bit.ly/128zzMt>. This is the work
> >>ported from branch-1-win to a branch based on trunk.
> >>
> >>For details of the testing done, please see the thread -
> >>http://bit.ly/WpavJ4. Merge patch for this is available on HADOOP-8562<
> >>https://issues.apache.org/jira/browse/HADOOP-8562>.
> >>
> >>This was a large undertaking that involved developing code, testing the
> >>entire Hadoop stack, including scale tests. This is made possible only
> >>with
> >>the contribution from many many folks in the community. Following people
> >>contributed to this work: Ivan Mitic, Chuan Liu, Ramya Sunil, Bikas Saha,
> >>Kanna Karanam, John Gordon, Brandon Li, Chris Nauroth, David Lao,
> Sumadhur
> >>Reddy Bolli, Arpit Agarwal, Ahmed El Baz, Mike Liddell, Jing Zhao, Thejas
> >>Nair, Steve Maine, Ganeshan Iyer, Raja Aluri, Giridharan Kesavan, Ramya
> >>Bharathi Nimmagadda, Daryn Sharp, Arun Murthy, Tsz-Wo Nicholas Sze,
> Suresh
> >>Srinivas and Sanjay Radia. There are many others who contributed as well
> >>providing feedback and comments on numerous jiras.
> >>
> >>The vote will run for seven days and will end on March 5, 6:00PM PST.
> >>
> >>Regards,
> >>Suresh
> >>
> >>
> >>
> >>
> >>On Thu, Feb 7, 2013 at 6:41 PM, Mahadevan Venkatraman
> >><ma...@microsoft.com>wrote:
> >>
> >>> It is super exciting to look at the prospect of these changes being
> >>>merged
> >>> to trunk. Having Windows as one of the supported Hadoop platforms is a
> >>> fantastic opportunity both for the Hadoop project and Microsoft
> >>>customers.
> >>>
> >>> This work began around a year back when a few of us started with a
> basic
> >>> port of Hadoop on Windows. Ever since, the Hadoop team in Microsoft
> have
> >>> made significant progress in the following areas:
> >>> (PS: Some of these items are already included in Suresh's email, but
> >>> including again for completeness)
> >>>
> >>> - Command-line scripts for the Hadoop surface area
> >>> - Mapping the HDFS permissions model to Windows
> >>> - Abstracted and reconciled mismatches around differences in Path
> >>> semantics in Java and Windows
> >>> - Native Task Controller for Windows
> >>> - Implementation of a Block Placement Policy to support cloud
> >>> environments, more specifically Azure.
> >>> - Implementation of Hadoop native libraries for Windows (compression
> >>> codecs, native I/O) - Several reliability issues, including
> >>> race-conditions, intermittent test failures, resource leaks.
> >>> - Several new unit test cases written for the above changes
> >>>
> >>> In the process, we have closely engaged with the Apache open source
> >>> community and have got great support and assistance from the community
> >>>in
> >>> terms of contributing fixes, code review comments and commits.
> >>>
> >>> In addition, the Hadoop team at Microsoft has also made good progress
> in
> >>> other projects including Hive, Pig, Sqoop, Oozie, HCat and HBase. Many
> >>>of
> >>> these changes have already been committed to the respective trunks with
> >>> help from various committers and contributors. It is great to see the
> >>> commitment of the community to support multiple platforms, and we look
> >>> forward to the day when a developer/customer is able to successfully
> >>>deploy
> >>> a complete solution stack based on Apache Hadoop releases.
> >>>
> >>> Next Steps:
> >>>
> >>> All of the above changes are part of the Windows Azure HDInsight and
> >>> HDInsight Server products from Microsoft. We have successfully
> >>>on-boarded
> >>> several internal customers and have been running production workloads
> on
> >>> Windows Azure HDInsight. Our vision is to create a big data platform
> >>>based
> >>> on Hadoop, and we are committed to helping make Hadoop a world-class
> >>> solution that anyone can use to solve their biggest data challenges.
> >>>
> >>> As an immediate next step, we would like to have a discussion around
> how
> >>> we can ensure that the quality of the mainline Hadoop branches on
> >>>Windows
> >>> is maintained. To this end, we would like to get to the state where we
> >>>have
> >>> pre-checkin validation gates and nightly test runs enabled on Windows.
> >>>If
> >>> you have any suggestions around this, please do send an email.  We are
> >>> committed to helping sustain the long-term quality of Hadoop on both
> >>>Linux
> >>> and Windows.
> >>>
> >>> We sincerely thank the community for their contribution and support so
> >>> far. And hope to continue having a close engagement in the future.
> >>>
> >>> -Microsoft HDInsight Team
> >>>
> >>>
> >>> -----Original Message-----
> >>> From: Suresh Srinivas [mailto:suresh@hortonworks.com]
> >>> Sent: Thursday, February 7, 2013 5:42 PM
> >>> To: common-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org;
> >>> hdfs-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
> >>> Subject: Heads up - merge branch-trunk-win to trunk
> >>>
> >>> The support for Hadoop on Windows was proposed in HADOOP-8079<
> >>> https://issues.apache.org/jira/browse/HADOOP-8079> almost a year ago.
> >>>The
> >>> goal was to make Hadoop natively integrated, full-featured, and
> >>>performance
> >>> and scalability tuned on Windows Server or Windows Azure.
> >>> We are happy to announce that a lot of progress has been made in this
> >>> regard.
> >>>
> >>> Initial work started in a feature branch, branch-1-win, based on
> >>>branch-1.
> >>> The details related to the work done in the branch can be seen in
> >>> CHANGES.txt<
> >>>
> >>>
> http://svn.apache.org/viewvc/hadoop/common/branches/branch-1-win/CHANGES.
> >>>branch-1-win.txt?view=markup
> >>> >.
> >>> This work has been ported to a branch, branch-trunk-win, based on
> trunk.
> >>> Merge patch for this is available on
> >>> HADOOP-8562<https://issues.apache.org/jira/browse/HADOOP-8562>
> >>> .
> >>>
> >>> Highlights of the work done so far:
> >>> 1. Necessary changes in Hadoop to run natively on Windows. These
> changes
> >>> handle differences in platforms related to path names, process/task
> >>> management etc.
> >>> 2. Addition of winutils tools for managing file permissions and
> >>>ownership,
> >>> user group mapping, hardlinks, symbolic links, chmod, disk utilization,
> >>>and
> >>> process/task management.
> >>> 3. Added cmd scripts equivalent to existing shell scripts
> >>> hadoop-daemon.sh, start and stop scripts.
> >>> 4. Addition of block placement policy implemnation to support cloud
> >>> enviroment, more specifically Azure.
> >>>
> >>> We are very close to wrapping up the work in branch-trunk-win and
> >>>getting
> >>> ready for a merge. Currently the merge patch is passing close to 100%
> of
> >>> unit tests on Linux. Soon I will call for a vote to merge this branch
> >>>into
> >>> trunk.
> >>>
> >>> Next steps:
> >>> 1. Call for vote to merge branch-trunk-win to trunk, when the work
> >>> completes and precommit build is clean.
> >>> 2. Start a discussion on adding Jenkins precommit builds on windows and
> >>> how to integrate that with the existing commit process.
> >>>
> >>> Let me know if you have any questions.
> >>>
> >>> Regards,
> >>> Suresh
> >>>
> >>>
> >>
> >>
> >>--
> >>http://hortonworks.com/download/
> >
>

Re: [Vote] Merge branch-trunk-win to trunk

[Suresh Srinivas <su...@hortonworks.com>]

Thanks for raising good questions.

Currently the merge patch passes all the tests on Linux, hence
the proposal for merging the patch to trunk. But as Bobby, Harsh
and Eli pointed out, before declaring support for Windows, we need the
discussion on the following:

1. Precommit and development process
Jenkins infrastructure for Windows build will be made available.
Giri and Microsoft contributors have volunteered to help make
this happen.

With that we need to decide how our precommit process looks.
My inclination is to wait for +1 from precommit builds on
both the platforms to ensure no issues are introduced.
Thoughts?

2. Feature development impact
Some questions have been raised about would new features
need to be supported on both the platforms. Yes. I do not see a
reason why features cannot work on both the platforms, with
the exception of platform specific optimizations. This what Java
gives us.

3. Platform specific features/optimizations
As regards platform specific optimization, each platform can
evolve at its own pace and should not block progress of a
specific platform.

As indicated in my earlier email, there is a sizable number
of contributors to work on issues and support of Hadoop on Windows
platform. I am excited to see Hadoop reach the other large
part of server market.

Eli, as pointed out by you, the TODO items need to be addressed.
Also we realized we still need to add information on how to
build on Windows in BUILDING.txt. We will address this ASAP.
Giri and Matt have some expirience with this and should be able
to provide more information.



On Wed, Feb 27, 2013 at 11:56 AM, Eli Collins <el...@cloudera.com> wrote:

> Bobby raises some good questions.  A related one, since most current
> developers won't add Windows support for new features that are
> platform specific is it assumed that Windows development will either
> lag or will people actively work on keeping Windows up with the
> latest?  And vice versa in case Windows support is implemented first.
>
> Is there a jira for resolving the outstanding TODOs in the code base
> (similar to HDFS-2148)?  Looks like this merge doesn't introduce many
> which is great (just did a quick diff and grep).
>
> Thanks,
> Eli
>
> On Wed, Feb 27, 2013 at 8:17 AM, Robert Evans <ev...@yahoo-inc.com> wrote:
> > After this is merged in is Windows still going to be a second class
> > citizen but happens to work for more than just development or is it a
> > fully supported platform where if something breaks it can block a
> release?
> >  How do we as a community intend to keep Windows support from breaking?
> > We don't have any Jenkins slaves to be able to run nightly tests to
> > validate everything still compiles/runs.  This is not a blocker for me
> > because we often rely on individuals and groups to test Hadoop, but I do
> > think we need to have this discussion before we put it in.
> >
> > --Bobby
> >
> > On 2/26/13 4:55 PM, "Suresh Srinivas" <su...@hortonworks.com> wrote:
> >
> >>I had posted heads up about merging branch-trunk-win to trunk on Feb 8th.
> >>I
> >>am happy to announce that we are ready for the merge.
> >>
> >>Here is a brief recap on the highlights of the work done:
> >>- Command-line scripts for the Hadoop surface area
> >>- Mapping the HDFS permissions model to Windows
> >>- Abstracted and reconciled mismatches around differences in Path
> >>semantics
> >>in Java and Windows
> >>- Native Task Controller for Windows
> >>- Implementation of a Block Placement Policy to support cloud
> >>environments,
> >>more specifically Azure.
> >>- Implementation of Hadoop native libraries for Windows (compression
> >>codecs, native I/O)
> >>- Several reliability issues, including race-conditions, intermittent
> test
> >>failures, resource leaks.
> >>- Several new unit test cases written for the above changes
> >>
> >>Please find the details of the work in CHANGES.branch-trunk-win.txt -
> >>Common changes<http://bit.ly/Xe7Ynv>, HDFS changes<http://bit.ly/13QOSo9
> >,
> >>and YARN and MapReduce changes <http://bit.ly/128zzMt>. This is the work
> >>ported from branch-1-win to a branch based on trunk.
> >>
> >>For details of the testing done, please see the thread -
> >>http://bit.ly/WpavJ4. Merge patch for this is available on HADOOP-8562<
> >>https://issues.apache.org/jira/browse/HADOOP-8562>.
> >>
> >>This was a large undertaking that involved developing code, testing the
> >>entire Hadoop stack, including scale tests. This is made possible only
> >>with
> >>the contribution from many many folks in the community. Following people
> >>contributed to this work: Ivan Mitic, Chuan Liu, Ramya Sunil, Bikas Saha,
> >>Kanna Karanam, John Gordon, Brandon Li, Chris Nauroth, David Lao,
> Sumadhur
> >>Reddy Bolli, Arpit Agarwal, Ahmed El Baz, Mike Liddell, Jing Zhao, Thejas
> >>Nair, Steve Maine, Ganeshan Iyer, Raja Aluri, Giridharan Kesavan, Ramya
> >>Bharathi Nimmagadda, Daryn Sharp, Arun Murthy, Tsz-Wo Nicholas Sze,
> Suresh
> >>Srinivas and Sanjay Radia. There are many others who contributed as well
> >>providing feedback and comments on numerous jiras.
> >>
> >>The vote will run for seven days and will end on March 5, 6:00PM PST.
> >>
> >>Regards,
> >>Suresh
> >>
> >>
> >>
> >>
> >>On Thu, Feb 7, 2013 at 6:41 PM, Mahadevan Venkatraman
> >><ma...@microsoft.com>wrote:
> >>
> >>> It is super exciting to look at the prospect of these changes being
> >>>merged
> >>> to trunk. Having Windows as one of the supported Hadoop platforms is a
> >>> fantastic opportunity both for the Hadoop project and Microsoft
> >>>customers.
> >>>
> >>> This work began around a year back when a few of us started with a
> basic
> >>> port of Hadoop on Windows. Ever since, the Hadoop team in Microsoft
> have
> >>> made significant progress in the following areas:
> >>> (PS: Some of these items are already included in Suresh's email, but
> >>> including again for completeness)
> >>>
> >>> - Command-line scripts for the Hadoop surface area
> >>> - Mapping the HDFS permissions model to Windows
> >>> - Abstracted and reconciled mismatches around differences in Path
> >>> semantics in Java and Windows
> >>> - Native Task Controller for Windows
> >>> - Implementation of a Block Placement Policy to support cloud
> >>> environments, more specifically Azure.
> >>> - Implementation of Hadoop native libraries for Windows (compression
> >>> codecs, native I/O) - Several reliability issues, including
> >>> race-conditions, intermittent test failures, resource leaks.
> >>> - Several new unit test cases written for the above changes
> >>>
> >>> In the process, we have closely engaged with the Apache open source
> >>> community and have got great support and assistance from the community
> >>>in
> >>> terms of contributing fixes, code review comments and commits.
> >>>
> >>> In addition, the Hadoop team at Microsoft has also made good progress
> in
> >>> other projects including Hive, Pig, Sqoop, Oozie, HCat and HBase. Many
> >>>of
> >>> these changes have already been committed to the respective trunks with
> >>> help from various committers and contributors. It is great to see the
> >>> commitment of the community to support multiple platforms, and we look
> >>> forward to the day when a developer/customer is able to successfully
> >>>deploy
> >>> a complete solution stack based on Apache Hadoop releases.
> >>>
> >>> Next Steps:
> >>>
> >>> All of the above changes are part of the Windows Azure HDInsight and
> >>> HDInsight Server products from Microsoft. We have successfully
> >>>on-boarded
> >>> several internal customers and have been running production workloads
> on
> >>> Windows Azure HDInsight. Our vision is to create a big data platform
> >>>based
> >>> on Hadoop, and we are committed to helping make Hadoop a world-class
> >>> solution that anyone can use to solve their biggest data challenges.
> >>>
> >>> As an immediate next step, we would like to have a discussion around
> how
> >>> we can ensure that the quality of the mainline Hadoop branches on
> >>>Windows
> >>> is maintained. To this end, we would like to get to the state where we
> >>>have
> >>> pre-checkin validation gates and nightly test runs enabled on Windows.
> >>>If
> >>> you have any suggestions around this, please do send an email.  We are
> >>> committed to helping sustain the long-term quality of Hadoop on both
> >>>Linux
> >>> and Windows.
> >>>
> >>> We sincerely thank the community for their contribution and support so
> >>> far. And hope to continue having a close engagement in the future.
> >>>
> >>> -Microsoft HDInsight Team
> >>>
> >>>
> >>> -----Original Message-----
> >>> From: Suresh Srinivas [mailto:suresh@hortonworks.com]
> >>> Sent: Thursday, February 7, 2013 5:42 PM
> >>> To: common-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org;
> >>> hdfs-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
> >>> Subject: Heads up - merge branch-trunk-win to trunk
> >>>
> >>> The support for Hadoop on Windows was proposed in HADOOP-8079<
> >>> https://issues.apache.org/jira/browse/HADOOP-8079> almost a year ago.
> >>>The
> >>> goal was to make Hadoop natively integrated, full-featured, and
> >>>performance
> >>> and scalability tuned on Windows Server or Windows Azure.
> >>> We are happy to announce that a lot of progress has been made in this
> >>> regard.
> >>>
> >>> Initial work started in a feature branch, branch-1-win, based on
> >>>branch-1.
> >>> The details related to the work done in the branch can be seen in
> >>> CHANGES.txt<
> >>>
> >>>
> http://svn.apache.org/viewvc/hadoop/common/branches/branch-1-win/CHANGES.
> >>>branch-1-win.txt?view=markup
> >>> >.
> >>> This work has been ported to a branch, branch-trunk-win, based on
> trunk.
> >>> Merge patch for this is available on
> >>> HADOOP-8562<https://issues.apache.org/jira/browse/HADOOP-8562>
> >>> .
> >>>
> >>> Highlights of the work done so far:
> >>> 1. Necessary changes in Hadoop to run natively on Windows. These
> changes
> >>> handle differences in platforms related to path names, process/task
> >>> management etc.
> >>> 2. Addition of winutils tools for managing file permissions and
> >>>ownership,
> >>> user group mapping, hardlinks, symbolic links, chmod, disk utilization,
> >>>and
> >>> process/task management.
> >>> 3. Added cmd scripts equivalent to existing shell scripts
> >>> hadoop-daemon.sh, start and stop scripts.
> >>> 4. Addition of block placement policy implemnation to support cloud
> >>> enviroment, more specifically Azure.
> >>>
> >>> We are very close to wrapping up the work in branch-trunk-win and
> >>>getting
> >>> ready for a merge. Currently the merge patch is passing close to 100%
> of
> >>> unit tests on Linux. Soon I will call for a vote to merge this branch
> >>>into
> >>> trunk.
> >>>
> >>> Next steps:
> >>> 1. Call for vote to merge branch-trunk-win to trunk, when the work
> >>> completes and precommit build is clean.
> >>> 2. Start a discussion on adding Jenkins precommit builds on windows and
> >>> how to integrate that with the existing commit process.
> >>>
> >>> Let me know if you have any questions.
> >>>
> >>> Regards,
> >>> Suresh
> >>>
> >>>
> >>
> >>
> >>--
> >>http://hortonworks.com/download/
> >
>



-- 
http://hortonworks.com/download/

Re: [Vote] Merge branch-trunk-win to trunk

[Suresh Srinivas <su...@hortonworks.com>]

Thanks for raising good questions.

Currently the merge patch passes all the tests on Linux, hence
the proposal for merging the patch to trunk. But as Bobby, Harsh
and Eli pointed out, before declaring support for Windows, we need the
discussion on the following:

1. Precommit and development process
Jenkins infrastructure for Windows build will be made available.
Giri and Microsoft contributors have volunteered to help make
this happen.

With that we need to decide how our precommit process looks.
My inclination is to wait for +1 from precommit builds on
both the platforms to ensure no issues are introduced.
Thoughts?

2. Feature development impact
Some questions have been raised about would new features
need to be supported on both the platforms. Yes. I do not see a
reason why features cannot work on both the platforms, with
the exception of platform specific optimizations. This what Java
gives us.

3. Platform specific features/optimizations
As regards platform specific optimization, each platform can
evolve at its own pace and should not block progress of a
specific platform.

As indicated in my earlier email, there is a sizable number
of contributors to work on issues and support of Hadoop on Windows
platform. I am excited to see Hadoop reach the other large
part of server market.

Eli, as pointed out by you, the TODO items need to be addressed.
Also we realized we still need to add information on how to
build on Windows in BUILDING.txt. We will address this ASAP.
Giri and Matt have some expirience with this and should be able
to provide more information.



On Wed, Feb 27, 2013 at 11:56 AM, Eli Collins <el...@cloudera.com> wrote:

> Bobby raises some good questions.  A related one, since most current
> developers won't add Windows support for new features that are
> platform specific is it assumed that Windows development will either
> lag or will people actively work on keeping Windows up with the
> latest?  And vice versa in case Windows support is implemented first.
>
> Is there a jira for resolving the outstanding TODOs in the code base
> (similar to HDFS-2148)?  Looks like this merge doesn't introduce many
> which is great (just did a quick diff and grep).
>
> Thanks,
> Eli
>
> On Wed, Feb 27, 2013 at 8:17 AM, Robert Evans <ev...@yahoo-inc.com> wrote:
> > After this is merged in is Windows still going to be a second class
> > citizen but happens to work for more than just development or is it a
> > fully supported platform where if something breaks it can block a
> release?
> >  How do we as a community intend to keep Windows support from breaking?
> > We don't have any Jenkins slaves to be able to run nightly tests to
> > validate everything still compiles/runs.  This is not a blocker for me
> > because we often rely on individuals and groups to test Hadoop, but I do
> > think we need to have this discussion before we put it in.
> >
> > --Bobby
> >
> > On 2/26/13 4:55 PM, "Suresh Srinivas" <su...@hortonworks.com> wrote:
> >
> >>I had posted heads up about merging branch-trunk-win to trunk on Feb 8th.
> >>I
> >>am happy to announce that we are ready for the merge.
> >>
> >>Here is a brief recap on the highlights of the work done:
> >>- Command-line scripts for the Hadoop surface area
> >>- Mapping the HDFS permissions model to Windows
> >>- Abstracted and reconciled mismatches around differences in Path
> >>semantics
> >>in Java and Windows
> >>- Native Task Controller for Windows
> >>- Implementation of a Block Placement Policy to support cloud
> >>environments,
> >>more specifically Azure.
> >>- Implementation of Hadoop native libraries for Windows (compression
> >>codecs, native I/O)
> >>- Several reliability issues, including race-conditions, intermittent
> test
> >>failures, resource leaks.
> >>- Several new unit test cases written for the above changes
> >>
> >>Please find the details of the work in CHANGES.branch-trunk-win.txt -
> >>Common changes<http://bit.ly/Xe7Ynv>, HDFS changes<http://bit.ly/13QOSo9
> >,
> >>and YARN and MapReduce changes <http://bit.ly/128zzMt>. This is the work
> >>ported from branch-1-win to a branch based on trunk.
> >>
> >>For details of the testing done, please see the thread -
> >>http://bit.ly/WpavJ4. Merge patch for this is available on HADOOP-8562<
> >>https://issues.apache.org/jira/browse/HADOOP-8562>.
> >>
> >>This was a large undertaking that involved developing code, testing the
> >>entire Hadoop stack, including scale tests. This is made possible only
> >>with
> >>the contribution from many many folks in the community. Following people
> >>contributed to this work: Ivan Mitic, Chuan Liu, Ramya Sunil, Bikas Saha,
> >>Kanna Karanam, John Gordon, Brandon Li, Chris Nauroth, David Lao,
> Sumadhur
> >>Reddy Bolli, Arpit Agarwal, Ahmed El Baz, Mike Liddell, Jing Zhao, Thejas
> >>Nair, Steve Maine, Ganeshan Iyer, Raja Aluri, Giridharan Kesavan, Ramya
> >>Bharathi Nimmagadda, Daryn Sharp, Arun Murthy, Tsz-Wo Nicholas Sze,
> Suresh
> >>Srinivas and Sanjay Radia. There are many others who contributed as well
> >>providing feedback and comments on numerous jiras.
> >>
> >>The vote will run for seven days and will end on March 5, 6:00PM PST.
> >>
> >>Regards,
> >>Suresh
> >>
> >>
> >>
> >>
> >>On Thu, Feb 7, 2013 at 6:41 PM, Mahadevan Venkatraman
> >><ma...@microsoft.com>wrote:
> >>
> >>> It is super exciting to look at the prospect of these changes being
> >>>merged
> >>> to trunk. Having Windows as one of the supported Hadoop platforms is a
> >>> fantastic opportunity both for the Hadoop project and Microsoft
> >>>customers.
> >>>
> >>> This work began around a year back when a few of us started with a
> basic
> >>> port of Hadoop on Windows. Ever since, the Hadoop team in Microsoft
> have
> >>> made significant progress in the following areas:
> >>> (PS: Some of these items are already included in Suresh's email, but
> >>> including again for completeness)
> >>>
> >>> - Command-line scripts for the Hadoop surface area
> >>> - Mapping the HDFS permissions model to Windows
> >>> - Abstracted and reconciled mismatches around differences in Path
> >>> semantics in Java and Windows
> >>> - Native Task Controller for Windows
> >>> - Implementation of a Block Placement Policy to support cloud
> >>> environments, more specifically Azure.
> >>> - Implementation of Hadoop native libraries for Windows (compression
> >>> codecs, native I/O) - Several reliability issues, including
> >>> race-conditions, intermittent test failures, resource leaks.
> >>> - Several new unit test cases written for the above changes
> >>>
> >>> In the process, we have closely engaged with the Apache open source
> >>> community and have got great support and assistance from the community
> >>>in
> >>> terms of contributing fixes, code review comments and commits.
> >>>
> >>> In addition, the Hadoop team at Microsoft has also made good progress
> in
> >>> other projects including Hive, Pig, Sqoop, Oozie, HCat and HBase. Many
> >>>of
> >>> these changes have already been committed to the respective trunks with
> >>> help from various committers and contributors. It is great to see the
> >>> commitment of the community to support multiple platforms, and we look
> >>> forward to the day when a developer/customer is able to successfully
> >>>deploy
> >>> a complete solution stack based on Apache Hadoop releases.
> >>>
> >>> Next Steps:
> >>>
> >>> All of the above changes are part of the Windows Azure HDInsight and
> >>> HDInsight Server products from Microsoft. We have successfully
> >>>on-boarded
> >>> several internal customers and have been running production workloads
> on
> >>> Windows Azure HDInsight. Our vision is to create a big data platform
> >>>based
> >>> on Hadoop, and we are committed to helping make Hadoop a world-class
> >>> solution that anyone can use to solve their biggest data challenges.
> >>>
> >>> As an immediate next step, we would like to have a discussion around
> how
> >>> we can ensure that the quality of the mainline Hadoop branches on
> >>>Windows
> >>> is maintained. To this end, we would like to get to the state where we
> >>>have
> >>> pre-checkin validation gates and nightly test runs enabled on Windows.
> >>>If
> >>> you have any suggestions around this, please do send an email.  We are
> >>> committed to helping sustain the long-term quality of Hadoop on both
> >>>Linux
> >>> and Windows.
> >>>
> >>> We sincerely thank the community for their contribution and support so
> >>> far. And hope to continue having a close engagement in the future.
> >>>
> >>> -Microsoft HDInsight Team
> >>>
> >>>
> >>> -----Original Message-----
> >>> From: Suresh Srinivas [mailto:suresh@hortonworks.com]
> >>> Sent: Thursday, February 7, 2013 5:42 PM
> >>> To: common-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org;
> >>> hdfs-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
> >>> Subject: Heads up - merge branch-trunk-win to trunk
> >>>
> >>> The support for Hadoop on Windows was proposed in HADOOP-8079<
> >>> https://issues.apache.org/jira/browse/HADOOP-8079> almost a year ago.
> >>>The
> >>> goal was to make Hadoop natively integrated, full-featured, and
> >>>performance
> >>> and scalability tuned on Windows Server or Windows Azure.
> >>> We are happy to announce that a lot of progress has been made in this
> >>> regard.
> >>>
> >>> Initial work started in a feature branch, branch-1-win, based on
> >>>branch-1.
> >>> The details related to the work done in the branch can be seen in
> >>> CHANGES.txt<
> >>>
> >>>
> http://svn.apache.org/viewvc/hadoop/common/branches/branch-1-win/CHANGES.
> >>>branch-1-win.txt?view=markup
> >>> >.
> >>> This work has been ported to a branch, branch-trunk-win, based on
> trunk.
> >>> Merge patch for this is available on
> >>> HADOOP-8562<https://issues.apache.org/jira/browse/HADOOP-8562>
> >>> .
> >>>
> >>> Highlights of the work done so far:
> >>> 1. Necessary changes in Hadoop to run natively on Windows. These
> changes
> >>> handle differences in platforms related to path names, process/task
> >>> management etc.
> >>> 2. Addition of winutils tools for managing file permissions and
> >>>ownership,
> >>> user group mapping, hardlinks, symbolic links, chmod, disk utilization,
> >>>and
> >>> process/task management.
> >>> 3. Added cmd scripts equivalent to existing shell scripts
> >>> hadoop-daemon.sh, start and stop scripts.
> >>> 4. Addition of block placement policy implemnation to support cloud
> >>> enviroment, more specifically Azure.
> >>>
> >>> We are very close to wrapping up the work in branch-trunk-win and
> >>>getting
> >>> ready for a merge. Currently the merge patch is passing close to 100%
> of
> >>> unit tests on Linux. Soon I will call for a vote to merge this branch
> >>>into
> >>> trunk.
> >>>
> >>> Next steps:
> >>> 1. Call for vote to merge branch-trunk-win to trunk, when the work
> >>> completes and precommit build is clean.
> >>> 2. Start a discussion on adding Jenkins precommit builds on windows and
> >>> how to integrate that with the existing commit process.
> >>>
> >>> Let me know if you have any questions.
> >>>
> >>> Regards,
> >>> Suresh
> >>>
> >>>
> >>
> >>
> >>--
> >>http://hortonworks.com/download/
> >
>



-- 
http://hortonworks.com/download/

Re: [Vote] Merge branch-trunk-win to trunk

[Arpit Agarwal <aa...@hortonworks.com>]

+1 non-binding.

I have extensively tested this on both Windows and Linux over the last few
months.

Thanks,
-Arpit

On Wed, Feb 27, 2013 at 11:56 AM, Eli Collins <el...@cloudera.com> wrote:

> Bobby raises some good questions.  A related one, since most current
> developers won't add Windows support for new features that are
> platform specific is it assumed that Windows development will either
> lag or will people actively work on keeping Windows up with the
> latest?  And vice versa in case Windows support is implemented first.
>
> Is there a jira for resolving the outstanding TODOs in the code base
> (similar to HDFS-2148)?  Looks like this merge doesn't introduce many
> which is great (just did a quick diff and grep).
>
> Thanks,
> Eli
>
> On Wed, Feb 27, 2013 at 8:17 AM, Robert Evans <ev...@yahoo-inc.com> wrote:
> > After this is merged in is Windows still going to be a second class
> > citizen but happens to work for more than just development or is it a
> > fully supported platform where if something breaks it can block a
> release?
> >  How do we as a community intend to keep Windows support from breaking?
> > We don't have any Jenkins slaves to be able to run nightly tests to
> > validate everything still compiles/runs.  This is not a blocker for me
> > because we often rely on individuals and groups to test Hadoop, but I do
> > think we need to have this discussion before we put it in.
> >
> > --Bobby
> >
> > On 2/26/13 4:55 PM, "Suresh Srinivas" <su...@hortonworks.com> wrote:
> >
> >>I had posted heads up about merging branch-trunk-win to trunk on Feb 8th.
> >>I
> >>am happy to announce that we are ready for the merge.
> >>
> >>Here is a brief recap on the highlights of the work done:
> >>- Command-line scripts for the Hadoop surface area
> >>- Mapping the HDFS permissions model to Windows
> >>- Abstracted and reconciled mismatches around differences in Path
> >>semantics
> >>in Java and Windows
> >>- Native Task Controller for Windows
> >>- Implementation of a Block Placement Policy to support cloud
> >>environments,
> >>more specifically Azure.
> >>- Implementation of Hadoop native libraries for Windows (compression
> >>codecs, native I/O)
> >>- Several reliability issues, including race-conditions, intermittent
> test
> >>failures, resource leaks.
> >>- Several new unit test cases written for the above changes
> >>
> >>Please find the details of the work in CHANGES.branch-trunk-win.txt -
> >>Common changes<http://bit.ly/Xe7Ynv>, HDFS changes<http://bit.ly/13QOSo9
> >,
> >>and YARN and MapReduce changes <http://bit.ly/128zzMt>. This is the work
> >>ported from branch-1-win to a branch based on trunk.
> >>
> >>For details of the testing done, please see the thread -
> >>http://bit.ly/WpavJ4. Merge patch for this is available on HADOOP-8562<
> >>https://issues.apache.org/jira/browse/HADOOP-8562>.
> >>
> >>This was a large undertaking that involved developing code, testing the
> >>entire Hadoop stack, including scale tests. This is made possible only
> >>with
> >>the contribution from many many folks in the community. Following people
> >>contributed to this work: Ivan Mitic, Chuan Liu, Ramya Sunil, Bikas Saha,
> >>Kanna Karanam, John Gordon, Brandon Li, Chris Nauroth, David Lao,
> Sumadhur
> >>Reddy Bolli, Arpit Agarwal, Ahmed El Baz, Mike Liddell, Jing Zhao, Thejas
> >>Nair, Steve Maine, Ganeshan Iyer, Raja Aluri, Giridharan Kesavan, Ramya
> >>Bharathi Nimmagadda, Daryn Sharp, Arun Murthy, Tsz-Wo Nicholas Sze,
> Suresh
> >>Srinivas and Sanjay Radia. There are many others who contributed as well
> >>providing feedback and comments on numerous jiras.
> >>
> >>The vote will run for seven days and will end on March 5, 6:00PM PST.
> >>
> >>Regards,
> >>Suresh
> >>
> >>
> >>
> >>
> >>On Thu, Feb 7, 2013 at 6:41 PM, Mahadevan Venkatraman
> >><ma...@microsoft.com>wrote:
> >>
> >>> It is super exciting to look at the prospect of these changes being
> >>>merged
> >>> to trunk. Having Windows as one of the supported Hadoop platforms is a
> >>> fantastic opportunity both for the Hadoop project and Microsoft
> >>>customers.
> >>>
> >>> This work began around a year back when a few of us started with a
> basic
> >>> port of Hadoop on Windows. Ever since, the Hadoop team in Microsoft
> have
> >>> made significant progress in the following areas:
> >>> (PS: Some of these items are already included in Suresh's email, but
> >>> including again for completeness)
> >>>
> >>> - Command-line scripts for the Hadoop surface area
> >>> - Mapping the HDFS permissions model to Windows
> >>> - Abstracted and reconciled mismatches around differences in Path
> >>> semantics in Java and Windows
> >>> - Native Task Controller for Windows
> >>> - Implementation of a Block Placement Policy to support cloud
> >>> environments, more specifically Azure.
> >>> - Implementation of Hadoop native libraries for Windows (compression
> >>> codecs, native I/O) - Several reliability issues, including
> >>> race-conditions, intermittent test failures, resource leaks.
> >>> - Several new unit test cases written for the above changes
> >>>
> >>> In the process, we have closely engaged with the Apache open source
> >>> community and have got great support and assistance from the community
> >>>in
> >>> terms of contributing fixes, code review comments and commits.
> >>>
> >>> In addition, the Hadoop team at Microsoft has also made good progress
> in
> >>> other projects including Hive, Pig, Sqoop, Oozie, HCat and HBase. Many
> >>>of
> >>> these changes have already been committed to the respective trunks with
> >>> help from various committers and contributors. It is great to see the
> >>> commitment of the community to support multiple platforms, and we look
> >>> forward to the day when a developer/customer is able to successfully
> >>>deploy
> >>> a complete solution stack based on Apache Hadoop releases.
> >>>
> >>> Next Steps:
> >>>
> >>> All of the above changes are part of the Windows Azure HDInsight and
> >>> HDInsight Server products from Microsoft. We have successfully
> >>>on-boarded
> >>> several internal customers and have been running production workloads
> on
> >>> Windows Azure HDInsight. Our vision is to create a big data platform
> >>>based
> >>> on Hadoop, and we are committed to helping make Hadoop a world-class
> >>> solution that anyone can use to solve their biggest data challenges.
> >>>
> >>> As an immediate next step, we would like to have a discussion around
> how
> >>> we can ensure that the quality of the mainline Hadoop branches on
> >>>Windows
> >>> is maintained. To this end, we would like to get to the state where we
> >>>have
> >>> pre-checkin validation gates and nightly test runs enabled on Windows.
> >>>If
> >>> you have any suggestions around this, please do send an email.  We are
> >>> committed to helping sustain the long-term quality of Hadoop on both
> >>>Linux
> >>> and Windows.
> >>>
> >>> We sincerely thank the community for their contribution and support so
> >>> far. And hope to continue having a close engagement in the future.
> >>>
> >>> -Microsoft HDInsight Team
> >>>
> >>>
> >>> -----Original Message-----
> >>> From: Suresh Srinivas [mailto:suresh@hortonworks.com]
> >>> Sent: Thursday, February 7, 2013 5:42 PM
> >>> To: common-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org;
> >>> hdfs-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
> >>> Subject: Heads up - merge branch-trunk-win to trunk
> >>>
> >>> The support for Hadoop on Windows was proposed in HADOOP-8079<
> >>> https://issues.apache.org/jira/browse/HADOOP-8079> almost a year ago.
> >>>The
> >>> goal was to make Hadoop natively integrated, full-featured, and
> >>>performance
> >>> and scalability tuned on Windows Server or Windows Azure.
> >>> We are happy to announce that a lot of progress has been made in this
> >>> regard.
> >>>
> >>> Initial work started in a feature branch, branch-1-win, based on
> >>>branch-1.
> >>> The details related to the work done in the branch can be seen in
> >>> CHANGES.txt<
> >>>
> >>>
> http://svn.apache.org/viewvc/hadoop/common/branches/branch-1-win/CHANGES.
> >>>branch-1-win.txt?view=markup
> >>> >.
> >>> This work has been ported to a branch, branch-trunk-win, based on
> trunk.
> >>> Merge patch for this is available on
> >>> HADOOP-8562<https://issues.apache.org/jira/browse/HADOOP-8562>
> >>> .
> >>>
> >>> Highlights of the work done so far:
> >>> 1. Necessary changes in Hadoop to run natively on Windows. These
> changes
> >>> handle differences in platforms related to path names, process/task
> >>> management etc.
> >>> 2. Addition of winutils tools for managing file permissions and
> >>>ownership,
> >>> user group mapping, hardlinks, symbolic links, chmod, disk utilization,
> >>>and
> >>> process/task management.
> >>> 3. Added cmd scripts equivalent to existing shell scripts
> >>> hadoop-daemon.sh, start and stop scripts.
> >>> 4. Addition of block placement policy implemnation to support cloud
> >>> enviroment, more specifically Azure.
> >>>
> >>> We are very close to wrapping up the work in branch-trunk-win and
> >>>getting
> >>> ready for a merge. Currently the merge patch is passing close to 100%
> of
> >>> unit tests on Linux. Soon I will call for a vote to merge this branch
> >>>into
> >>> trunk.
> >>>
> >>> Next steps:
> >>> 1. Call for vote to merge branch-trunk-win to trunk, when the work
> >>> completes and precommit build is clean.
> >>> 2. Start a discussion on adding Jenkins precommit builds on windows and
> >>> how to integrate that with the existing commit process.
> >>>
> >>> Let me know if you have any questions.
> >>>
> >>> Regards,
> >>> Suresh
> >>>
> >>>
> >>
> >>
> >>--
> >>http://hortonworks.com/download/
> >
>

Re: [Vote] Merge branch-trunk-win to trunk

[Suresh Srinivas <su...@hortonworks.com>]

Thanks for raising good questions.

Currently the merge patch passes all the tests on Linux, hence
the proposal for merging the patch to trunk. But as Bobby, Harsh
and Eli pointed out, before declaring support for Windows, we need the
discussion on the following:

1. Precommit and development process
Jenkins infrastructure for Windows build will be made available.
Giri and Microsoft contributors have volunteered to help make
this happen.

With that we need to decide how our precommit process looks.
My inclination is to wait for +1 from precommit builds on
both the platforms to ensure no issues are introduced.
Thoughts?

2. Feature development impact
Some questions have been raised about would new features
need to be supported on both the platforms. Yes. I do not see a
reason why features cannot work on both the platforms, with
the exception of platform specific optimizations. This what Java
gives us.

3. Platform specific features/optimizations
As regards platform specific optimization, each platform can
evolve at its own pace and should not block progress of a
specific platform.

As indicated in my earlier email, there is a sizable number
of contributors to work on issues and support of Hadoop on Windows
platform. I am excited to see Hadoop reach the other large
part of server market.

Eli, as pointed out by you, the TODO items need to be addressed.
Also we realized we still need to add information on how to
build on Windows in BUILDING.txt. We will address this ASAP.
Giri and Matt have some expirience with this and should be able
to provide more information.



On Wed, Feb 27, 2013 at 11:56 AM, Eli Collins <el...@cloudera.com> wrote:

> Bobby raises some good questions.  A related one, since most current
> developers won't add Windows support for new features that are
> platform specific is it assumed that Windows development will either
> lag or will people actively work on keeping Windows up with the
> latest?  And vice versa in case Windows support is implemented first.
>
> Is there a jira for resolving the outstanding TODOs in the code base
> (similar to HDFS-2148)?  Looks like this merge doesn't introduce many
> which is great (just did a quick diff and grep).
>
> Thanks,
> Eli
>
> On Wed, Feb 27, 2013 at 8:17 AM, Robert Evans <ev...@yahoo-inc.com> wrote:
> > After this is merged in is Windows still going to be a second class
> > citizen but happens to work for more than just development or is it a
> > fully supported platform where if something breaks it can block a
> release?
> >  How do we as a community intend to keep Windows support from breaking?
> > We don't have any Jenkins slaves to be able to run nightly tests to
> > validate everything still compiles/runs.  This is not a blocker for me
> > because we often rely on individuals and groups to test Hadoop, but I do
> > think we need to have this discussion before we put it in.
> >
> > --Bobby
> >
> > On 2/26/13 4:55 PM, "Suresh Srinivas" <su...@hortonworks.com> wrote:
> >
> >>I had posted heads up about merging branch-trunk-win to trunk on Feb 8th.
> >>I
> >>am happy to announce that we are ready for the merge.
> >>
> >>Here is a brief recap on the highlights of the work done:
> >>- Command-line scripts for the Hadoop surface area
> >>- Mapping the HDFS permissions model to Windows
> >>- Abstracted and reconciled mismatches around differences in Path
> >>semantics
> >>in Java and Windows
> >>- Native Task Controller for Windows
> >>- Implementation of a Block Placement Policy to support cloud
> >>environments,
> >>more specifically Azure.
> >>- Implementation of Hadoop native libraries for Windows (compression
> >>codecs, native I/O)
> >>- Several reliability issues, including race-conditions, intermittent
> test
> >>failures, resource leaks.
> >>- Several new unit test cases written for the above changes
> >>
> >>Please find the details of the work in CHANGES.branch-trunk-win.txt -
> >>Common changes<http://bit.ly/Xe7Ynv>, HDFS changes<http://bit.ly/13QOSo9
> >,
> >>and YARN and MapReduce changes <http://bit.ly/128zzMt>. This is the work
> >>ported from branch-1-win to a branch based on trunk.
> >>
> >>For details of the testing done, please see the thread -
> >>http://bit.ly/WpavJ4. Merge patch for this is available on HADOOP-8562<
> >>https://issues.apache.org/jira/browse/HADOOP-8562>.
> >>
> >>This was a large undertaking that involved developing code, testing the
> >>entire Hadoop stack, including scale tests. This is made possible only
> >>with
> >>the contribution from many many folks in the community. Following people
> >>contributed to this work: Ivan Mitic, Chuan Liu, Ramya Sunil, Bikas Saha,
> >>Kanna Karanam, John Gordon, Brandon Li, Chris Nauroth, David Lao,
> Sumadhur
> >>Reddy Bolli, Arpit Agarwal, Ahmed El Baz, Mike Liddell, Jing Zhao, Thejas
> >>Nair, Steve Maine, Ganeshan Iyer, Raja Aluri, Giridharan Kesavan, Ramya
> >>Bharathi Nimmagadda, Daryn Sharp, Arun Murthy, Tsz-Wo Nicholas Sze,
> Suresh
> >>Srinivas and Sanjay Radia. There are many others who contributed as well
> >>providing feedback and comments on numerous jiras.
> >>
> >>The vote will run for seven days and will end on March 5, 6:00PM PST.
> >>
> >>Regards,
> >>Suresh
> >>
> >>
> >>
> >>
> >>On Thu, Feb 7, 2013 at 6:41 PM, Mahadevan Venkatraman
> >><ma...@microsoft.com>wrote:
> >>
> >>> It is super exciting to look at the prospect of these changes being
> >>>merged
> >>> to trunk. Having Windows as one of the supported Hadoop platforms is a
> >>> fantastic opportunity both for the Hadoop project and Microsoft
> >>>customers.
> >>>
> >>> This work began around a year back when a few of us started with a
> basic
> >>> port of Hadoop on Windows. Ever since, the Hadoop team in Microsoft
> have
> >>> made significant progress in the following areas:
> >>> (PS: Some of these items are already included in Suresh's email, but
> >>> including again for completeness)
> >>>
> >>> - Command-line scripts for the Hadoop surface area
> >>> - Mapping the HDFS permissions model to Windows
> >>> - Abstracted and reconciled mismatches around differences in Path
> >>> semantics in Java and Windows
> >>> - Native Task Controller for Windows
> >>> - Implementation of a Block Placement Policy to support cloud
> >>> environments, more specifically Azure.
> >>> - Implementation of Hadoop native libraries for Windows (compression
> >>> codecs, native I/O) - Several reliability issues, including
> >>> race-conditions, intermittent test failures, resource leaks.
> >>> - Several new unit test cases written for the above changes
> >>>
> >>> In the process, we have closely engaged with the Apache open source
> >>> community and have got great support and assistance from the community
> >>>in
> >>> terms of contributing fixes, code review comments and commits.
> >>>
> >>> In addition, the Hadoop team at Microsoft has also made good progress
> in
> >>> other projects including Hive, Pig, Sqoop, Oozie, HCat and HBase. Many
> >>>of
> >>> these changes have already been committed to the respective trunks with
> >>> help from various committers and contributors. It is great to see the
> >>> commitment of the community to support multiple platforms, and we look
> >>> forward to the day when a developer/customer is able to successfully
> >>>deploy
> >>> a complete solution stack based on Apache Hadoop releases.
> >>>
> >>> Next Steps:
> >>>
> >>> All of the above changes are part of the Windows Azure HDInsight and
> >>> HDInsight Server products from Microsoft. We have successfully
> >>>on-boarded
> >>> several internal customers and have been running production workloads
> on
> >>> Windows Azure HDInsight. Our vision is to create a big data platform
> >>>based
> >>> on Hadoop, and we are committed to helping make Hadoop a world-class
> >>> solution that anyone can use to solve their biggest data challenges.
> >>>
> >>> As an immediate next step, we would like to have a discussion around
> how
> >>> we can ensure that the quality of the mainline Hadoop branches on
> >>>Windows
> >>> is maintained. To this end, we would like to get to the state where we
> >>>have
> >>> pre-checkin validation gates and nightly test runs enabled on Windows.
> >>>If
> >>> you have any suggestions around this, please do send an email.  We are
> >>> committed to helping sustain the long-term quality of Hadoop on both
> >>>Linux
> >>> and Windows.
> >>>
> >>> We sincerely thank the community for their contribution and support so
> >>> far. And hope to continue having a close engagement in the future.
> >>>
> >>> -Microsoft HDInsight Team
> >>>
> >>>
> >>> -----Original Message-----
> >>> From: Suresh Srinivas [mailto:suresh@hortonworks.com]
> >>> Sent: Thursday, February 7, 2013 5:42 PM
> >>> To: common-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org;
> >>> hdfs-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
> >>> Subject: Heads up - merge branch-trunk-win to trunk
> >>>
> >>> The support for Hadoop on Windows was proposed in HADOOP-8079<
> >>> https://issues.apache.org/jira/browse/HADOOP-8079> almost a year ago.
> >>>The
> >>> goal was to make Hadoop natively integrated, full-featured, and
> >>>performance
> >>> and scalability tuned on Windows Server or Windows Azure.
> >>> We are happy to announce that a lot of progress has been made in this
> >>> regard.
> >>>
> >>> Initial work started in a feature branch, branch-1-win, based on
> >>>branch-1.
> >>> The details related to the work done in the branch can be seen in
> >>> CHANGES.txt<
> >>>
> >>>
> http://svn.apache.org/viewvc/hadoop/common/branches/branch-1-win/CHANGES.
> >>>branch-1-win.txt?view=markup
> >>> >.
> >>> This work has been ported to a branch, branch-trunk-win, based on
> trunk.
> >>> Merge patch for this is available on
> >>> HADOOP-8562<https://issues.apache.org/jira/browse/HADOOP-8562>
> >>> .
> >>>
> >>> Highlights of the work done so far:
> >>> 1. Necessary changes in Hadoop to run natively on Windows. These
> changes
> >>> handle differences in platforms related to path names, process/task
> >>> management etc.
> >>> 2. Addition of winutils tools for managing file permissions and
> >>>ownership,
> >>> user group mapping, hardlinks, symbolic links, chmod, disk utilization,
> >>>and
> >>> process/task management.
> >>> 3. Added cmd scripts equivalent to existing shell scripts
> >>> hadoop-daemon.sh, start and stop scripts.
> >>> 4. Addition of block placement policy implemnation to support cloud
> >>> enviroment, more specifically Azure.
> >>>
> >>> We are very close to wrapping up the work in branch-trunk-win and
> >>>getting
> >>> ready for a merge. Currently the merge patch is passing close to 100%
> of
> >>> unit tests on Linux. Soon I will call for a vote to merge this branch
> >>>into
> >>> trunk.
> >>>
> >>> Next steps:
> >>> 1. Call for vote to merge branch-trunk-win to trunk, when the work
> >>> completes and precommit build is clean.
> >>> 2. Start a discussion on adding Jenkins precommit builds on windows and
> >>> how to integrate that with the existing commit process.
> >>>
> >>> Let me know if you have any questions.
> >>>
> >>> Regards,
> >>> Suresh
> >>>
> >>>
> >>
> >>
> >>--
> >>http://hortonworks.com/download/
> >
>



-- 
http://hortonworks.com/download/

Re: [Vote] Merge branch-trunk-win to trunk

[Arpit Agarwal <aa...@hortonworks.com>]

+1 non-binding.

I have extensively tested this on both Windows and Linux over the last few
months.

Thanks,
-Arpit

On Wed, Feb 27, 2013 at 11:56 AM, Eli Collins <el...@cloudera.com> wrote:

> Bobby raises some good questions.  A related one, since most current
> developers won't add Windows support for new features that are
> platform specific is it assumed that Windows development will either
> lag or will people actively work on keeping Windows up with the
> latest?  And vice versa in case Windows support is implemented first.
>
> Is there a jira for resolving the outstanding TODOs in the code base
> (similar to HDFS-2148)?  Looks like this merge doesn't introduce many
> which is great (just did a quick diff and grep).
>
> Thanks,
> Eli
>
> On Wed, Feb 27, 2013 at 8:17 AM, Robert Evans <ev...@yahoo-inc.com> wrote:
> > After this is merged in is Windows still going to be a second class
> > citizen but happens to work for more than just development or is it a
> > fully supported platform where if something breaks it can block a
> release?
> >  How do we as a community intend to keep Windows support from breaking?
> > We don't have any Jenkins slaves to be able to run nightly tests to
> > validate everything still compiles/runs.  This is not a blocker for me
> > because we often rely on individuals and groups to test Hadoop, but I do
> > think we need to have this discussion before we put it in.
> >
> > --Bobby
> >
> > On 2/26/13 4:55 PM, "Suresh Srinivas" <su...@hortonworks.com> wrote:
> >
> >>I had posted heads up about merging branch-trunk-win to trunk on Feb 8th.
> >>I
> >>am happy to announce that we are ready for the merge.
> >>
> >>Here is a brief recap on the highlights of the work done:
> >>- Command-line scripts for the Hadoop surface area
> >>- Mapping the HDFS permissions model to Windows
> >>- Abstracted and reconciled mismatches around differences in Path
> >>semantics
> >>in Java and Windows
> >>- Native Task Controller for Windows
> >>- Implementation of a Block Placement Policy to support cloud
> >>environments,
> >>more specifically Azure.
> >>- Implementation of Hadoop native libraries for Windows (compression
> >>codecs, native I/O)
> >>- Several reliability issues, including race-conditions, intermittent
> test
> >>failures, resource leaks.
> >>- Several new unit test cases written for the above changes
> >>
> >>Please find the details of the work in CHANGES.branch-trunk-win.txt -
> >>Common changes<http://bit.ly/Xe7Ynv>, HDFS changes<http://bit.ly/13QOSo9
> >,
> >>and YARN and MapReduce changes <http://bit.ly/128zzMt>. This is the work
> >>ported from branch-1-win to a branch based on trunk.
> >>
> >>For details of the testing done, please see the thread -
> >>http://bit.ly/WpavJ4. Merge patch for this is available on HADOOP-8562<
> >>https://issues.apache.org/jira/browse/HADOOP-8562>.
> >>
> >>This was a large undertaking that involved developing code, testing the
> >>entire Hadoop stack, including scale tests. This is made possible only
> >>with
> >>the contribution from many many folks in the community. Following people
> >>contributed to this work: Ivan Mitic, Chuan Liu, Ramya Sunil, Bikas Saha,
> >>Kanna Karanam, John Gordon, Brandon Li, Chris Nauroth, David Lao,
> Sumadhur
> >>Reddy Bolli, Arpit Agarwal, Ahmed El Baz, Mike Liddell, Jing Zhao, Thejas
> >>Nair, Steve Maine, Ganeshan Iyer, Raja Aluri, Giridharan Kesavan, Ramya
> >>Bharathi Nimmagadda, Daryn Sharp, Arun Murthy, Tsz-Wo Nicholas Sze,
> Suresh
> >>Srinivas and Sanjay Radia. There are many others who contributed as well
> >>providing feedback and comments on numerous jiras.
> >>
> >>The vote will run for seven days and will end on March 5, 6:00PM PST.
> >>
> >>Regards,
> >>Suresh
> >>
> >>
> >>
> >>
> >>On Thu, Feb 7, 2013 at 6:41 PM, Mahadevan Venkatraman
> >><ma...@microsoft.com>wrote:
> >>
> >>> It is super exciting to look at the prospect of these changes being
> >>>merged
> >>> to trunk. Having Windows as one of the supported Hadoop platforms is a
> >>> fantastic opportunity both for the Hadoop project and Microsoft
> >>>customers.
> >>>
> >>> This work began around a year back when a few of us started with a
> basic
> >>> port of Hadoop on Windows. Ever since, the Hadoop team in Microsoft
> have
> >>> made significant progress in the following areas:
> >>> (PS: Some of these items are already included in Suresh's email, but
> >>> including again for completeness)
> >>>
> >>> - Command-line scripts for the Hadoop surface area
> >>> - Mapping the HDFS permissions model to Windows
> >>> - Abstracted and reconciled mismatches around differences in Path
> >>> semantics in Java and Windows
> >>> - Native Task Controller for Windows
> >>> - Implementation of a Block Placement Policy to support cloud
> >>> environments, more specifically Azure.
> >>> - Implementation of Hadoop native libraries for Windows (compression
> >>> codecs, native I/O) - Several reliability issues, including
> >>> race-conditions, intermittent test failures, resource leaks.
> >>> - Several new unit test cases written for the above changes
> >>>
> >>> In the process, we have closely engaged with the Apache open source
> >>> community and have got great support and assistance from the community
> >>>in
> >>> terms of contributing fixes, code review comments and commits.
> >>>
> >>> In addition, the Hadoop team at Microsoft has also made good progress
> in
> >>> other projects including Hive, Pig, Sqoop, Oozie, HCat and HBase. Many
> >>>of
> >>> these changes have already been committed to the respective trunks with
> >>> help from various committers and contributors. It is great to see the
> >>> commitment of the community to support multiple platforms, and we look
> >>> forward to the day when a developer/customer is able to successfully
> >>>deploy
> >>> a complete solution stack based on Apache Hadoop releases.
> >>>
> >>> Next Steps:
> >>>
> >>> All of the above changes are part of the Windows Azure HDInsight and
> >>> HDInsight Server products from Microsoft. We have successfully
> >>>on-boarded
> >>> several internal customers and have been running production workloads
> on
> >>> Windows Azure HDInsight. Our vision is to create a big data platform
> >>>based
> >>> on Hadoop, and we are committed to helping make Hadoop a world-class
> >>> solution that anyone can use to solve their biggest data challenges.
> >>>
> >>> As an immediate next step, we would like to have a discussion around
> how
> >>> we can ensure that the quality of the mainline Hadoop branches on
> >>>Windows
> >>> is maintained. To this end, we would like to get to the state where we
> >>>have
> >>> pre-checkin validation gates and nightly test runs enabled on Windows.
> >>>If
> >>> you have any suggestions around this, please do send an email.  We are
> >>> committed to helping sustain the long-term quality of Hadoop on both
> >>>Linux
> >>> and Windows.
> >>>
> >>> We sincerely thank the community for their contribution and support so
> >>> far. And hope to continue having a close engagement in the future.
> >>>
> >>> -Microsoft HDInsight Team
> >>>
> >>>
> >>> -----Original Message-----
> >>> From: Suresh Srinivas [mailto:suresh@hortonworks.com]
> >>> Sent: Thursday, February 7, 2013 5:42 PM
> >>> To: common-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org;
> >>> hdfs-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
> >>> Subject: Heads up - merge branch-trunk-win to trunk
> >>>
> >>> The support for Hadoop on Windows was proposed in HADOOP-8079<
> >>> https://issues.apache.org/jira/browse/HADOOP-8079> almost a year ago.
> >>>The
> >>> goal was to make Hadoop natively integrated, full-featured, and
> >>>performance
> >>> and scalability tuned on Windows Server or Windows Azure.
> >>> We are happy to announce that a lot of progress has been made in this
> >>> regard.
> >>>
> >>> Initial work started in a feature branch, branch-1-win, based on
> >>>branch-1.
> >>> The details related to the work done in the branch can be seen in
> >>> CHANGES.txt<
> >>>
> >>>
> http://svn.apache.org/viewvc/hadoop/common/branches/branch-1-win/CHANGES.
> >>>branch-1-win.txt?view=markup
> >>> >.
> >>> This work has been ported to a branch, branch-trunk-win, based on
> trunk.
> >>> Merge patch for this is available on
> >>> HADOOP-8562<https://issues.apache.org/jira/browse/HADOOP-8562>
> >>> .
> >>>
> >>> Highlights of the work done so far:
> >>> 1. Necessary changes in Hadoop to run natively on Windows. These
> changes
> >>> handle differences in platforms related to path names, process/task
> >>> management etc.
> >>> 2. Addition of winutils tools for managing file permissions and
> >>>ownership,
> >>> user group mapping, hardlinks, symbolic links, chmod, disk utilization,
> >>>and
> >>> process/task management.
> >>> 3. Added cmd scripts equivalent to existing shell scripts
> >>> hadoop-daemon.sh, start and stop scripts.
> >>> 4. Addition of block placement policy implemnation to support cloud
> >>> enviroment, more specifically Azure.
> >>>
> >>> We are very close to wrapping up the work in branch-trunk-win and
> >>>getting
> >>> ready for a merge. Currently the merge patch is passing close to 100%
> of
> >>> unit tests on Linux. Soon I will call for a vote to merge this branch
> >>>into
> >>> trunk.
> >>>
> >>> Next steps:
> >>> 1. Call for vote to merge branch-trunk-win to trunk, when the work
> >>> completes and precommit build is clean.
> >>> 2. Start a discussion on adding Jenkins precommit builds on windows and
> >>> how to integrate that with the existing commit process.
> >>>
> >>> Let me know if you have any questions.
> >>>
> >>> Regards,
> >>> Suresh
> >>>
> >>>
> >>
> >>
> >>--
> >>http://hortonworks.com/download/
> >
>

Re: [Vote] Merge branch-trunk-win to trunk

[Eli Collins <el...@cloudera.com>]

Bobby raises some good questions.  A related one, since most current
developers won't add Windows support for new features that are
platform specific is it assumed that Windows development will either
lag or will people actively work on keeping Windows up with the
latest?  And vice versa in case Windows support is implemented first.

Is there a jira for resolving the outstanding TODOs in the code base
(similar to HDFS-2148)?  Looks like this merge doesn't introduce many
which is great (just did a quick diff and grep).

Thanks,
Eli

On Wed, Feb 27, 2013 at 8:17 AM, Robert Evans <ev...@yahoo-inc.com> wrote:
> After this is merged in is Windows still going to be a second class
> citizen but happens to work for more than just development or is it a
> fully supported platform where if something breaks it can block a release?
>  How do we as a community intend to keep Windows support from breaking?
> We don't have any Jenkins slaves to be able to run nightly tests to
> validate everything still compiles/runs.  This is not a blocker for me
> because we often rely on individuals and groups to test Hadoop, but I do
> think we need to have this discussion before we put it in.
>
> --Bobby
>
> On 2/26/13 4:55 PM, "Suresh Srinivas" <su...@hortonworks.com> wrote:
>
>>I had posted heads up about merging branch-trunk-win to trunk on Feb 8th.
>>I
>>am happy to announce that we are ready for the merge.
>>
>>Here is a brief recap on the highlights of the work done:
>>- Command-line scripts for the Hadoop surface area
>>- Mapping the HDFS permissions model to Windows
>>- Abstracted and reconciled mismatches around differences in Path
>>semantics
>>in Java and Windows
>>- Native Task Controller for Windows
>>- Implementation of a Block Placement Policy to support cloud
>>environments,
>>more specifically Azure.
>>- Implementation of Hadoop native libraries for Windows (compression
>>codecs, native I/O)
>>- Several reliability issues, including race-conditions, intermittent test
>>failures, resource leaks.
>>- Several new unit test cases written for the above changes
>>
>>Please find the details of the work in CHANGES.branch-trunk-win.txt -
>>Common changes<http://bit.ly/Xe7Ynv>, HDFS changes<http://bit.ly/13QOSo9>,
>>and YARN and MapReduce changes <http://bit.ly/128zzMt>. This is the work
>>ported from branch-1-win to a branch based on trunk.
>>
>>For details of the testing done, please see the thread -
>>http://bit.ly/WpavJ4. Merge patch for this is available on HADOOP-8562<
>>https://issues.apache.org/jira/browse/HADOOP-8562>.
>>
>>This was a large undertaking that involved developing code, testing the
>>entire Hadoop stack, including scale tests. This is made possible only
>>with
>>the contribution from many many folks in the community. Following people
>>contributed to this work: Ivan Mitic, Chuan Liu, Ramya Sunil, Bikas Saha,
>>Kanna Karanam, John Gordon, Brandon Li, Chris Nauroth, David Lao, Sumadhur
>>Reddy Bolli, Arpit Agarwal, Ahmed El Baz, Mike Liddell, Jing Zhao, Thejas
>>Nair, Steve Maine, Ganeshan Iyer, Raja Aluri, Giridharan Kesavan, Ramya
>>Bharathi Nimmagadda, Daryn Sharp, Arun Murthy, Tsz-Wo Nicholas Sze, Suresh
>>Srinivas and Sanjay Radia. There are many others who contributed as well
>>providing feedback and comments on numerous jiras.
>>
>>The vote will run for seven days and will end on March 5, 6:00PM PST.
>>
>>Regards,
>>Suresh
>>
>>
>>
>>
>>On Thu, Feb 7, 2013 at 6:41 PM, Mahadevan Venkatraman
>><ma...@microsoft.com>wrote:
>>
>>> It is super exciting to look at the prospect of these changes being
>>>merged
>>> to trunk. Having Windows as one of the supported Hadoop platforms is a
>>> fantastic opportunity both for the Hadoop project and Microsoft
>>>customers.
>>>
>>> This work began around a year back when a few of us started with a basic
>>> port of Hadoop on Windows. Ever since, the Hadoop team in Microsoft have
>>> made significant progress in the following areas:
>>> (PS: Some of these items are already included in Suresh's email, but
>>> including again for completeness)
>>>
>>> - Command-line scripts for the Hadoop surface area
>>> - Mapping the HDFS permissions model to Windows
>>> - Abstracted and reconciled mismatches around differences in Path
>>> semantics in Java and Windows
>>> - Native Task Controller for Windows
>>> - Implementation of a Block Placement Policy to support cloud
>>> environments, more specifically Azure.
>>> - Implementation of Hadoop native libraries for Windows (compression
>>> codecs, native I/O) - Several reliability issues, including
>>> race-conditions, intermittent test failures, resource leaks.
>>> - Several new unit test cases written for the above changes
>>>
>>> In the process, we have closely engaged with the Apache open source
>>> community and have got great support and assistance from the community
>>>in
>>> terms of contributing fixes, code review comments and commits.
>>>
>>> In addition, the Hadoop team at Microsoft has also made good progress in
>>> other projects including Hive, Pig, Sqoop, Oozie, HCat and HBase. Many
>>>of
>>> these changes have already been committed to the respective trunks with
>>> help from various committers and contributors. It is great to see the
>>> commitment of the community to support multiple platforms, and we look
>>> forward to the day when a developer/customer is able to successfully
>>>deploy
>>> a complete solution stack based on Apache Hadoop releases.
>>>
>>> Next Steps:
>>>
>>> All of the above changes are part of the Windows Azure HDInsight and
>>> HDInsight Server products from Microsoft. We have successfully
>>>on-boarded
>>> several internal customers and have been running production workloads on
>>> Windows Azure HDInsight. Our vision is to create a big data platform
>>>based
>>> on Hadoop, and we are committed to helping make Hadoop a world-class
>>> solution that anyone can use to solve their biggest data challenges.
>>>
>>> As an immediate next step, we would like to have a discussion around how
>>> we can ensure that the quality of the mainline Hadoop branches on
>>>Windows
>>> is maintained. To this end, we would like to get to the state where we
>>>have
>>> pre-checkin validation gates and nightly test runs enabled on Windows.
>>>If
>>> you have any suggestions around this, please do send an email.  We are
>>> committed to helping sustain the long-term quality of Hadoop on both
>>>Linux
>>> and Windows.
>>>
>>> We sincerely thank the community for their contribution and support so
>>> far. And hope to continue having a close engagement in the future.
>>>
>>> -Microsoft HDInsight Team
>>>
>>>
>>> -----Original Message-----
>>> From: Suresh Srinivas [mailto:suresh@hortonworks.com]
>>> Sent: Thursday, February 7, 2013 5:42 PM
>>> To: common-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org;
>>> hdfs-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
>>> Subject: Heads up - merge branch-trunk-win to trunk
>>>
>>> The support for Hadoop on Windows was proposed in HADOOP-8079<
>>> https://issues.apache.org/jira/browse/HADOOP-8079> almost a year ago.
>>>The
>>> goal was to make Hadoop natively integrated, full-featured, and
>>>performance
>>> and scalability tuned on Windows Server or Windows Azure.
>>> We are happy to announce that a lot of progress has been made in this
>>> regard.
>>>
>>> Initial work started in a feature branch, branch-1-win, based on
>>>branch-1.
>>> The details related to the work done in the branch can be seen in
>>> CHANGES.txt<
>>>
>>>http://svn.apache.org/viewvc/hadoop/common/branches/branch-1-win/CHANGES.
>>>branch-1-win.txt?view=markup
>>> >.
>>> This work has been ported to a branch, branch-trunk-win, based on trunk.
>>> Merge patch for this is available on
>>> HADOOP-8562<https://issues.apache.org/jira/browse/HADOOP-8562>
>>> .
>>>
>>> Highlights of the work done so far:
>>> 1. Necessary changes in Hadoop to run natively on Windows. These changes
>>> handle differences in platforms related to path names, process/task
>>> management etc.
>>> 2. Addition of winutils tools for managing file permissions and
>>>ownership,
>>> user group mapping, hardlinks, symbolic links, chmod, disk utilization,
>>>and
>>> process/task management.
>>> 3. Added cmd scripts equivalent to existing shell scripts
>>> hadoop-daemon.sh, start and stop scripts.
>>> 4. Addition of block placement policy implemnation to support cloud
>>> enviroment, more specifically Azure.
>>>
>>> We are very close to wrapping up the work in branch-trunk-win and
>>>getting
>>> ready for a merge. Currently the merge patch is passing close to 100% of
>>> unit tests on Linux. Soon I will call for a vote to merge this branch
>>>into
>>> trunk.
>>>
>>> Next steps:
>>> 1. Call for vote to merge branch-trunk-win to trunk, when the work
>>> completes and precommit build is clean.
>>> 2. Start a discussion on adding Jenkins precommit builds on windows and
>>> how to integrate that with the existing commit process.
>>>
>>> Let me know if you have any questions.
>>>
>>> Regards,
>>> Suresh
>>>
>>>
>>
>>
>>--
>>http://hortonworks.com/download/
>

Re: [Vote] Merge branch-trunk-win to trunk

[Eli Collins <el...@cloudera.com>]

Bobby raises some good questions.  A related one, since most current
developers won't add Windows support for new features that are
platform specific is it assumed that Windows development will either
lag or will people actively work on keeping Windows up with the
latest?  And vice versa in case Windows support is implemented first.

Is there a jira for resolving the outstanding TODOs in the code base
(similar to HDFS-2148)?  Looks like this merge doesn't introduce many
which is great (just did a quick diff and grep).

Thanks,
Eli

On Wed, Feb 27, 2013 at 8:17 AM, Robert Evans <ev...@yahoo-inc.com> wrote:
> After this is merged in is Windows still going to be a second class
> citizen but happens to work for more than just development or is it a
> fully supported platform where if something breaks it can block a release?
>  How do we as a community intend to keep Windows support from breaking?
> We don't have any Jenkins slaves to be able to run nightly tests to
> validate everything still compiles/runs.  This is not a blocker for me
> because we often rely on individuals and groups to test Hadoop, but I do
> think we need to have this discussion before we put it in.
>
> --Bobby
>
> On 2/26/13 4:55 PM, "Suresh Srinivas" <su...@hortonworks.com> wrote:
>
>>I had posted heads up about merging branch-trunk-win to trunk on Feb 8th.
>>I
>>am happy to announce that we are ready for the merge.
>>
>>Here is a brief recap on the highlights of the work done:
>>- Command-line scripts for the Hadoop surface area
>>- Mapping the HDFS permissions model to Windows
>>- Abstracted and reconciled mismatches around differences in Path
>>semantics
>>in Java and Windows
>>- Native Task Controller for Windows
>>- Implementation of a Block Placement Policy to support cloud
>>environments,
>>more specifically Azure.
>>- Implementation of Hadoop native libraries for Windows (compression
>>codecs, native I/O)
>>- Several reliability issues, including race-conditions, intermittent test
>>failures, resource leaks.
>>- Several new unit test cases written for the above changes
>>
>>Please find the details of the work in CHANGES.branch-trunk-win.txt -
>>Common changes<http://bit.ly/Xe7Ynv>, HDFS changes<http://bit.ly/13QOSo9>,
>>and YARN and MapReduce changes <http://bit.ly/128zzMt>. This is the work
>>ported from branch-1-win to a branch based on trunk.
>>
>>For details of the testing done, please see the thread -
>>http://bit.ly/WpavJ4. Merge patch for this is available on HADOOP-8562<
>>https://issues.apache.org/jira/browse/HADOOP-8562>.
>>
>>This was a large undertaking that involved developing code, testing the
>>entire Hadoop stack, including scale tests. This is made possible only
>>with
>>the contribution from many many folks in the community. Following people
>>contributed to this work: Ivan Mitic, Chuan Liu, Ramya Sunil, Bikas Saha,
>>Kanna Karanam, John Gordon, Brandon Li, Chris Nauroth, David Lao, Sumadhur
>>Reddy Bolli, Arpit Agarwal, Ahmed El Baz, Mike Liddell, Jing Zhao, Thejas
>>Nair, Steve Maine, Ganeshan Iyer, Raja Aluri, Giridharan Kesavan, Ramya
>>Bharathi Nimmagadda, Daryn Sharp, Arun Murthy, Tsz-Wo Nicholas Sze, Suresh
>>Srinivas and Sanjay Radia. There are many others who contributed as well
>>providing feedback and comments on numerous jiras.
>>
>>The vote will run for seven days and will end on March 5, 6:00PM PST.
>>
>>Regards,
>>Suresh
>>
>>
>>
>>
>>On Thu, Feb 7, 2013 at 6:41 PM, Mahadevan Venkatraman
>><ma...@microsoft.com>wrote:
>>
>>> It is super exciting to look at the prospect of these changes being
>>>merged
>>> to trunk. Having Windows as one of the supported Hadoop platforms is a
>>> fantastic opportunity both for the Hadoop project and Microsoft
>>>customers.
>>>
>>> This work began around a year back when a few of us started with a basic
>>> port of Hadoop on Windows. Ever since, the Hadoop team in Microsoft have
>>> made significant progress in the following areas:
>>> (PS: Some of these items are already included in Suresh's email, but
>>> including again for completeness)
>>>
>>> - Command-line scripts for the Hadoop surface area
>>> - Mapping the HDFS permissions model to Windows
>>> - Abstracted and reconciled mismatches around differences in Path
>>> semantics in Java and Windows
>>> - Native Task Controller for Windows
>>> - Implementation of a Block Placement Policy to support cloud
>>> environments, more specifically Azure.
>>> - Implementation of Hadoop native libraries for Windows (compression
>>> codecs, native I/O) - Several reliability issues, including
>>> race-conditions, intermittent test failures, resource leaks.
>>> - Several new unit test cases written for the above changes
>>>
>>> In the process, we have closely engaged with the Apache open source
>>> community and have got great support and assistance from the community
>>>in
>>> terms of contributing fixes, code review comments and commits.
>>>
>>> In addition, the Hadoop team at Microsoft has also made good progress in
>>> other projects including Hive, Pig, Sqoop, Oozie, HCat and HBase. Many
>>>of
>>> these changes have already been committed to the respective trunks with
>>> help from various committers and contributors. It is great to see the
>>> commitment of the community to support multiple platforms, and we look
>>> forward to the day when a developer/customer is able to successfully
>>>deploy
>>> a complete solution stack based on Apache Hadoop releases.
>>>
>>> Next Steps:
>>>
>>> All of the above changes are part of the Windows Azure HDInsight and
>>> HDInsight Server products from Microsoft. We have successfully
>>>on-boarded
>>> several internal customers and have been running production workloads on
>>> Windows Azure HDInsight. Our vision is to create a big data platform
>>>based
>>> on Hadoop, and we are committed to helping make Hadoop a world-class
>>> solution that anyone can use to solve their biggest data challenges.
>>>
>>> As an immediate next step, we would like to have a discussion around how
>>> we can ensure that the quality of the mainline Hadoop branches on
>>>Windows
>>> is maintained. To this end, we would like to get to the state where we
>>>have
>>> pre-checkin validation gates and nightly test runs enabled on Windows.
>>>If
>>> you have any suggestions around this, please do send an email.  We are
>>> committed to helping sustain the long-term quality of Hadoop on both
>>>Linux
>>> and Windows.
>>>
>>> We sincerely thank the community for their contribution and support so
>>> far. And hope to continue having a close engagement in the future.
>>>
>>> -Microsoft HDInsight Team
>>>
>>>
>>> -----Original Message-----
>>> From: Suresh Srinivas [mailto:suresh@hortonworks.com]
>>> Sent: Thursday, February 7, 2013 5:42 PM
>>> To: common-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org;
>>> hdfs-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
>>> Subject: Heads up - merge branch-trunk-win to trunk
>>>
>>> The support for Hadoop on Windows was proposed in HADOOP-8079<
>>> https://issues.apache.org/jira/browse/HADOOP-8079> almost a year ago.
>>>The
>>> goal was to make Hadoop natively integrated, full-featured, and
>>>performance
>>> and scalability tuned on Windows Server or Windows Azure.
>>> We are happy to announce that a lot of progress has been made in this
>>> regard.
>>>
>>> Initial work started in a feature branch, branch-1-win, based on
>>>branch-1.
>>> The details related to the work done in the branch can be seen in
>>> CHANGES.txt<
>>>
>>>http://svn.apache.org/viewvc/hadoop/common/branches/branch-1-win/CHANGES.
>>>branch-1-win.txt?view=markup
>>> >.
>>> This work has been ported to a branch, branch-trunk-win, based on trunk.
>>> Merge patch for this is available on
>>> HADOOP-8562<https://issues.apache.org/jira/browse/HADOOP-8562>
>>> .
>>>
>>> Highlights of the work done so far:
>>> 1. Necessary changes in Hadoop to run natively on Windows. These changes
>>> handle differences in platforms related to path names, process/task
>>> management etc.
>>> 2. Addition of winutils tools for managing file permissions and
>>>ownership,
>>> user group mapping, hardlinks, symbolic links, chmod, disk utilization,
>>>and
>>> process/task management.
>>> 3. Added cmd scripts equivalent to existing shell scripts
>>> hadoop-daemon.sh, start and stop scripts.
>>> 4. Addition of block placement policy implemnation to support cloud
>>> enviroment, more specifically Azure.
>>>
>>> We are very close to wrapping up the work in branch-trunk-win and
>>>getting
>>> ready for a merge. Currently the merge patch is passing close to 100% of
>>> unit tests on Linux. Soon I will call for a vote to merge this branch
>>>into
>>> trunk.
>>>
>>> Next steps:
>>> 1. Call for vote to merge branch-trunk-win to trunk, when the work
>>> completes and precommit build is clean.
>>> 2. Start a discussion on adding Jenkins precommit builds on windows and
>>> how to integrate that with the existing commit process.
>>>
>>> Let me know if you have any questions.
>>>
>>> Regards,
>>> Suresh
>>>
>>>
>>
>>
>>--
>>http://hortonworks.com/download/
>

Re: [Vote] Merge branch-trunk-win to trunk

[Harsh J <ha...@cloudera.com>]

Similar personal concern as Robert: Does this bring about a
development process change? Do new features all need to work on
Windows as well to go into trunk (i.e. immediately or eventually,
either way requires a new policy for all of us devs)? Not that anyone
would be avoiding doing that, I just ask cause it could impact time
and effort required for any major undertaking.

While most of the project today is cross-platform (via Java, etc.,
which thankfully remove path handling problems and the sorts),
performance improvements at least are going the native side these
days, which is where I see this have some impact. We've not been
perfectly successful in having the natives continuously work on
Solaris/etc. in the past, mainly due to the platform focus of the
majority (not all) of devs working on the project(s). Some form of a
development policy here would ensure proper Windows support for the
features we intend to ship along are not very divergent such that we
end up having to maintain docs as well, detailing each task yet to be
done (these tend to grow if allowed).

Useful to also note from another OSS project KDE, that their working
builds of Windows are usually on 1-2 releases of the past (i.e. for
example, KDE release is currently at 4.10, but the last released
Windows port is still 4.8 today). KDE uses Qt, which is cross-platform
by itself, but there's still a port team and a ported release
maintained separately (but under the same org.) due to the major
development happening on Linux. Same case for *BSD as well. My own
patches there at some point have caused trouble cause I did something
that I only tested on one platform, and about a bit later things got
revised to support the other ones where it was unnecessarily breaking.

Or if am being too concerned about feature/performance divergence, let me know.

On Wed, Feb 27, 2013 at 9:47 PM, Robert Evans <ev...@yahoo-inc.com> wrote:
> After this is merged in is Windows still going to be a second class
> citizen but happens to work for more than just development or is it a
> fully supported platform where if something breaks it can block a release?
>  How do we as a community intend to keep Windows support from breaking?
> We don't have any Jenkins slaves to be able to run nightly tests to
> validate everything still compiles/runs.  This is not a blocker for me
> because we often rely on individuals and groups to test Hadoop, but I do
> think we need to have this discussion before we put it in.
>
> --Bobby
>
> On 2/26/13 4:55 PM, "Suresh Srinivas" <su...@hortonworks.com> wrote:
>
>>I had posted heads up about merging branch-trunk-win to trunk on Feb 8th.
>>I
>>am happy to announce that we are ready for the merge.
>>
>>Here is a brief recap on the highlights of the work done:
>>- Command-line scripts for the Hadoop surface area
>>- Mapping the HDFS permissions model to Windows
>>- Abstracted and reconciled mismatches around differences in Path
>>semantics
>>in Java and Windows
>>- Native Task Controller for Windows
>>- Implementation of a Block Placement Policy to support cloud
>>environments,
>>more specifically Azure.
>>- Implementation of Hadoop native libraries for Windows (compression
>>codecs, native I/O)
>>- Several reliability issues, including race-conditions, intermittent test
>>failures, resource leaks.
>>- Several new unit test cases written for the above changes
>>
>>Please find the details of the work in CHANGES.branch-trunk-win.txt -
>>Common changes<http://bit.ly/Xe7Ynv>, HDFS changes<http://bit.ly/13QOSo9>,
>>and YARN and MapReduce changes <http://bit.ly/128zzMt>. This is the work
>>ported from branch-1-win to a branch based on trunk.
>>
>>For details of the testing done, please see the thread -
>>http://bit.ly/WpavJ4. Merge patch for this is available on HADOOP-8562<
>>https://issues.apache.org/jira/browse/HADOOP-8562>.
>>
>>This was a large undertaking that involved developing code, testing the
>>entire Hadoop stack, including scale tests. This is made possible only
>>with
>>the contribution from many many folks in the community. Following people
>>contributed to this work: Ivan Mitic, Chuan Liu, Ramya Sunil, Bikas Saha,
>>Kanna Karanam, John Gordon, Brandon Li, Chris Nauroth, David Lao, Sumadhur
>>Reddy Bolli, Arpit Agarwal, Ahmed El Baz, Mike Liddell, Jing Zhao, Thejas
>>Nair, Steve Maine, Ganeshan Iyer, Raja Aluri, Giridharan Kesavan, Ramya
>>Bharathi Nimmagadda, Daryn Sharp, Arun Murthy, Tsz-Wo Nicholas Sze, Suresh
>>Srinivas and Sanjay Radia. There are many others who contributed as well
>>providing feedback and comments on numerous jiras.
>>
>>The vote will run for seven days and will end on March 5, 6:00PM PST.
>>
>>Regards,
>>Suresh
>>
>>
>>
>>
>>On Thu, Feb 7, 2013 at 6:41 PM, Mahadevan Venkatraman
>><ma...@microsoft.com>wrote:
>>
>>> It is super exciting to look at the prospect of these changes being
>>>merged
>>> to trunk. Having Windows as one of the supported Hadoop platforms is a
>>> fantastic opportunity both for the Hadoop project and Microsoft
>>>customers.
>>>
>>> This work began around a year back when a few of us started with a basic
>>> port of Hadoop on Windows. Ever since, the Hadoop team in Microsoft have
>>> made significant progress in the following areas:
>>> (PS: Some of these items are already included in Suresh's email, but
>>> including again for completeness)
>>>
>>> - Command-line scripts for the Hadoop surface area
>>> - Mapping the HDFS permissions model to Windows
>>> - Abstracted and reconciled mismatches around differences in Path
>>> semantics in Java and Windows
>>> - Native Task Controller for Windows
>>> - Implementation of a Block Placement Policy to support cloud
>>> environments, more specifically Azure.
>>> - Implementation of Hadoop native libraries for Windows (compression
>>> codecs, native I/O) - Several reliability issues, including
>>> race-conditions, intermittent test failures, resource leaks.
>>> - Several new unit test cases written for the above changes
>>>
>>> In the process, we have closely engaged with the Apache open source
>>> community and have got great support and assistance from the community
>>>in
>>> terms of contributing fixes, code review comments and commits.
>>>
>>> In addition, the Hadoop team at Microsoft has also made good progress in
>>> other projects including Hive, Pig, Sqoop, Oozie, HCat and HBase. Many
>>>of
>>> these changes have already been committed to the respective trunks with
>>> help from various committers and contributors. It is great to see the
>>> commitment of the community to support multiple platforms, and we look
>>> forward to the day when a developer/customer is able to successfully
>>>deploy
>>> a complete solution stack based on Apache Hadoop releases.
>>>
>>> Next Steps:
>>>
>>> All of the above changes are part of the Windows Azure HDInsight and
>>> HDInsight Server products from Microsoft. We have successfully
>>>on-boarded
>>> several internal customers and have been running production workloads on
>>> Windows Azure HDInsight. Our vision is to create a big data platform
>>>based
>>> on Hadoop, and we are committed to helping make Hadoop a world-class
>>> solution that anyone can use to solve their biggest data challenges.
>>>
>>> As an immediate next step, we would like to have a discussion around how
>>> we can ensure that the quality of the mainline Hadoop branches on
>>>Windows
>>> is maintained. To this end, we would like to get to the state where we
>>>have
>>> pre-checkin validation gates and nightly test runs enabled on Windows.
>>>If
>>> you have any suggestions around this, please do send an email.  We are
>>> committed to helping sustain the long-term quality of Hadoop on both
>>>Linux
>>> and Windows.
>>>
>>> We sincerely thank the community for their contribution and support so
>>> far. And hope to continue having a close engagement in the future.
>>>
>>> -Microsoft HDInsight Team
>>>
>>>
>>> -----Original Message-----
>>> From: Suresh Srinivas [mailto:suresh@hortonworks.com]
>>> Sent: Thursday, February 7, 2013 5:42 PM
>>> To: common-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org;
>>> hdfs-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
>>> Subject: Heads up - merge branch-trunk-win to trunk
>>>
>>> The support for Hadoop on Windows was proposed in HADOOP-8079<
>>> https://issues.apache.org/jira/browse/HADOOP-8079> almost a year ago.
>>>The
>>> goal was to make Hadoop natively integrated, full-featured, and
>>>performance
>>> and scalability tuned on Windows Server or Windows Azure.
>>> We are happy to announce that a lot of progress has been made in this
>>> regard.
>>>
>>> Initial work started in a feature branch, branch-1-win, based on
>>>branch-1.
>>> The details related to the work done in the branch can be seen in
>>> CHANGES.txt<
>>>
>>>http://svn.apache.org/viewvc/hadoop/common/branches/branch-1-win/CHANGES.
>>>branch-1-win.txt?view=markup
>>> >.
>>> This work has been ported to a branch, branch-trunk-win, based on trunk.
>>> Merge patch for this is available on
>>> HADOOP-8562<https://issues.apache.org/jira/browse/HADOOP-8562>
>>> .
>>>
>>> Highlights of the work done so far:
>>> 1. Necessary changes in Hadoop to run natively on Windows. These changes
>>> handle differences in platforms related to path names, process/task
>>> management etc.
>>> 2. Addition of winutils tools for managing file permissions and
>>>ownership,
>>> user group mapping, hardlinks, symbolic links, chmod, disk utilization,
>>>and
>>> process/task management.
>>> 3. Added cmd scripts equivalent to existing shell scripts
>>> hadoop-daemon.sh, start and stop scripts.
>>> 4. Addition of block placement policy implemnation to support cloud
>>> enviroment, more specifically Azure.
>>>
>>> We are very close to wrapping up the work in branch-trunk-win and
>>>getting
>>> ready for a merge. Currently the merge patch is passing close to 100% of
>>> unit tests on Linux. Soon I will call for a vote to merge this branch
>>>into
>>> trunk.
>>>
>>> Next steps:
>>> 1. Call for vote to merge branch-trunk-win to trunk, when the work
>>> completes and precommit build is clean.
>>> 2. Start a discussion on adding Jenkins precommit builds on windows and
>>> how to integrate that with the existing commit process.
>>>
>>> Let me know if you have any questions.
>>>
>>> Regards,
>>> Suresh
>>>
>>>
>>
>>
>>--
>>http://hortonworks.com/download/
>



--
Harsh J

Re: [Vote] Merge branch-trunk-win to trunk

[Eli Collins <el...@cloudera.com>]

Bobby raises some good questions.  A related one, since most current
developers won't add Windows support for new features that are
platform specific is it assumed that Windows development will either
lag or will people actively work on keeping Windows up with the
latest?  And vice versa in case Windows support is implemented first.

Is there a jira for resolving the outstanding TODOs in the code base
(similar to HDFS-2148)?  Looks like this merge doesn't introduce many
which is great (just did a quick diff and grep).

Thanks,
Eli

On Wed, Feb 27, 2013 at 8:17 AM, Robert Evans <ev...@yahoo-inc.com> wrote:
> After this is merged in is Windows still going to be a second class
> citizen but happens to work for more than just development or is it a
> fully supported platform where if something breaks it can block a release?
>  How do we as a community intend to keep Windows support from breaking?
> We don't have any Jenkins slaves to be able to run nightly tests to
> validate everything still compiles/runs.  This is not a blocker for me
> because we often rely on individuals and groups to test Hadoop, but I do
> think we need to have this discussion before we put it in.
>
> --Bobby
>
> On 2/26/13 4:55 PM, "Suresh Srinivas" <su...@hortonworks.com> wrote:
>
>>I had posted heads up about merging branch-trunk-win to trunk on Feb 8th.
>>I
>>am happy to announce that we are ready for the merge.
>>
>>Here is a brief recap on the highlights of the work done:
>>- Command-line scripts for the Hadoop surface area
>>- Mapping the HDFS permissions model to Windows
>>- Abstracted and reconciled mismatches around differences in Path
>>semantics
>>in Java and Windows
>>- Native Task Controller for Windows
>>- Implementation of a Block Placement Policy to support cloud
>>environments,
>>more specifically Azure.
>>- Implementation of Hadoop native libraries for Windows (compression
>>codecs, native I/O)
>>- Several reliability issues, including race-conditions, intermittent test
>>failures, resource leaks.
>>- Several new unit test cases written for the above changes
>>
>>Please find the details of the work in CHANGES.branch-trunk-win.txt -
>>Common changes<http://bit.ly/Xe7Ynv>, HDFS changes<http://bit.ly/13QOSo9>,
>>and YARN and MapReduce changes <http://bit.ly/128zzMt>. This is the work
>>ported from branch-1-win to a branch based on trunk.
>>
>>For details of the testing done, please see the thread -
>>http://bit.ly/WpavJ4. Merge patch for this is available on HADOOP-8562<
>>https://issues.apache.org/jira/browse/HADOOP-8562>.
>>
>>This was a large undertaking that involved developing code, testing the
>>entire Hadoop stack, including scale tests. This is made possible only
>>with
>>the contribution from many many folks in the community. Following people
>>contributed to this work: Ivan Mitic, Chuan Liu, Ramya Sunil, Bikas Saha,
>>Kanna Karanam, John Gordon, Brandon Li, Chris Nauroth, David Lao, Sumadhur
>>Reddy Bolli, Arpit Agarwal, Ahmed El Baz, Mike Liddell, Jing Zhao, Thejas
>>Nair, Steve Maine, Ganeshan Iyer, Raja Aluri, Giridharan Kesavan, Ramya
>>Bharathi Nimmagadda, Daryn Sharp, Arun Murthy, Tsz-Wo Nicholas Sze, Suresh
>>Srinivas and Sanjay Radia. There are many others who contributed as well
>>providing feedback and comments on numerous jiras.
>>
>>The vote will run for seven days and will end on March 5, 6:00PM PST.
>>
>>Regards,
>>Suresh
>>
>>
>>
>>
>>On Thu, Feb 7, 2013 at 6:41 PM, Mahadevan Venkatraman
>><ma...@microsoft.com>wrote:
>>
>>> It is super exciting to look at the prospect of these changes being
>>>merged
>>> to trunk. Having Windows as one of the supported Hadoop platforms is a
>>> fantastic opportunity both for the Hadoop project and Microsoft
>>>customers.
>>>
>>> This work began around a year back when a few of us started with a basic
>>> port of Hadoop on Windows. Ever since, the Hadoop team in Microsoft have
>>> made significant progress in the following areas:
>>> (PS: Some of these items are already included in Suresh's email, but
>>> including again for completeness)
>>>
>>> - Command-line scripts for the Hadoop surface area
>>> - Mapping the HDFS permissions model to Windows
>>> - Abstracted and reconciled mismatches around differences in Path
>>> semantics in Java and Windows
>>> - Native Task Controller for Windows
>>> - Implementation of a Block Placement Policy to support cloud
>>> environments, more specifically Azure.
>>> - Implementation of Hadoop native libraries for Windows (compression
>>> codecs, native I/O) - Several reliability issues, including
>>> race-conditions, intermittent test failures, resource leaks.
>>> - Several new unit test cases written for the above changes
>>>
>>> In the process, we have closely engaged with the Apache open source
>>> community and have got great support and assistance from the community
>>>in
>>> terms of contributing fixes, code review comments and commits.
>>>
>>> In addition, the Hadoop team at Microsoft has also made good progress in
>>> other projects including Hive, Pig, Sqoop, Oozie, HCat and HBase. Many
>>>of
>>> these changes have already been committed to the respective trunks with
>>> help from various committers and contributors. It is great to see the
>>> commitment of the community to support multiple platforms, and we look
>>> forward to the day when a developer/customer is able to successfully
>>>deploy
>>> a complete solution stack based on Apache Hadoop releases.
>>>
>>> Next Steps:
>>>
>>> All of the above changes are part of the Windows Azure HDInsight and
>>> HDInsight Server products from Microsoft. We have successfully
>>>on-boarded
>>> several internal customers and have been running production workloads on
>>> Windows Azure HDInsight. Our vision is to create a big data platform
>>>based
>>> on Hadoop, and we are committed to helping make Hadoop a world-class
>>> solution that anyone can use to solve their biggest data challenges.
>>>
>>> As an immediate next step, we would like to have a discussion around how
>>> we can ensure that the quality of the mainline Hadoop branches on
>>>Windows
>>> is maintained. To this end, we would like to get to the state where we
>>>have
>>> pre-checkin validation gates and nightly test runs enabled on Windows.
>>>If
>>> you have any suggestions around this, please do send an email.  We are
>>> committed to helping sustain the long-term quality of Hadoop on both
>>>Linux
>>> and Windows.
>>>
>>> We sincerely thank the community for their contribution and support so
>>> far. And hope to continue having a close engagement in the future.
>>>
>>> -Microsoft HDInsight Team
>>>
>>>
>>> -----Original Message-----
>>> From: Suresh Srinivas [mailto:suresh@hortonworks.com]
>>> Sent: Thursday, February 7, 2013 5:42 PM
>>> To: common-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org;
>>> hdfs-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
>>> Subject: Heads up - merge branch-trunk-win to trunk
>>>
>>> The support for Hadoop on Windows was proposed in HADOOP-8079<
>>> https://issues.apache.org/jira/browse/HADOOP-8079> almost a year ago.
>>>The
>>> goal was to make Hadoop natively integrated, full-featured, and
>>>performance
>>> and scalability tuned on Windows Server or Windows Azure.
>>> We are happy to announce that a lot of progress has been made in this
>>> regard.
>>>
>>> Initial work started in a feature branch, branch-1-win, based on
>>>branch-1.
>>> The details related to the work done in the branch can be seen in
>>> CHANGES.txt<
>>>
>>>http://svn.apache.org/viewvc/hadoop/common/branches/branch-1-win/CHANGES.
>>>branch-1-win.txt?view=markup
>>> >.
>>> This work has been ported to a branch, branch-trunk-win, based on trunk.
>>> Merge patch for this is available on
>>> HADOOP-8562<https://issues.apache.org/jira/browse/HADOOP-8562>
>>> .
>>>
>>> Highlights of the work done so far:
>>> 1. Necessary changes in Hadoop to run natively on Windows. These changes
>>> handle differences in platforms related to path names, process/task
>>> management etc.
>>> 2. Addition of winutils tools for managing file permissions and
>>>ownership,
>>> user group mapping, hardlinks, symbolic links, chmod, disk utilization,
>>>and
>>> process/task management.
>>> 3. Added cmd scripts equivalent to existing shell scripts
>>> hadoop-daemon.sh, start and stop scripts.
>>> 4. Addition of block placement policy implemnation to support cloud
>>> enviroment, more specifically Azure.
>>>
>>> We are very close to wrapping up the work in branch-trunk-win and
>>>getting
>>> ready for a merge. Currently the merge patch is passing close to 100% of
>>> unit tests on Linux. Soon I will call for a vote to merge this branch
>>>into
>>> trunk.
>>>
>>> Next steps:
>>> 1. Call for vote to merge branch-trunk-win to trunk, when the work
>>> completes and precommit build is clean.
>>> 2. Start a discussion on adding Jenkins precommit builds on windows and
>>> how to integrate that with the existing commit process.
>>>
>>> Let me know if you have any questions.
>>>
>>> Regards,
>>> Suresh
>>>
>>>
>>
>>
>>--
>>http://hortonworks.com/download/
>

Re: [Vote] Merge branch-trunk-win to trunk

[Eli Collins <el...@cloudera.com>]

Bobby raises some good questions.  A related one, since most current
developers won't add Windows support for new features that are
platform specific is it assumed that Windows development will either
lag or will people actively work on keeping Windows up with the
latest?  And vice versa in case Windows support is implemented first.

Is there a jira for resolving the outstanding TODOs in the code base
(similar to HDFS-2148)?  Looks like this merge doesn't introduce many
which is great (just did a quick diff and grep).

Thanks,
Eli

On Wed, Feb 27, 2013 at 8:17 AM, Robert Evans <ev...@yahoo-inc.com> wrote:
> After this is merged in is Windows still going to be a second class
> citizen but happens to work for more than just development or is it a
> fully supported platform where if something breaks it can block a release?
>  How do we as a community intend to keep Windows support from breaking?
> We don't have any Jenkins slaves to be able to run nightly tests to
> validate everything still compiles/runs.  This is not a blocker for me
> because we often rely on individuals and groups to test Hadoop, but I do
> think we need to have this discussion before we put it in.
>
> --Bobby
>
> On 2/26/13 4:55 PM, "Suresh Srinivas" <su...@hortonworks.com> wrote:
>
>>I had posted heads up about merging branch-trunk-win to trunk on Feb 8th.
>>I
>>am happy to announce that we are ready for the merge.
>>
>>Here is a brief recap on the highlights of the work done:
>>- Command-line scripts for the Hadoop surface area
>>- Mapping the HDFS permissions model to Windows
>>- Abstracted and reconciled mismatches around differences in Path
>>semantics
>>in Java and Windows
>>- Native Task Controller for Windows
>>- Implementation of a Block Placement Policy to support cloud
>>environments,
>>more specifically Azure.
>>- Implementation of Hadoop native libraries for Windows (compression
>>codecs, native I/O)
>>- Several reliability issues, including race-conditions, intermittent test
>>failures, resource leaks.
>>- Several new unit test cases written for the above changes
>>
>>Please find the details of the work in CHANGES.branch-trunk-win.txt -
>>Common changes<http://bit.ly/Xe7Ynv>, HDFS changes<http://bit.ly/13QOSo9>,
>>and YARN and MapReduce changes <http://bit.ly/128zzMt>. This is the work
>>ported from branch-1-win to a branch based on trunk.
>>
>>For details of the testing done, please see the thread -
>>http://bit.ly/WpavJ4. Merge patch for this is available on HADOOP-8562<
>>https://issues.apache.org/jira/browse/HADOOP-8562>.
>>
>>This was a large undertaking that involved developing code, testing the
>>entire Hadoop stack, including scale tests. This is made possible only
>>with
>>the contribution from many many folks in the community. Following people
>>contributed to this work: Ivan Mitic, Chuan Liu, Ramya Sunil, Bikas Saha,
>>Kanna Karanam, John Gordon, Brandon Li, Chris Nauroth, David Lao, Sumadhur
>>Reddy Bolli, Arpit Agarwal, Ahmed El Baz, Mike Liddell, Jing Zhao, Thejas
>>Nair, Steve Maine, Ganeshan Iyer, Raja Aluri, Giridharan Kesavan, Ramya
>>Bharathi Nimmagadda, Daryn Sharp, Arun Murthy, Tsz-Wo Nicholas Sze, Suresh
>>Srinivas and Sanjay Radia. There are many others who contributed as well
>>providing feedback and comments on numerous jiras.
>>
>>The vote will run for seven days and will end on March 5, 6:00PM PST.
>>
>>Regards,
>>Suresh
>>
>>
>>
>>
>>On Thu, Feb 7, 2013 at 6:41 PM, Mahadevan Venkatraman
>><ma...@microsoft.com>wrote:
>>
>>> It is super exciting to look at the prospect of these changes being
>>>merged
>>> to trunk. Having Windows as one of the supported Hadoop platforms is a
>>> fantastic opportunity both for the Hadoop project and Microsoft
>>>customers.
>>>
>>> This work began around a year back when a few of us started with a basic
>>> port of Hadoop on Windows. Ever since, the Hadoop team in Microsoft have
>>> made significant progress in the following areas:
>>> (PS: Some of these items are already included in Suresh's email, but
>>> including again for completeness)
>>>
>>> - Command-line scripts for the Hadoop surface area
>>> - Mapping the HDFS permissions model to Windows
>>> - Abstracted and reconciled mismatches around differences in Path
>>> semantics in Java and Windows
>>> - Native Task Controller for Windows
>>> - Implementation of a Block Placement Policy to support cloud
>>> environments, more specifically Azure.
>>> - Implementation of Hadoop native libraries for Windows (compression
>>> codecs, native I/O) - Several reliability issues, including
>>> race-conditions, intermittent test failures, resource leaks.
>>> - Several new unit test cases written for the above changes
>>>
>>> In the process, we have closely engaged with the Apache open source
>>> community and have got great support and assistance from the community
>>>in
>>> terms of contributing fixes, code review comments and commits.
>>>
>>> In addition, the Hadoop team at Microsoft has also made good progress in
>>> other projects including Hive, Pig, Sqoop, Oozie, HCat and HBase. Many
>>>of
>>> these changes have already been committed to the respective trunks with
>>> help from various committers and contributors. It is great to see the
>>> commitment of the community to support multiple platforms, and we look
>>> forward to the day when a developer/customer is able to successfully
>>>deploy
>>> a complete solution stack based on Apache Hadoop releases.
>>>
>>> Next Steps:
>>>
>>> All of the above changes are part of the Windows Azure HDInsight and
>>> HDInsight Server products from Microsoft. We have successfully
>>>on-boarded
>>> several internal customers and have been running production workloads on
>>> Windows Azure HDInsight. Our vision is to create a big data platform
>>>based
>>> on Hadoop, and we are committed to helping make Hadoop a world-class
>>> solution that anyone can use to solve their biggest data challenges.
>>>
>>> As an immediate next step, we would like to have a discussion around how
>>> we can ensure that the quality of the mainline Hadoop branches on
>>>Windows
>>> is maintained. To this end, we would like to get to the state where we
>>>have
>>> pre-checkin validation gates and nightly test runs enabled on Windows.
>>>If
>>> you have any suggestions around this, please do send an email.  We are
>>> committed to helping sustain the long-term quality of Hadoop on both
>>>Linux
>>> and Windows.
>>>
>>> We sincerely thank the community for their contribution and support so
>>> far. And hope to continue having a close engagement in the future.
>>>
>>> -Microsoft HDInsight Team
>>>
>>>
>>> -----Original Message-----
>>> From: Suresh Srinivas [mailto:suresh@hortonworks.com]
>>> Sent: Thursday, February 7, 2013 5:42 PM
>>> To: common-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org;
>>> hdfs-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
>>> Subject: Heads up - merge branch-trunk-win to trunk
>>>
>>> The support for Hadoop on Windows was proposed in HADOOP-8079<
>>> https://issues.apache.org/jira/browse/HADOOP-8079> almost a year ago.
>>>The
>>> goal was to make Hadoop natively integrated, full-featured, and
>>>performance
>>> and scalability tuned on Windows Server or Windows Azure.
>>> We are happy to announce that a lot of progress has been made in this
>>> regard.
>>>
>>> Initial work started in a feature branch, branch-1-win, based on
>>>branch-1.
>>> The details related to the work done in the branch can be seen in
>>> CHANGES.txt<
>>>
>>>http://svn.apache.org/viewvc/hadoop/common/branches/branch-1-win/CHANGES.
>>>branch-1-win.txt?view=markup
>>> >.
>>> This work has been ported to a branch, branch-trunk-win, based on trunk.
>>> Merge patch for this is available on
>>> HADOOP-8562<https://issues.apache.org/jira/browse/HADOOP-8562>
>>> .
>>>
>>> Highlights of the work done so far:
>>> 1. Necessary changes in Hadoop to run natively on Windows. These changes
>>> handle differences in platforms related to path names, process/task
>>> management etc.
>>> 2. Addition of winutils tools for managing file permissions and
>>>ownership,
>>> user group mapping, hardlinks, symbolic links, chmod, disk utilization,
>>>and
>>> process/task management.
>>> 3. Added cmd scripts equivalent to existing shell scripts
>>> hadoop-daemon.sh, start and stop scripts.
>>> 4. Addition of block placement policy implemnation to support cloud
>>> enviroment, more specifically Azure.
>>>
>>> We are very close to wrapping up the work in branch-trunk-win and
>>>getting
>>> ready for a merge. Currently the merge patch is passing close to 100% of
>>> unit tests on Linux. Soon I will call for a vote to merge this branch
>>>into
>>> trunk.
>>>
>>> Next steps:
>>> 1. Call for vote to merge branch-trunk-win to trunk, when the work
>>> completes and precommit build is clean.
>>> 2. Start a discussion on adding Jenkins precommit builds on windows and
>>> how to integrate that with the existing commit process.
>>>
>>> Let me know if you have any questions.
>>>
>>> Regards,
>>> Suresh
>>>
>>>
>>
>>
>>--
>>http://hortonworks.com/download/
>

Re: [Vote] Merge branch-trunk-win to trunk

[Harsh J <ha...@cloudera.com>]

Similar personal concern as Robert: Does this bring about a
development process change? Do new features all need to work on
Windows as well to go into trunk (i.e. immediately or eventually,
either way requires a new policy for all of us devs)? Not that anyone
would be avoiding doing that, I just ask cause it could impact time
and effort required for any major undertaking.

While most of the project today is cross-platform (via Java, etc.,
which thankfully remove path handling problems and the sorts),
performance improvements at least are going the native side these
days, which is where I see this have some impact. We've not been
perfectly successful in having the natives continuously work on
Solaris/etc. in the past, mainly due to the platform focus of the
majority (not all) of devs working on the project(s). Some form of a
development policy here would ensure proper Windows support for the
features we intend to ship along are not very divergent such that we
end up having to maintain docs as well, detailing each task yet to be
done (these tend to grow if allowed).

Useful to also note from another OSS project KDE, that their working
builds of Windows are usually on 1-2 releases of the past (i.e. for
example, KDE release is currently at 4.10, but the last released
Windows port is still 4.8 today). KDE uses Qt, which is cross-platform
by itself, but there's still a port team and a ported release
maintained separately (but under the same org.) due to the major
development happening on Linux. Same case for *BSD as well. My own
patches there at some point have caused trouble cause I did something
that I only tested on one platform, and about a bit later things got
revised to support the other ones where it was unnecessarily breaking.

Or if am being too concerned about feature/performance divergence, let me know.

On Wed, Feb 27, 2013 at 9:47 PM, Robert Evans <ev...@yahoo-inc.com> wrote:
> After this is merged in is Windows still going to be a second class
> citizen but happens to work for more than just development or is it a
> fully supported platform where if something breaks it can block a release?
>  How do we as a community intend to keep Windows support from breaking?
> We don't have any Jenkins slaves to be able to run nightly tests to
> validate everything still compiles/runs.  This is not a blocker for me
> because we often rely on individuals and groups to test Hadoop, but I do
> think we need to have this discussion before we put it in.
>
> --Bobby
>
> On 2/26/13 4:55 PM, "Suresh Srinivas" <su...@hortonworks.com> wrote:
>
>>I had posted heads up about merging branch-trunk-win to trunk on Feb 8th.
>>I
>>am happy to announce that we are ready for the merge.
>>
>>Here is a brief recap on the highlights of the work done:
>>- Command-line scripts for the Hadoop surface area
>>- Mapping the HDFS permissions model to Windows
>>- Abstracted and reconciled mismatches around differences in Path
>>semantics
>>in Java and Windows
>>- Native Task Controller for Windows
>>- Implementation of a Block Placement Policy to support cloud
>>environments,
>>more specifically Azure.
>>- Implementation of Hadoop native libraries for Windows (compression
>>codecs, native I/O)
>>- Several reliability issues, including race-conditions, intermittent test
>>failures, resource leaks.
>>- Several new unit test cases written for the above changes
>>
>>Please find the details of the work in CHANGES.branch-trunk-win.txt -
>>Common changes<http://bit.ly/Xe7Ynv>, HDFS changes<http://bit.ly/13QOSo9>,
>>and YARN and MapReduce changes <http://bit.ly/128zzMt>. This is the work
>>ported from branch-1-win to a branch based on trunk.
>>
>>For details of the testing done, please see the thread -
>>http://bit.ly/WpavJ4. Merge patch for this is available on HADOOP-8562<
>>https://issues.apache.org/jira/browse/HADOOP-8562>.
>>
>>This was a large undertaking that involved developing code, testing the
>>entire Hadoop stack, including scale tests. This is made possible only
>>with
>>the contribution from many many folks in the community. Following people
>>contributed to this work: Ivan Mitic, Chuan Liu, Ramya Sunil, Bikas Saha,
>>Kanna Karanam, John Gordon, Brandon Li, Chris Nauroth, David Lao, Sumadhur
>>Reddy Bolli, Arpit Agarwal, Ahmed El Baz, Mike Liddell, Jing Zhao, Thejas
>>Nair, Steve Maine, Ganeshan Iyer, Raja Aluri, Giridharan Kesavan, Ramya
>>Bharathi Nimmagadda, Daryn Sharp, Arun Murthy, Tsz-Wo Nicholas Sze, Suresh
>>Srinivas and Sanjay Radia. There are many others who contributed as well
>>providing feedback and comments on numerous jiras.
>>
>>The vote will run for seven days and will end on March 5, 6:00PM PST.
>>
>>Regards,
>>Suresh
>>
>>
>>
>>
>>On Thu, Feb 7, 2013 at 6:41 PM, Mahadevan Venkatraman
>><ma...@microsoft.com>wrote:
>>
>>> It is super exciting to look at the prospect of these changes being
>>>merged
>>> to trunk. Having Windows as one of the supported Hadoop platforms is a
>>> fantastic opportunity both for the Hadoop project and Microsoft
>>>customers.
>>>
>>> This work began around a year back when a few of us started with a basic
>>> port of Hadoop on Windows. Ever since, the Hadoop team in Microsoft have
>>> made significant progress in the following areas:
>>> (PS: Some of these items are already included in Suresh's email, but
>>> including again for completeness)
>>>
>>> - Command-line scripts for the Hadoop surface area
>>> - Mapping the HDFS permissions model to Windows
>>> - Abstracted and reconciled mismatches around differences in Path
>>> semantics in Java and Windows
>>> - Native Task Controller for Windows
>>> - Implementation of a Block Placement Policy to support cloud
>>> environments, more specifically Azure.
>>> - Implementation of Hadoop native libraries for Windows (compression
>>> codecs, native I/O) - Several reliability issues, including
>>> race-conditions, intermittent test failures, resource leaks.
>>> - Several new unit test cases written for the above changes
>>>
>>> In the process, we have closely engaged with the Apache open source
>>> community and have got great support and assistance from the community
>>>in
>>> terms of contributing fixes, code review comments and commits.
>>>
>>> In addition, the Hadoop team at Microsoft has also made good progress in
>>> other projects including Hive, Pig, Sqoop, Oozie, HCat and HBase. Many
>>>of
>>> these changes have already been committed to the respective trunks with
>>> help from various committers and contributors. It is great to see the
>>> commitment of the community to support multiple platforms, and we look
>>> forward to the day when a developer/customer is able to successfully
>>>deploy
>>> a complete solution stack based on Apache Hadoop releases.
>>>
>>> Next Steps:
>>>
>>> All of the above changes are part of the Windows Azure HDInsight and
>>> HDInsight Server products from Microsoft. We have successfully
>>>on-boarded
>>> several internal customers and have been running production workloads on
>>> Windows Azure HDInsight. Our vision is to create a big data platform
>>>based
>>> on Hadoop, and we are committed to helping make Hadoop a world-class
>>> solution that anyone can use to solve their biggest data challenges.
>>>
>>> As an immediate next step, we would like to have a discussion around how
>>> we can ensure that the quality of the mainline Hadoop branches on
>>>Windows
>>> is maintained. To this end, we would like to get to the state where we
>>>have
>>> pre-checkin validation gates and nightly test runs enabled on Windows.
>>>If
>>> you have any suggestions around this, please do send an email.  We are
>>> committed to helping sustain the long-term quality of Hadoop on both
>>>Linux
>>> and Windows.
>>>
>>> We sincerely thank the community for their contribution and support so
>>> far. And hope to continue having a close engagement in the future.
>>>
>>> -Microsoft HDInsight Team
>>>
>>>
>>> -----Original Message-----
>>> From: Suresh Srinivas [mailto:suresh@hortonworks.com]
>>> Sent: Thursday, February 7, 2013 5:42 PM
>>> To: common-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org;
>>> hdfs-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
>>> Subject: Heads up - merge branch-trunk-win to trunk
>>>
>>> The support for Hadoop on Windows was proposed in HADOOP-8079<
>>> https://issues.apache.org/jira/browse/HADOOP-8079> almost a year ago.
>>>The
>>> goal was to make Hadoop natively integrated, full-featured, and
>>>performance
>>> and scalability tuned on Windows Server or Windows Azure.
>>> We are happy to announce that a lot of progress has been made in this
>>> regard.
>>>
>>> Initial work started in a feature branch, branch-1-win, based on
>>>branch-1.
>>> The details related to the work done in the branch can be seen in
>>> CHANGES.txt<
>>>
>>>http://svn.apache.org/viewvc/hadoop/common/branches/branch-1-win/CHANGES.
>>>branch-1-win.txt?view=markup
>>> >.
>>> This work has been ported to a branch, branch-trunk-win, based on trunk.
>>> Merge patch for this is available on
>>> HADOOP-8562<https://issues.apache.org/jira/browse/HADOOP-8562>
>>> .
>>>
>>> Highlights of the work done so far:
>>> 1. Necessary changes in Hadoop to run natively on Windows. These changes
>>> handle differences in platforms related to path names, process/task
>>> management etc.
>>> 2. Addition of winutils tools for managing file permissions and
>>>ownership,
>>> user group mapping, hardlinks, symbolic links, chmod, disk utilization,
>>>and
>>> process/task management.
>>> 3. Added cmd scripts equivalent to existing shell scripts
>>> hadoop-daemon.sh, start and stop scripts.
>>> 4. Addition of block placement policy implemnation to support cloud
>>> enviroment, more specifically Azure.
>>>
>>> We are very close to wrapping up the work in branch-trunk-win and
>>>getting
>>> ready for a merge. Currently the merge patch is passing close to 100% of
>>> unit tests on Linux. Soon I will call for a vote to merge this branch
>>>into
>>> trunk.
>>>
>>> Next steps:
>>> 1. Call for vote to merge branch-trunk-win to trunk, when the work
>>> completes and precommit build is clean.
>>> 2. Start a discussion on adding Jenkins precommit builds on windows and
>>> how to integrate that with the existing commit process.
>>>
>>> Let me know if you have any questions.
>>>
>>> Regards,
>>> Suresh
>>>
>>>
>>
>>
>>--
>>http://hortonworks.com/download/
>



--
Harsh J

Re: [Vote] Merge branch-trunk-win to trunk

[Harsh J <ha...@cloudera.com>]

Similar personal concern as Robert: Does this bring about a
development process change? Do new features all need to work on
Windows as well to go into trunk (i.e. immediately or eventually,
either way requires a new policy for all of us devs)? Not that anyone
would be avoiding doing that, I just ask cause it could impact time
and effort required for any major undertaking.

While most of the project today is cross-platform (via Java, etc.,
which thankfully remove path handling problems and the sorts),
performance improvements at least are going the native side these
days, which is where I see this have some impact. We've not been
perfectly successful in having the natives continuously work on
Solaris/etc. in the past, mainly due to the platform focus of the
majority (not all) of devs working on the project(s). Some form of a
development policy here would ensure proper Windows support for the
features we intend to ship along are not very divergent such that we
end up having to maintain docs as well, detailing each task yet to be
done (these tend to grow if allowed).

Useful to also note from another OSS project KDE, that their working
builds of Windows are usually on 1-2 releases of the past (i.e. for
example, KDE release is currently at 4.10, but the last released
Windows port is still 4.8 today). KDE uses Qt, which is cross-platform
by itself, but there's still a port team and a ported release
maintained separately (but under the same org.) due to the major
development happening on Linux. Same case for *BSD as well. My own
patches there at some point have caused trouble cause I did something
that I only tested on one platform, and about a bit later things got
revised to support the other ones where it was unnecessarily breaking.

Or if am being too concerned about feature/performance divergence, let me know.

On Wed, Feb 27, 2013 at 9:47 PM, Robert Evans <ev...@yahoo-inc.com> wrote:
> After this is merged in is Windows still going to be a second class
> citizen but happens to work for more than just development or is it a
> fully supported platform where if something breaks it can block a release?
>  How do we as a community intend to keep Windows support from breaking?
> We don't have any Jenkins slaves to be able to run nightly tests to
> validate everything still compiles/runs.  This is not a blocker for me
> because we often rely on individuals and groups to test Hadoop, but I do
> think we need to have this discussion before we put it in.
>
> --Bobby
>
> On 2/26/13 4:55 PM, "Suresh Srinivas" <su...@hortonworks.com> wrote:
>
>>I had posted heads up about merging branch-trunk-win to trunk on Feb 8th.
>>I
>>am happy to announce that we are ready for the merge.
>>
>>Here is a brief recap on the highlights of the work done:
>>- Command-line scripts for the Hadoop surface area
>>- Mapping the HDFS permissions model to Windows
>>- Abstracted and reconciled mismatches around differences in Path
>>semantics
>>in Java and Windows
>>- Native Task Controller for Windows
>>- Implementation of a Block Placement Policy to support cloud
>>environments,
>>more specifically Azure.
>>- Implementation of Hadoop native libraries for Windows (compression
>>codecs, native I/O)
>>- Several reliability issues, including race-conditions, intermittent test
>>failures, resource leaks.
>>- Several new unit test cases written for the above changes
>>
>>Please find the details of the work in CHANGES.branch-trunk-win.txt -
>>Common changes<http://bit.ly/Xe7Ynv>, HDFS changes<http://bit.ly/13QOSo9>,
>>and YARN and MapReduce changes <http://bit.ly/128zzMt>. This is the work
>>ported from branch-1-win to a branch based on trunk.
>>
>>For details of the testing done, please see the thread -
>>http://bit.ly/WpavJ4. Merge patch for this is available on HADOOP-8562<
>>https://issues.apache.org/jira/browse/HADOOP-8562>.
>>
>>This was a large undertaking that involved developing code, testing the
>>entire Hadoop stack, including scale tests. This is made possible only
>>with
>>the contribution from many many folks in the community. Following people
>>contributed to this work: Ivan Mitic, Chuan Liu, Ramya Sunil, Bikas Saha,
>>Kanna Karanam, John Gordon, Brandon Li, Chris Nauroth, David Lao, Sumadhur
>>Reddy Bolli, Arpit Agarwal, Ahmed El Baz, Mike Liddell, Jing Zhao, Thejas
>>Nair, Steve Maine, Ganeshan Iyer, Raja Aluri, Giridharan Kesavan, Ramya
>>Bharathi Nimmagadda, Daryn Sharp, Arun Murthy, Tsz-Wo Nicholas Sze, Suresh
>>Srinivas and Sanjay Radia. There are many others who contributed as well
>>providing feedback and comments on numerous jiras.
>>
>>The vote will run for seven days and will end on March 5, 6:00PM PST.
>>
>>Regards,
>>Suresh
>>
>>
>>
>>
>>On Thu, Feb 7, 2013 at 6:41 PM, Mahadevan Venkatraman
>><ma...@microsoft.com>wrote:
>>
>>> It is super exciting to look at the prospect of these changes being
>>>merged
>>> to trunk. Having Windows as one of the supported Hadoop platforms is a
>>> fantastic opportunity both for the Hadoop project and Microsoft
>>>customers.
>>>
>>> This work began around a year back when a few of us started with a basic
>>> port of Hadoop on Windows. Ever since, the Hadoop team in Microsoft have
>>> made significant progress in the following areas:
>>> (PS: Some of these items are already included in Suresh's email, but
>>> including again for completeness)
>>>
>>> - Command-line scripts for the Hadoop surface area
>>> - Mapping the HDFS permissions model to Windows
>>> - Abstracted and reconciled mismatches around differences in Path
>>> semantics in Java and Windows
>>> - Native Task Controller for Windows
>>> - Implementation of a Block Placement Policy to support cloud
>>> environments, more specifically Azure.
>>> - Implementation of Hadoop native libraries for Windows (compression
>>> codecs, native I/O) - Several reliability issues, including
>>> race-conditions, intermittent test failures, resource leaks.
>>> - Several new unit test cases written for the above changes
>>>
>>> In the process, we have closely engaged with the Apache open source
>>> community and have got great support and assistance from the community
>>>in
>>> terms of contributing fixes, code review comments and commits.
>>>
>>> In addition, the Hadoop team at Microsoft has also made good progress in
>>> other projects including Hive, Pig, Sqoop, Oozie, HCat and HBase. Many
>>>of
>>> these changes have already been committed to the respective trunks with
>>> help from various committers and contributors. It is great to see the
>>> commitment of the community to support multiple platforms, and we look
>>> forward to the day when a developer/customer is able to successfully
>>>deploy
>>> a complete solution stack based on Apache Hadoop releases.
>>>
>>> Next Steps:
>>>
>>> All of the above changes are part of the Windows Azure HDInsight and
>>> HDInsight Server products from Microsoft. We have successfully
>>>on-boarded
>>> several internal customers and have been running production workloads on
>>> Windows Azure HDInsight. Our vision is to create a big data platform
>>>based
>>> on Hadoop, and we are committed to helping make Hadoop a world-class
>>> solution that anyone can use to solve their biggest data challenges.
>>>
>>> As an immediate next step, we would like to have a discussion around how
>>> we can ensure that the quality of the mainline Hadoop branches on
>>>Windows
>>> is maintained. To this end, we would like to get to the state where we
>>>have
>>> pre-checkin validation gates and nightly test runs enabled on Windows.
>>>If
>>> you have any suggestions around this, please do send an email.  We are
>>> committed to helping sustain the long-term quality of Hadoop on both
>>>Linux
>>> and Windows.
>>>
>>> We sincerely thank the community for their contribution and support so
>>> far. And hope to continue having a close engagement in the future.
>>>
>>> -Microsoft HDInsight Team
>>>
>>>
>>> -----Original Message-----
>>> From: Suresh Srinivas [mailto:suresh@hortonworks.com]
>>> Sent: Thursday, February 7, 2013 5:42 PM
>>> To: common-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org;
>>> hdfs-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
>>> Subject: Heads up - merge branch-trunk-win to trunk
>>>
>>> The support for Hadoop on Windows was proposed in HADOOP-8079<
>>> https://issues.apache.org/jira/browse/HADOOP-8079> almost a year ago.
>>>The
>>> goal was to make Hadoop natively integrated, full-featured, and
>>>performance
>>> and scalability tuned on Windows Server or Windows Azure.
>>> We are happy to announce that a lot of progress has been made in this
>>> regard.
>>>
>>> Initial work started in a feature branch, branch-1-win, based on
>>>branch-1.
>>> The details related to the work done in the branch can be seen in
>>> CHANGES.txt<
>>>
>>>http://svn.apache.org/viewvc/hadoop/common/branches/branch-1-win/CHANGES.
>>>branch-1-win.txt?view=markup
>>> >.
>>> This work has been ported to a branch, branch-trunk-win, based on trunk.
>>> Merge patch for this is available on
>>> HADOOP-8562<https://issues.apache.org/jira/browse/HADOOP-8562>
>>> .
>>>
>>> Highlights of the work done so far:
>>> 1. Necessary changes in Hadoop to run natively on Windows. These changes
>>> handle differences in platforms related to path names, process/task
>>> management etc.
>>> 2. Addition of winutils tools for managing file permissions and
>>>ownership,
>>> user group mapping, hardlinks, symbolic links, chmod, disk utilization,
>>>and
>>> process/task management.
>>> 3. Added cmd scripts equivalent to existing shell scripts
>>> hadoop-daemon.sh, start and stop scripts.
>>> 4. Addition of block placement policy implemnation to support cloud
>>> enviroment, more specifically Azure.
>>>
>>> We are very close to wrapping up the work in branch-trunk-win and
>>>getting
>>> ready for a merge. Currently the merge patch is passing close to 100% of
>>> unit tests on Linux. Soon I will call for a vote to merge this branch
>>>into
>>> trunk.
>>>
>>> Next steps:
>>> 1. Call for vote to merge branch-trunk-win to trunk, when the work
>>> completes and precommit build is clean.
>>> 2. Start a discussion on adding Jenkins precommit builds on windows and
>>> how to integrate that with the existing commit process.
>>>
>>> Let me know if you have any questions.
>>>
>>> Regards,
>>> Suresh
>>>
>>>
>>
>>
>>--
>>http://hortonworks.com/download/
>



--
Harsh J

Re: [Vote] Merge branch-trunk-win to trunk

[Harsh J <ha...@cloudera.com>]

Similar personal concern as Robert: Does this bring about a
development process change? Do new features all need to work on
Windows as well to go into trunk (i.e. immediately or eventually,
either way requires a new policy for all of us devs)? Not that anyone
would be avoiding doing that, I just ask cause it could impact time
and effort required for any major undertaking.

While most of the project today is cross-platform (via Java, etc.,
which thankfully remove path handling problems and the sorts),
performance improvements at least are going the native side these
days, which is where I see this have some impact. We've not been
perfectly successful in having the natives continuously work on
Solaris/etc. in the past, mainly due to the platform focus of the
majority (not all) of devs working on the project(s). Some form of a
development policy here would ensure proper Windows support for the
features we intend to ship along are not very divergent such that we
end up having to maintain docs as well, detailing each task yet to be
done (these tend to grow if allowed).

Useful to also note from another OSS project KDE, that their working
builds of Windows are usually on 1-2 releases of the past (i.e. for
example, KDE release is currently at 4.10, but the last released
Windows port is still 4.8 today). KDE uses Qt, which is cross-platform
by itself, but there's still a port team and a ported release
maintained separately (but under the same org.) due to the major
development happening on Linux. Same case for *BSD as well. My own
patches there at some point have caused trouble cause I did something
that I only tested on one platform, and about a bit later things got
revised to support the other ones where it was unnecessarily breaking.

Or if am being too concerned about feature/performance divergence, let me know.

On Wed, Feb 27, 2013 at 9:47 PM, Robert Evans <ev...@yahoo-inc.com> wrote:
> After this is merged in is Windows still going to be a second class
> citizen but happens to work for more than just development or is it a
> fully supported platform where if something breaks it can block a release?
>  How do we as a community intend to keep Windows support from breaking?
> We don't have any Jenkins slaves to be able to run nightly tests to
> validate everything still compiles/runs.  This is not a blocker for me
> because we often rely on individuals and groups to test Hadoop, but I do
> think we need to have this discussion before we put it in.
>
> --Bobby
>
> On 2/26/13 4:55 PM, "Suresh Srinivas" <su...@hortonworks.com> wrote:
>
>>I had posted heads up about merging branch-trunk-win to trunk on Feb 8th.
>>I
>>am happy to announce that we are ready for the merge.
>>
>>Here is a brief recap on the highlights of the work done:
>>- Command-line scripts for the Hadoop surface area
>>- Mapping the HDFS permissions model to Windows
>>- Abstracted and reconciled mismatches around differences in Path
>>semantics
>>in Java and Windows
>>- Native Task Controller for Windows
>>- Implementation of a Block Placement Policy to support cloud
>>environments,
>>more specifically Azure.
>>- Implementation of Hadoop native libraries for Windows (compression
>>codecs, native I/O)
>>- Several reliability issues, including race-conditions, intermittent test
>>failures, resource leaks.
>>- Several new unit test cases written for the above changes
>>
>>Please find the details of the work in CHANGES.branch-trunk-win.txt -
>>Common changes<http://bit.ly/Xe7Ynv>, HDFS changes<http://bit.ly/13QOSo9>,
>>and YARN and MapReduce changes <http://bit.ly/128zzMt>. This is the work
>>ported from branch-1-win to a branch based on trunk.
>>
>>For details of the testing done, please see the thread -
>>http://bit.ly/WpavJ4. Merge patch for this is available on HADOOP-8562<
>>https://issues.apache.org/jira/browse/HADOOP-8562>.
>>
>>This was a large undertaking that involved developing code, testing the
>>entire Hadoop stack, including scale tests. This is made possible only
>>with
>>the contribution from many many folks in the community. Following people
>>contributed to this work: Ivan Mitic, Chuan Liu, Ramya Sunil, Bikas Saha,
>>Kanna Karanam, John Gordon, Brandon Li, Chris Nauroth, David Lao, Sumadhur
>>Reddy Bolli, Arpit Agarwal, Ahmed El Baz, Mike Liddell, Jing Zhao, Thejas
>>Nair, Steve Maine, Ganeshan Iyer, Raja Aluri, Giridharan Kesavan, Ramya
>>Bharathi Nimmagadda, Daryn Sharp, Arun Murthy, Tsz-Wo Nicholas Sze, Suresh
>>Srinivas and Sanjay Radia. There are many others who contributed as well
>>providing feedback and comments on numerous jiras.
>>
>>The vote will run for seven days and will end on March 5, 6:00PM PST.
>>
>>Regards,
>>Suresh
>>
>>
>>
>>
>>On Thu, Feb 7, 2013 at 6:41 PM, Mahadevan Venkatraman
>><ma...@microsoft.com>wrote:
>>
>>> It is super exciting to look at the prospect of these changes being
>>>merged
>>> to trunk. Having Windows as one of the supported Hadoop platforms is a
>>> fantastic opportunity both for the Hadoop project and Microsoft
>>>customers.
>>>
>>> This work began around a year back when a few of us started with a basic
>>> port of Hadoop on Windows. Ever since, the Hadoop team in Microsoft have
>>> made significant progress in the following areas:
>>> (PS: Some of these items are already included in Suresh's email, but
>>> including again for completeness)
>>>
>>> - Command-line scripts for the Hadoop surface area
>>> - Mapping the HDFS permissions model to Windows
>>> - Abstracted and reconciled mismatches around differences in Path
>>> semantics in Java and Windows
>>> - Native Task Controller for Windows
>>> - Implementation of a Block Placement Policy to support cloud
>>> environments, more specifically Azure.
>>> - Implementation of Hadoop native libraries for Windows (compression
>>> codecs, native I/O) - Several reliability issues, including
>>> race-conditions, intermittent test failures, resource leaks.
>>> - Several new unit test cases written for the above changes
>>>
>>> In the process, we have closely engaged with the Apache open source
>>> community and have got great support and assistance from the community
>>>in
>>> terms of contributing fixes, code review comments and commits.
>>>
>>> In addition, the Hadoop team at Microsoft has also made good progress in
>>> other projects including Hive, Pig, Sqoop, Oozie, HCat and HBase. Many
>>>of
>>> these changes have already been committed to the respective trunks with
>>> help from various committers and contributors. It is great to see the
>>> commitment of the community to support multiple platforms, and we look
>>> forward to the day when a developer/customer is able to successfully
>>>deploy
>>> a complete solution stack based on Apache Hadoop releases.
>>>
>>> Next Steps:
>>>
>>> All of the above changes are part of the Windows Azure HDInsight and
>>> HDInsight Server products from Microsoft. We have successfully
>>>on-boarded
>>> several internal customers and have been running production workloads on
>>> Windows Azure HDInsight. Our vision is to create a big data platform
>>>based
>>> on Hadoop, and we are committed to helping make Hadoop a world-class
>>> solution that anyone can use to solve their biggest data challenges.
>>>
>>> As an immediate next step, we would like to have a discussion around how
>>> we can ensure that the quality of the mainline Hadoop branches on
>>>Windows
>>> is maintained. To this end, we would like to get to the state where we
>>>have
>>> pre-checkin validation gates and nightly test runs enabled on Windows.
>>>If
>>> you have any suggestions around this, please do send an email.  We are
>>> committed to helping sustain the long-term quality of Hadoop on both
>>>Linux
>>> and Windows.
>>>
>>> We sincerely thank the community for their contribution and support so
>>> far. And hope to continue having a close engagement in the future.
>>>
>>> -Microsoft HDInsight Team
>>>
>>>
>>> -----Original Message-----
>>> From: Suresh Srinivas [mailto:suresh@hortonworks.com]
>>> Sent: Thursday, February 7, 2013 5:42 PM
>>> To: common-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org;
>>> hdfs-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
>>> Subject: Heads up - merge branch-trunk-win to trunk
>>>
>>> The support for Hadoop on Windows was proposed in HADOOP-8079<
>>> https://issues.apache.org/jira/browse/HADOOP-8079> almost a year ago.
>>>The
>>> goal was to make Hadoop natively integrated, full-featured, and
>>>performance
>>> and scalability tuned on Windows Server or Windows Azure.
>>> We are happy to announce that a lot of progress has been made in this
>>> regard.
>>>
>>> Initial work started in a feature branch, branch-1-win, based on
>>>branch-1.
>>> The details related to the work done in the branch can be seen in
>>> CHANGES.txt<
>>>
>>>http://svn.apache.org/viewvc/hadoop/common/branches/branch-1-win/CHANGES.
>>>branch-1-win.txt?view=markup
>>> >.
>>> This work has been ported to a branch, branch-trunk-win, based on trunk.
>>> Merge patch for this is available on
>>> HADOOP-8562<https://issues.apache.org/jira/browse/HADOOP-8562>
>>> .
>>>
>>> Highlights of the work done so far:
>>> 1. Necessary changes in Hadoop to run natively on Windows. These changes
>>> handle differences in platforms related to path names, process/task
>>> management etc.
>>> 2. Addition of winutils tools for managing file permissions and
>>>ownership,
>>> user group mapping, hardlinks, symbolic links, chmod, disk utilization,
>>>and
>>> process/task management.
>>> 3. Added cmd scripts equivalent to existing shell scripts
>>> hadoop-daemon.sh, start and stop scripts.
>>> 4. Addition of block placement policy implemnation to support cloud
>>> enviroment, more specifically Azure.
>>>
>>> We are very close to wrapping up the work in branch-trunk-win and
>>>getting
>>> ready for a merge. Currently the merge patch is passing close to 100% of
>>> unit tests on Linux. Soon I will call for a vote to merge this branch
>>>into
>>> trunk.
>>>
>>> Next steps:
>>> 1. Call for vote to merge branch-trunk-win to trunk, when the work
>>> completes and precommit build is clean.
>>> 2. Start a discussion on adding Jenkins precommit builds on windows and
>>> how to integrate that with the existing commit process.
>>>
>>> Let me know if you have any questions.
>>>
>>> Regards,
>>> Suresh
>>>
>>>
>>
>>
>>--
>>http://hortonworks.com/download/
>



--
Harsh J

Re: [Vote] Merge branch-trunk-win to trunk

[Robert Evans <ev...@yahoo-inc.com>]

After this is merged in is Windows still going to be a second class
citizen but happens to work for more than just development or is it a
fully supported platform where if something breaks it can block a release?
 How do we as a community intend to keep Windows support from breaking?
We don't have any Jenkins slaves to be able to run nightly tests to
validate everything still compiles/runs.  This is not a blocker for me
because we often rely on individuals and groups to test Hadoop, but I do
think we need to have this discussion before we put it in.

--Bobby

On 2/26/13 4:55 PM, "Suresh Srinivas" <su...@hortonworks.com> wrote:

>I had posted heads up about merging branch-trunk-win to trunk on Feb 8th.
>I
>am happy to announce that we are ready for the merge.
>
>Here is a brief recap on the highlights of the work done:
>- Command-line scripts for the Hadoop surface area
>- Mapping the HDFS permissions model to Windows
>- Abstracted and reconciled mismatches around differences in Path
>semantics
>in Java and Windows
>- Native Task Controller for Windows
>- Implementation of a Block Placement Policy to support cloud
>environments,
>more specifically Azure.
>- Implementation of Hadoop native libraries for Windows (compression
>codecs, native I/O)
>- Several reliability issues, including race-conditions, intermittent test
>failures, resource leaks.
>- Several new unit test cases written for the above changes
>
>Please find the details of the work in CHANGES.branch-trunk-win.txt -
>Common changes<http://bit.ly/Xe7Ynv>, HDFS changes<http://bit.ly/13QOSo9>,
>and YARN and MapReduce changes <http://bit.ly/128zzMt>. This is the work
>ported from branch-1-win to a branch based on trunk.
>
>For details of the testing done, please see the thread -
>http://bit.ly/WpavJ4. Merge patch for this is available on HADOOP-8562<
>https://issues.apache.org/jira/browse/HADOOP-8562>.
>
>This was a large undertaking that involved developing code, testing the
>entire Hadoop stack, including scale tests. This is made possible only
>with
>the contribution from many many folks in the community. Following people
>contributed to this work: Ivan Mitic, Chuan Liu, Ramya Sunil, Bikas Saha,
>Kanna Karanam, John Gordon, Brandon Li, Chris Nauroth, David Lao, Sumadhur
>Reddy Bolli, Arpit Agarwal, Ahmed El Baz, Mike Liddell, Jing Zhao, Thejas
>Nair, Steve Maine, Ganeshan Iyer, Raja Aluri, Giridharan Kesavan, Ramya
>Bharathi Nimmagadda, Daryn Sharp, Arun Murthy, Tsz-Wo Nicholas Sze, Suresh
>Srinivas and Sanjay Radia. There are many others who contributed as well
>providing feedback and comments on numerous jiras.
>
>The vote will run for seven days and will end on March 5, 6:00PM PST.
>
>Regards,
>Suresh
>
>
>
>
>On Thu, Feb 7, 2013 at 6:41 PM, Mahadevan Venkatraman
><ma...@microsoft.com>wrote:
>
>> It is super exciting to look at the prospect of these changes being
>>merged
>> to trunk. Having Windows as one of the supported Hadoop platforms is a
>> fantastic opportunity both for the Hadoop project and Microsoft
>>customers.
>>
>> This work began around a year back when a few of us started with a basic
>> port of Hadoop on Windows. Ever since, the Hadoop team in Microsoft have
>> made significant progress in the following areas:
>> (PS: Some of these items are already included in Suresh's email, but
>> including again for completeness)
>>
>> - Command-line scripts for the Hadoop surface area
>> - Mapping the HDFS permissions model to Windows
>> - Abstracted and reconciled mismatches around differences in Path
>> semantics in Java and Windows
>> - Native Task Controller for Windows
>> - Implementation of a Block Placement Policy to support cloud
>> environments, more specifically Azure.
>> - Implementation of Hadoop native libraries for Windows (compression
>> codecs, native I/O) - Several reliability issues, including
>> race-conditions, intermittent test failures, resource leaks.
>> - Several new unit test cases written for the above changes
>>
>> In the process, we have closely engaged with the Apache open source
>> community and have got great support and assistance from the community
>>in
>> terms of contributing fixes, code review comments and commits.
>>
>> In addition, the Hadoop team at Microsoft has also made good progress in
>> other projects including Hive, Pig, Sqoop, Oozie, HCat and HBase. Many
>>of
>> these changes have already been committed to the respective trunks with
>> help from various committers and contributors. It is great to see the
>> commitment of the community to support multiple platforms, and we look
>> forward to the day when a developer/customer is able to successfully
>>deploy
>> a complete solution stack based on Apache Hadoop releases.
>>
>> Next Steps:
>>
>> All of the above changes are part of the Windows Azure HDInsight and
>> HDInsight Server products from Microsoft. We have successfully
>>on-boarded
>> several internal customers and have been running production workloads on
>> Windows Azure HDInsight. Our vision is to create a big data platform
>>based
>> on Hadoop, and we are committed to helping make Hadoop a world-class
>> solution that anyone can use to solve their biggest data challenges.
>>
>> As an immediate next step, we would like to have a discussion around how
>> we can ensure that the quality of the mainline Hadoop branches on
>>Windows
>> is maintained. To this end, we would like to get to the state where we
>>have
>> pre-checkin validation gates and nightly test runs enabled on Windows.
>>If
>> you have any suggestions around this, please do send an email.  We are
>> committed to helping sustain the long-term quality of Hadoop on both
>>Linux
>> and Windows.
>>
>> We sincerely thank the community for their contribution and support so
>> far. And hope to continue having a close engagement in the future.
>>
>> -Microsoft HDInsight Team
>>
>>
>> -----Original Message-----
>> From: Suresh Srinivas [mailto:suresh@hortonworks.com]
>> Sent: Thursday, February 7, 2013 5:42 PM
>> To: common-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org;
>> hdfs-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
>> Subject: Heads up - merge branch-trunk-win to trunk
>>
>> The support for Hadoop on Windows was proposed in HADOOP-8079<
>> https://issues.apache.org/jira/browse/HADOOP-8079> almost a year ago.
>>The
>> goal was to make Hadoop natively integrated, full-featured, and
>>performance
>> and scalability tuned on Windows Server or Windows Azure.
>> We are happy to announce that a lot of progress has been made in this
>> regard.
>>
>> Initial work started in a feature branch, branch-1-win, based on
>>branch-1.
>> The details related to the work done in the branch can be seen in
>> CHANGES.txt<
>> 
>>http://svn.apache.org/viewvc/hadoop/common/branches/branch-1-win/CHANGES.
>>branch-1-win.txt?view=markup
>> >.
>> This work has been ported to a branch, branch-trunk-win, based on trunk.
>> Merge patch for this is available on
>> HADOOP-8562<https://issues.apache.org/jira/browse/HADOOP-8562>
>> .
>>
>> Highlights of the work done so far:
>> 1. Necessary changes in Hadoop to run natively on Windows. These changes
>> handle differences in platforms related to path names, process/task
>> management etc.
>> 2. Addition of winutils tools for managing file permissions and
>>ownership,
>> user group mapping, hardlinks, symbolic links, chmod, disk utilization,
>>and
>> process/task management.
>> 3. Added cmd scripts equivalent to existing shell scripts
>> hadoop-daemon.sh, start and stop scripts.
>> 4. Addition of block placement policy implemnation to support cloud
>> enviroment, more specifically Azure.
>>
>> We are very close to wrapping up the work in branch-trunk-win and
>>getting
>> ready for a merge. Currently the merge patch is passing close to 100% of
>> unit tests on Linux. Soon I will call for a vote to merge this branch
>>into
>> trunk.
>>
>> Next steps:
>> 1. Call for vote to merge branch-trunk-win to trunk, when the work
>> completes and precommit build is clean.
>> 2. Start a discussion on adding Jenkins precommit builds on windows and
>> how to integrate that with the existing commit process.
>>
>> Let me know if you have any questions.
>>
>> Regards,
>> Suresh
>>
>>
>
>
>-- 
>http://hortonworks.com/download/

Re: [Vote] Merge branch-trunk-win to trunk

[Robert Evans <ev...@yahoo-inc.com>]

After this is merged in is Windows still going to be a second class
citizen but happens to work for more than just development or is it a
fully supported platform where if something breaks it can block a release?
 How do we as a community intend to keep Windows support from breaking?
We don't have any Jenkins slaves to be able to run nightly tests to
validate everything still compiles/runs.  This is not a blocker for me
because we often rely on individuals and groups to test Hadoop, but I do
think we need to have this discussion before we put it in.

--Bobby

On 2/26/13 4:55 PM, "Suresh Srinivas" <su...@hortonworks.com> wrote:

>I had posted heads up about merging branch-trunk-win to trunk on Feb 8th.
>I
>am happy to announce that we are ready for the merge.
>
>Here is a brief recap on the highlights of the work done:
>- Command-line scripts for the Hadoop surface area
>- Mapping the HDFS permissions model to Windows
>- Abstracted and reconciled mismatches around differences in Path
>semantics
>in Java and Windows
>- Native Task Controller for Windows
>- Implementation of a Block Placement Policy to support cloud
>environments,
>more specifically Azure.
>- Implementation of Hadoop native libraries for Windows (compression
>codecs, native I/O)
>- Several reliability issues, including race-conditions, intermittent test
>failures, resource leaks.
>- Several new unit test cases written for the above changes
>
>Please find the details of the work in CHANGES.branch-trunk-win.txt -
>Common changes<http://bit.ly/Xe7Ynv>, HDFS changes<http://bit.ly/13QOSo9>,
>and YARN and MapReduce changes <http://bit.ly/128zzMt>. This is the work
>ported from branch-1-win to a branch based on trunk.
>
>For details of the testing done, please see the thread -
>http://bit.ly/WpavJ4. Merge patch for this is available on HADOOP-8562<
>https://issues.apache.org/jira/browse/HADOOP-8562>.
>
>This was a large undertaking that involved developing code, testing the
>entire Hadoop stack, including scale tests. This is made possible only
>with
>the contribution from many many folks in the community. Following people
>contributed to this work: Ivan Mitic, Chuan Liu, Ramya Sunil, Bikas Saha,
>Kanna Karanam, John Gordon, Brandon Li, Chris Nauroth, David Lao, Sumadhur
>Reddy Bolli, Arpit Agarwal, Ahmed El Baz, Mike Liddell, Jing Zhao, Thejas
>Nair, Steve Maine, Ganeshan Iyer, Raja Aluri, Giridharan Kesavan, Ramya
>Bharathi Nimmagadda, Daryn Sharp, Arun Murthy, Tsz-Wo Nicholas Sze, Suresh
>Srinivas and Sanjay Radia. There are many others who contributed as well
>providing feedback and comments on numerous jiras.
>
>The vote will run for seven days and will end on March 5, 6:00PM PST.
>
>Regards,
>Suresh
>
>
>
>
>On Thu, Feb 7, 2013 at 6:41 PM, Mahadevan Venkatraman
><ma...@microsoft.com>wrote:
>
>> It is super exciting to look at the prospect of these changes being
>>merged
>> to trunk. Having Windows as one of the supported Hadoop platforms is a
>> fantastic opportunity both for the Hadoop project and Microsoft
>>customers.
>>
>> This work began around a year back when a few of us started with a basic
>> port of Hadoop on Windows. Ever since, the Hadoop team in Microsoft have
>> made significant progress in the following areas:
>> (PS: Some of these items are already included in Suresh's email, but
>> including again for completeness)
>>
>> - Command-line scripts for the Hadoop surface area
>> - Mapping the HDFS permissions model to Windows
>> - Abstracted and reconciled mismatches around differences in Path
>> semantics in Java and Windows
>> - Native Task Controller for Windows
>> - Implementation of a Block Placement Policy to support cloud
>> environments, more specifically Azure.
>> - Implementation of Hadoop native libraries for Windows (compression
>> codecs, native I/O) - Several reliability issues, including
>> race-conditions, intermittent test failures, resource leaks.
>> - Several new unit test cases written for the above changes
>>
>> In the process, we have closely engaged with the Apache open source
>> community and have got great support and assistance from the community
>>in
>> terms of contributing fixes, code review comments and commits.
>>
>> In addition, the Hadoop team at Microsoft has also made good progress in
>> other projects including Hive, Pig, Sqoop, Oozie, HCat and HBase. Many
>>of
>> these changes have already been committed to the respective trunks with
>> help from various committers and contributors. It is great to see the
>> commitment of the community to support multiple platforms, and we look
>> forward to the day when a developer/customer is able to successfully
>>deploy
>> a complete solution stack based on Apache Hadoop releases.
>>
>> Next Steps:
>>
>> All of the above changes are part of the Windows Azure HDInsight and
>> HDInsight Server products from Microsoft. We have successfully
>>on-boarded
>> several internal customers and have been running production workloads on
>> Windows Azure HDInsight. Our vision is to create a big data platform
>>based
>> on Hadoop, and we are committed to helping make Hadoop a world-class
>> solution that anyone can use to solve their biggest data challenges.
>>
>> As an immediate next step, we would like to have a discussion around how
>> we can ensure that the quality of the mainline Hadoop branches on
>>Windows
>> is maintained. To this end, we would like to get to the state where we
>>have
>> pre-checkin validation gates and nightly test runs enabled on Windows.
>>If
>> you have any suggestions around this, please do send an email.  We are
>> committed to helping sustain the long-term quality of Hadoop on both
>>Linux
>> and Windows.
>>
>> We sincerely thank the community for their contribution and support so
>> far. And hope to continue having a close engagement in the future.
>>
>> -Microsoft HDInsight Team
>>
>>
>> -----Original Message-----
>> From: Suresh Srinivas [mailto:suresh@hortonworks.com]
>> Sent: Thursday, February 7, 2013 5:42 PM
>> To: common-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org;
>> hdfs-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
>> Subject: Heads up - merge branch-trunk-win to trunk
>>
>> The support for Hadoop on Windows was proposed in HADOOP-8079<
>> https://issues.apache.org/jira/browse/HADOOP-8079> almost a year ago.
>>The
>> goal was to make Hadoop natively integrated, full-featured, and
>>performance
>> and scalability tuned on Windows Server or Windows Azure.
>> We are happy to announce that a lot of progress has been made in this
>> regard.
>>
>> Initial work started in a feature branch, branch-1-win, based on
>>branch-1.
>> The details related to the work done in the branch can be seen in
>> CHANGES.txt<
>> 
>>http://svn.apache.org/viewvc/hadoop/common/branches/branch-1-win/CHANGES.
>>branch-1-win.txt?view=markup
>> >.
>> This work has been ported to a branch, branch-trunk-win, based on trunk.
>> Merge patch for this is available on
>> HADOOP-8562<https://issues.apache.org/jira/browse/HADOOP-8562>
>> .
>>
>> Highlights of the work done so far:
>> 1. Necessary changes in Hadoop to run natively on Windows. These changes
>> handle differences in platforms related to path names, process/task
>> management etc.
>> 2. Addition of winutils tools for managing file permissions and
>>ownership,
>> user group mapping, hardlinks, symbolic links, chmod, disk utilization,
>>and
>> process/task management.
>> 3. Added cmd scripts equivalent to existing shell scripts
>> hadoop-daemon.sh, start and stop scripts.
>> 4. Addition of block placement policy implemnation to support cloud
>> enviroment, more specifically Azure.
>>
>> We are very close to wrapping up the work in branch-trunk-win and
>>getting
>> ready for a merge. Currently the merge patch is passing close to 100% of
>> unit tests on Linux. Soon I will call for a vote to merge this branch
>>into
>> trunk.
>>
>> Next steps:
>> 1. Call for vote to merge branch-trunk-win to trunk, when the work
>> completes and precommit build is clean.
>> 2. Start a discussion on adding Jenkins precommit builds on windows and
>> how to integrate that with the existing commit process.
>>
>> Let me know if you have any questions.
>>
>> Regards,
>> Suresh
>>
>>
>
>
>-- 
>http://hortonworks.com/download/

Re: [Vote] Merge branch-trunk-win to trunk

[Robert Evans <ev...@yahoo-inc.com>]

After this is merged in is Windows still going to be a second class
citizen but happens to work for more than just development or is it a
fully supported platform where if something breaks it can block a release?
 How do we as a community intend to keep Windows support from breaking?
We don't have any Jenkins slaves to be able to run nightly tests to
validate everything still compiles/runs.  This is not a blocker for me
because we often rely on individuals and groups to test Hadoop, but I do
think we need to have this discussion before we put it in.

--Bobby

On 2/26/13 4:55 PM, "Suresh Srinivas" <su...@hortonworks.com> wrote:

>I had posted heads up about merging branch-trunk-win to trunk on Feb 8th.
>I
>am happy to announce that we are ready for the merge.
>
>Here is a brief recap on the highlights of the work done:
>- Command-line scripts for the Hadoop surface area
>- Mapping the HDFS permissions model to Windows
>- Abstracted and reconciled mismatches around differences in Path
>semantics
>in Java and Windows
>- Native Task Controller for Windows
>- Implementation of a Block Placement Policy to support cloud
>environments,
>more specifically Azure.
>- Implementation of Hadoop native libraries for Windows (compression
>codecs, native I/O)
>- Several reliability issues, including race-conditions, intermittent test
>failures, resource leaks.
>- Several new unit test cases written for the above changes
>
>Please find the details of the work in CHANGES.branch-trunk-win.txt -
>Common changes<http://bit.ly/Xe7Ynv>, HDFS changes<http://bit.ly/13QOSo9>,
>and YARN and MapReduce changes <http://bit.ly/128zzMt>. This is the work
>ported from branch-1-win to a branch based on trunk.
>
>For details of the testing done, please see the thread -
>http://bit.ly/WpavJ4. Merge patch for this is available on HADOOP-8562<
>https://issues.apache.org/jira/browse/HADOOP-8562>.
>
>This was a large undertaking that involved developing code, testing the
>entire Hadoop stack, including scale tests. This is made possible only
>with
>the contribution from many many folks in the community. Following people
>contributed to this work: Ivan Mitic, Chuan Liu, Ramya Sunil, Bikas Saha,
>Kanna Karanam, John Gordon, Brandon Li, Chris Nauroth, David Lao, Sumadhur
>Reddy Bolli, Arpit Agarwal, Ahmed El Baz, Mike Liddell, Jing Zhao, Thejas
>Nair, Steve Maine, Ganeshan Iyer, Raja Aluri, Giridharan Kesavan, Ramya
>Bharathi Nimmagadda, Daryn Sharp, Arun Murthy, Tsz-Wo Nicholas Sze, Suresh
>Srinivas and Sanjay Radia. There are many others who contributed as well
>providing feedback and comments on numerous jiras.
>
>The vote will run for seven days and will end on March 5, 6:00PM PST.
>
>Regards,
>Suresh
>
>
>
>
>On Thu, Feb 7, 2013 at 6:41 PM, Mahadevan Venkatraman
><ma...@microsoft.com>wrote:
>
>> It is super exciting to look at the prospect of these changes being
>>merged
>> to trunk. Having Windows as one of the supported Hadoop platforms is a
>> fantastic opportunity both for the Hadoop project and Microsoft
>>customers.
>>
>> This work began around a year back when a few of us started with a basic
>> port of Hadoop on Windows. Ever since, the Hadoop team in Microsoft have
>> made significant progress in the following areas:
>> (PS: Some of these items are already included in Suresh's email, but
>> including again for completeness)
>>
>> - Command-line scripts for the Hadoop surface area
>> - Mapping the HDFS permissions model to Windows
>> - Abstracted and reconciled mismatches around differences in Path
>> semantics in Java and Windows
>> - Native Task Controller for Windows
>> - Implementation of a Block Placement Policy to support cloud
>> environments, more specifically Azure.
>> - Implementation of Hadoop native libraries for Windows (compression
>> codecs, native I/O) - Several reliability issues, including
>> race-conditions, intermittent test failures, resource leaks.
>> - Several new unit test cases written for the above changes
>>
>> In the process, we have closely engaged with the Apache open source
>> community and have got great support and assistance from the community
>>in
>> terms of contributing fixes, code review comments and commits.
>>
>> In addition, the Hadoop team at Microsoft has also made good progress in
>> other projects including Hive, Pig, Sqoop, Oozie, HCat and HBase. Many
>>of
>> these changes have already been committed to the respective trunks with
>> help from various committers and contributors. It is great to see the
>> commitment of the community to support multiple platforms, and we look
>> forward to the day when a developer/customer is able to successfully
>>deploy
>> a complete solution stack based on Apache Hadoop releases.
>>
>> Next Steps:
>>
>> All of the above changes are part of the Windows Azure HDInsight and
>> HDInsight Server products from Microsoft. We have successfully
>>on-boarded
>> several internal customers and have been running production workloads on
>> Windows Azure HDInsight. Our vision is to create a big data platform
>>based
>> on Hadoop, and we are committed to helping make Hadoop a world-class
>> solution that anyone can use to solve their biggest data challenges.
>>
>> As an immediate next step, we would like to have a discussion around how
>> we can ensure that the quality of the mainline Hadoop branches on
>>Windows
>> is maintained. To this end, we would like to get to the state where we
>>have
>> pre-checkin validation gates and nightly test runs enabled on Windows.
>>If
>> you have any suggestions around this, please do send an email.  We are
>> committed to helping sustain the long-term quality of Hadoop on both
>>Linux
>> and Windows.
>>
>> We sincerely thank the community for their contribution and support so
>> far. And hope to continue having a close engagement in the future.
>>
>> -Microsoft HDInsight Team
>>
>>
>> -----Original Message-----
>> From: Suresh Srinivas [mailto:suresh@hortonworks.com]
>> Sent: Thursday, February 7, 2013 5:42 PM
>> To: common-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org;
>> hdfs-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
>> Subject: Heads up - merge branch-trunk-win to trunk
>>
>> The support for Hadoop on Windows was proposed in HADOOP-8079<
>> https://issues.apache.org/jira/browse/HADOOP-8079> almost a year ago.
>>The
>> goal was to make Hadoop natively integrated, full-featured, and
>>performance
>> and scalability tuned on Windows Server or Windows Azure.
>> We are happy to announce that a lot of progress has been made in this
>> regard.
>>
>> Initial work started in a feature branch, branch-1-win, based on
>>branch-1.
>> The details related to the work done in the branch can be seen in
>> CHANGES.txt<
>> 
>>http://svn.apache.org/viewvc/hadoop/common/branches/branch-1-win/CHANGES.
>>branch-1-win.txt?view=markup
>> >.
>> This work has been ported to a branch, branch-trunk-win, based on trunk.
>> Merge patch for this is available on
>> HADOOP-8562<https://issues.apache.org/jira/browse/HADOOP-8562>
>> .
>>
>> Highlights of the work done so far:
>> 1. Necessary changes in Hadoop to run natively on Windows. These changes
>> handle differences in platforms related to path names, process/task
>> management etc.
>> 2. Addition of winutils tools for managing file permissions and
>>ownership,
>> user group mapping, hardlinks, symbolic links, chmod, disk utilization,
>>and
>> process/task management.
>> 3. Added cmd scripts equivalent to existing shell scripts
>> hadoop-daemon.sh, start and stop scripts.
>> 4. Addition of block placement policy implemnation to support cloud
>> enviroment, more specifically Azure.
>>
>> We are very close to wrapping up the work in branch-trunk-win and
>>getting
>> ready for a merge. Currently the merge patch is passing close to 100% of
>> unit tests on Linux. Soon I will call for a vote to merge this branch
>>into
>> trunk.
>>
>> Next steps:
>> 1. Call for vote to merge branch-trunk-win to trunk, when the work
>> completes and precommit build is clean.
>> 2. Start a discussion on adding Jenkins precommit builds on windows and
>> how to integrate that with the existing commit process.
>>
>> Let me know if you have any questions.
>>
>> Regards,
>> Suresh
>>
>>
>
>
>-- 
>http://hortonworks.com/download/

Re: [Vote] Merge branch-trunk-win to trunk

[Robert Evans <ev...@yahoo-inc.com>]

After this is merged in is Windows still going to be a second class
citizen but happens to work for more than just development or is it a
fully supported platform where if something breaks it can block a release?
 How do we as a community intend to keep Windows support from breaking?
We don't have any Jenkins slaves to be able to run nightly tests to
validate everything still compiles/runs.  This is not a blocker for me
because we often rely on individuals and groups to test Hadoop, but I do
think we need to have this discussion before we put it in.

--Bobby

On 2/26/13 4:55 PM, "Suresh Srinivas" <su...@hortonworks.com> wrote:

>I had posted heads up about merging branch-trunk-win to trunk on Feb 8th.
>I
>am happy to announce that we are ready for the merge.
>
>Here is a brief recap on the highlights of the work done:
>- Command-line scripts for the Hadoop surface area
>- Mapping the HDFS permissions model to Windows
>- Abstracted and reconciled mismatches around differences in Path
>semantics
>in Java and Windows
>- Native Task Controller for Windows
>- Implementation of a Block Placement Policy to support cloud
>environments,
>more specifically Azure.
>- Implementation of Hadoop native libraries for Windows (compression
>codecs, native I/O)
>- Several reliability issues, including race-conditions, intermittent test
>failures, resource leaks.
>- Several new unit test cases written for the above changes
>
>Please find the details of the work in CHANGES.branch-trunk-win.txt -
>Common changes<http://bit.ly/Xe7Ynv>, HDFS changes<http://bit.ly/13QOSo9>,
>and YARN and MapReduce changes <http://bit.ly/128zzMt>. This is the work
>ported from branch-1-win to a branch based on trunk.
>
>For details of the testing done, please see the thread -
>http://bit.ly/WpavJ4. Merge patch for this is available on HADOOP-8562<
>https://issues.apache.org/jira/browse/HADOOP-8562>.
>
>This was a large undertaking that involved developing code, testing the
>entire Hadoop stack, including scale tests. This is made possible only
>with
>the contribution from many many folks in the community. Following people
>contributed to this work: Ivan Mitic, Chuan Liu, Ramya Sunil, Bikas Saha,
>Kanna Karanam, John Gordon, Brandon Li, Chris Nauroth, David Lao, Sumadhur
>Reddy Bolli, Arpit Agarwal, Ahmed El Baz, Mike Liddell, Jing Zhao, Thejas
>Nair, Steve Maine, Ganeshan Iyer, Raja Aluri, Giridharan Kesavan, Ramya
>Bharathi Nimmagadda, Daryn Sharp, Arun Murthy, Tsz-Wo Nicholas Sze, Suresh
>Srinivas and Sanjay Radia. There are many others who contributed as well
>providing feedback and comments on numerous jiras.
>
>The vote will run for seven days and will end on March 5, 6:00PM PST.
>
>Regards,
>Suresh
>
>
>
>
>On Thu, Feb 7, 2013 at 6:41 PM, Mahadevan Venkatraman
><ma...@microsoft.com>wrote:
>
>> It is super exciting to look at the prospect of these changes being
>>merged
>> to trunk. Having Windows as one of the supported Hadoop platforms is a
>> fantastic opportunity both for the Hadoop project and Microsoft
>>customers.
>>
>> This work began around a year back when a few of us started with a basic
>> port of Hadoop on Windows. Ever since, the Hadoop team in Microsoft have
>> made significant progress in the following areas:
>> (PS: Some of these items are already included in Suresh's email, but
>> including again for completeness)
>>
>> - Command-line scripts for the Hadoop surface area
>> - Mapping the HDFS permissions model to Windows
>> - Abstracted and reconciled mismatches around differences in Path
>> semantics in Java and Windows
>> - Native Task Controller for Windows
>> - Implementation of a Block Placement Policy to support cloud
>> environments, more specifically Azure.
>> - Implementation of Hadoop native libraries for Windows (compression
>> codecs, native I/O) - Several reliability issues, including
>> race-conditions, intermittent test failures, resource leaks.
>> - Several new unit test cases written for the above changes
>>
>> In the process, we have closely engaged with the Apache open source
>> community and have got great support and assistance from the community
>>in
>> terms of contributing fixes, code review comments and commits.
>>
>> In addition, the Hadoop team at Microsoft has also made good progress in
>> other projects including Hive, Pig, Sqoop, Oozie, HCat and HBase. Many
>>of
>> these changes have already been committed to the respective trunks with
>> help from various committers and contributors. It is great to see the
>> commitment of the community to support multiple platforms, and we look
>> forward to the day when a developer/customer is able to successfully
>>deploy
>> a complete solution stack based on Apache Hadoop releases.
>>
>> Next Steps:
>>
>> All of the above changes are part of the Windows Azure HDInsight and
>> HDInsight Server products from Microsoft. We have successfully
>>on-boarded
>> several internal customers and have been running production workloads on
>> Windows Azure HDInsight. Our vision is to create a big data platform
>>based
>> on Hadoop, and we are committed to helping make Hadoop a world-class
>> solution that anyone can use to solve their biggest data challenges.
>>
>> As an immediate next step, we would like to have a discussion around how
>> we can ensure that the quality of the mainline Hadoop branches on
>>Windows
>> is maintained. To this end, we would like to get to the state where we
>>have
>> pre-checkin validation gates and nightly test runs enabled on Windows.
>>If
>> you have any suggestions around this, please do send an email.  We are
>> committed to helping sustain the long-term quality of Hadoop on both
>>Linux
>> and Windows.
>>
>> We sincerely thank the community for their contribution and support so
>> far. And hope to continue having a close engagement in the future.
>>
>> -Microsoft HDInsight Team
>>
>>
>> -----Original Message-----
>> From: Suresh Srinivas [mailto:suresh@hortonworks.com]
>> Sent: Thursday, February 7, 2013 5:42 PM
>> To: common-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org;
>> hdfs-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
>> Subject: Heads up - merge branch-trunk-win to trunk
>>
>> The support for Hadoop on Windows was proposed in HADOOP-8079<
>> https://issues.apache.org/jira/browse/HADOOP-8079> almost a year ago.
>>The
>> goal was to make Hadoop natively integrated, full-featured, and
>>performance
>> and scalability tuned on Windows Server or Windows Azure.
>> We are happy to announce that a lot of progress has been made in this
>> regard.
>>
>> Initial work started in a feature branch, branch-1-win, based on
>>branch-1.
>> The details related to the work done in the branch can be seen in
>> CHANGES.txt<
>> 
>>http://svn.apache.org/viewvc/hadoop/common/branches/branch-1-win/CHANGES.
>>branch-1-win.txt?view=markup
>> >.
>> This work has been ported to a branch, branch-trunk-win, based on trunk.
>> Merge patch for this is available on
>> HADOOP-8562<https://issues.apache.org/jira/browse/HADOOP-8562>
>> .
>>
>> Highlights of the work done so far:
>> 1. Necessary changes in Hadoop to run natively on Windows. These changes
>> handle differences in platforms related to path names, process/task
>> management etc.
>> 2. Addition of winutils tools for managing file permissions and
>>ownership,
>> user group mapping, hardlinks, symbolic links, chmod, disk utilization,
>>and
>> process/task management.
>> 3. Added cmd scripts equivalent to existing shell scripts
>> hadoop-daemon.sh, start and stop scripts.
>> 4. Addition of block placement policy implemnation to support cloud
>> enviroment, more specifically Azure.
>>
>> We are very close to wrapping up the work in branch-trunk-win and
>>getting
>> ready for a merge. Currently the merge patch is passing close to 100% of
>> unit tests on Linux. Soon I will call for a vote to merge this branch
>>into
>> trunk.
>>
>> Next steps:
>> 1. Call for vote to merge branch-trunk-win to trunk, when the work
>> completes and precommit build is clean.
>> 2. Start a discussion on adding Jenkins precommit builds on windows and
>> how to integrate that with the existing commit process.
>>
>> Let me know if you have any questions.
>>
>> Regards,
>> Suresh
>>
>>
>
>
>-- 
>http://hortonworks.com/download/