You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flume.apache.org by Mike Percy <mp...@apache.org> on 2013/04/29 09:52:14 UTC
Review Request: FLUME-2027: Check for default replication fails on federated
cluster in hdfs sink
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/10834/
-----------------------------------------------------------
Review request for Flume.
Description
-------
Federated HDFS has a new API which has been ported to Hadoop 1.1.0 and Hadoop 2.0.0-alpha from HADOOP-8014 which takes a Path argument to the FileSystem.getDefaultReplication() call. This patch simply moves the existing implementation over to that new API.
This addresses bug FLUME-2027.
https://issues.apache.org/jira/browse/FLUME-2027
Diffs
-----
flume-ng-sinks/flume-hdfs-sink/pom.xml 2b6b5d2
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/AbstractHDFSWriter.java ff4f223
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSCompressedDataStream.java 0c618b5
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSDataStream.java c87fafe
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSSequenceFile.java 1a401d6
pom.xml 686ecb6
Diff: https://reviews.apache.org/r/10834/diff/
Testing
-------
Unit tests pass.
Thanks,
Mike Percy
Re: Review Request: FLUME-2027: Check for default replication fails on
federated cluster in hdfs sink
Posted by Hari Shreedharan <hs...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/10834/#review19938
-----------------------------------------------------------
pom.xml
<https://reviews.apache.org/r/10834/#comment41122>
We should probably not bump the version up to 1.1.0, rather use reflection to call the method.
- Hari Shreedharan
On April 29, 2013, 7:52 a.m., Mike Percy wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/10834/
> -----------------------------------------------------------
>
> (Updated April 29, 2013, 7:52 a.m.)
>
>
> Review request for Flume.
>
>
> Description
> -------
>
> Federated HDFS has a new API which has been ported to Hadoop 1.1.0 and Hadoop 2.0.0-alpha from HADOOP-8014 which takes a Path argument to the FileSystem.getDefaultReplication() call. This patch simply moves the existing implementation over to that new API.
>
>
> This addresses bug FLUME-2027.
> https://issues.apache.org/jira/browse/FLUME-2027
>
>
> Diffs
> -----
>
> flume-ng-sinks/flume-hdfs-sink/pom.xml 2b6b5d2
> flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/AbstractHDFSWriter.java ff4f223
> flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSCompressedDataStream.java 0c618b5
> flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSDataStream.java c87fafe
> flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSSequenceFile.java 1a401d6
> pom.xml 686ecb6
>
> Diff: https://reviews.apache.org/r/10834/diff/
>
>
> Testing
> -------
>
> Unit tests pass.
>
>
> Thanks,
>
> Mike Percy
>
>
Re: Review Request: FLUME-2027: Check for default replication fails on
federated cluster in hdfs sink
Posted by Hari Shreedharan <hs...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/10834/#review20233
-----------------------------------------------------------
Ship it!
Ship It!
- Hari Shreedharan
On May 1, 2013, 7:36 a.m., Mike Percy wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/10834/
> -----------------------------------------------------------
>
> (Updated May 1, 2013, 7:36 a.m.)
>
>
> Review request for Flume.
>
>
> Description
> -------
>
> Federated HDFS has a new API which has been ported to Hadoop 1.1.0 and Hadoop 2.0.0-alpha from HADOOP-8014 which takes a Path argument to the FileSystem.getDefaultReplication() call. This patch simply moves the existing implementation over to that new API.
>
>
> This addresses bug FLUME-2027.
> https://issues.apache.org/jira/browse/FLUME-2027
>
>
> Diffs
> -----
>
> flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/AbstractHDFSWriter.java ff4f223
> flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSCompressedDataStream.java 0c618b5
> flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSDataStream.java c87fafe
> flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSSequenceFile.java 1a401d6
>
> Diff: https://reviews.apache.org/r/10834/diff/
>
>
> Testing
> -------
>
> Unit tests pass.
>
> Tested this by running the unit tests under both Hadoop 1.0.1 and Hadoop 1.1.2 (manually modified the pom files) and looking at the DEBUG output from the unit tests. The debug output indicated that HADOOP-8014 is used if it's available, otherwise the existing API is used. This should be sufficient for all cases.
>
>
> Thanks,
>
> Mike Percy
>
>
Re: Review Request 10834: FLUME-2027: Check for default replication fails on
federated cluster in hdfs sink
Posted by Alexander Alten-Lorenz <al...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/10834/#review23344
-----------------------------------------------------------
Ship it!
looks good
- Alexander Alten-Lorenz
On May 1, 2013, 7:36 a.m., Mike Percy wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/10834/
> -----------------------------------------------------------
>
> (Updated May 1, 2013, 7:36 a.m.)
>
>
> Review request for Flume.
>
>
> Bugs: FLUME-2027
> https://issues.apache.org/jira/browse/FLUME-2027
>
>
> Repository: flume-git
>
>
> Description
> -------
>
> Federated HDFS has a new API which has been ported to Hadoop 1.1.0 and Hadoop 2.0.0-alpha from HADOOP-8014 which takes a Path argument to the FileSystem.getDefaultReplication() call. This patch simply moves the existing implementation over to that new API.
>
>
> Diffs
> -----
>
> flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/AbstractHDFSWriter.java ff4f223
> flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSCompressedDataStream.java 0c618b5
> flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSDataStream.java c87fafe
> flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSSequenceFile.java 1a401d6
>
> Diff: https://reviews.apache.org/r/10834/diff/
>
>
> Testing
> -------
>
> Unit tests pass.
>
> Tested this by running the unit tests under both Hadoop 1.0.1 and Hadoop 1.1.2 (manually modified the pom files) and looking at the DEBUG output from the unit tests. The debug output indicated that HADOOP-8014 is used if it's available, otherwise the existing API is used. This should be sufficient for all cases.
>
>
> Thanks,
>
> Mike Percy
>
>
Re: Review Request 10834: FLUME-2027: Check for default replication fails on
federated cluster in hdfs sink
Posted by Alexander Alten-Lorenz <al...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/10834/#review23343
-----------------------------------------------------------
Ship it!
looks good
- Alexander Alten-Lorenz
On May 1, 2013, 7:36 a.m., Mike Percy wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/10834/
> -----------------------------------------------------------
>
> (Updated May 1, 2013, 7:36 a.m.)
>
>
> Review request for Flume.
>
>
> Bugs: FLUME-2027
> https://issues.apache.org/jira/browse/FLUME-2027
>
>
> Repository: flume-git
>
>
> Description
> -------
>
> Federated HDFS has a new API which has been ported to Hadoop 1.1.0 and Hadoop 2.0.0-alpha from HADOOP-8014 which takes a Path argument to the FileSystem.getDefaultReplication() call. This patch simply moves the existing implementation over to that new API.
>
>
> Diffs
> -----
>
> flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/AbstractHDFSWriter.java ff4f223
> flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSCompressedDataStream.java 0c618b5
> flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSDataStream.java c87fafe
> flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSSequenceFile.java 1a401d6
>
> Diff: https://reviews.apache.org/r/10834/diff/
>
>
> Testing
> -------
>
> Unit tests pass.
>
> Tested this by running the unit tests under both Hadoop 1.0.1 and Hadoop 1.1.2 (manually modified the pom files) and looking at the DEBUG output from the unit tests. The debug output indicated that HADOOP-8014 is used if it's available, otherwise the existing API is used. This should be sufficient for all cases.
>
>
> Thanks,
>
> Mike Percy
>
>
Re: Review Request: FLUME-2027: Check for default replication fails on
federated cluster in hdfs sink
Posted by Mike Percy <mp...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/10834/
-----------------------------------------------------------
(Updated May 1, 2013, 7:36 a.m.)
Review request for Flume.
Description
-------
Federated HDFS has a new API which has been ported to Hadoop 1.1.0 and Hadoop 2.0.0-alpha from HADOOP-8014 which takes a Path argument to the FileSystem.getDefaultReplication() call. This patch simply moves the existing implementation over to that new API.
This addresses bug FLUME-2027.
https://issues.apache.org/jira/browse/FLUME-2027
Diffs
-----
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/AbstractHDFSWriter.java ff4f223
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSCompressedDataStream.java 0c618b5
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSDataStream.java c87fafe
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSSequenceFile.java 1a401d6
Diff: https://reviews.apache.org/r/10834/diff/
Testing (updated)
-------
Unit tests pass.
Tested this by running the unit tests under both Hadoop 1.0.1 and Hadoop 1.1.2 (manually modified the pom files) and looking at the DEBUG output from the unit tests. The debug output indicated that HADOOP-8014 is used if it's available, otherwise the existing API is used. This should be sufficient for all cases.
Thanks,
Mike Percy
Re: Review Request: FLUME-2027: Check for default replication fails on
federated cluster in hdfs sink
Posted by Mike Percy <mp...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/10834/
-----------------------------------------------------------
(Updated May 1, 2013, 7:34 a.m.)
Review request for Flume.
Changes
-------
Hari thanks for the review! I agree, updated the patch.
Description
-------
Federated HDFS has a new API which has been ported to Hadoop 1.1.0 and Hadoop 2.0.0-alpha from HADOOP-8014 which takes a Path argument to the FileSystem.getDefaultReplication() call. This patch simply moves the existing implementation over to that new API.
This addresses bug FLUME-2027.
https://issues.apache.org/jira/browse/FLUME-2027
Diffs (updated)
-----
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/AbstractHDFSWriter.java ff4f223
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSCompressedDataStream.java 0c618b5
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSDataStream.java c87fafe
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSSequenceFile.java 1a401d6
Diff: https://reviews.apache.org/r/10834/diff/
Testing
-------
Unit tests pass.
Thanks,
Mike Percy