You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by Jarek Cecho <ja...@apache.org> on 2014/01/23 16:51:04 UTC
Review Request 17240: PIG-3717 AvroStorage is not filtering hidden files on
directory recursion
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/17240/
-----------------------------------------------------------
Review request for pig.
Bugs: PIG-3717
https://issues.apache.org/jira/browse/PIG-3717
Repository: pig-git
Description
-------
We do have PATH_FILTER correctly defined to filter out hidden files, but it's used only during recursion entry point. When doing directory listing inside the recursion, we did not used the filter. I've added the filter there.
Diffs
-----
contrib/piggybank/java/src/main/java/org/apache/pig/piggybank/storage/avro/AvroStorageUtils.java 7235ab8df32a902dfb9b90622e02f16c10f6a6e4
contrib/piggybank/java/src/test/java/org/apache/pig/piggybank/test/storage/avro/TestAvroStorageUtils.java 99bc43eb83fbf272806dc7cca66c904bcbf71577
Diff: https://reviews.apache.org/r/17240/diff/
Testing
-------
Added new test case covering this issue.
Thanks,
Jarek Cecho
Re: Review Request 17240: PIG-3717 AvroStorage is not filtering hidden files
on directory recursion
Posted by Jarek Cecho <ja...@apache.org>.
> On Jan. 23, 2014, 4:57 p.m., Miklos Christine wrote:
> > contrib/piggybank/java/src/test/java/org/apache/pig/piggybank/test/storage/avro/TestAvroStorageUtils.java, line 152
> > <https://reviews.apache.org/r/17240/diff/1/?file=436062#file436062line152>
> >
> > I believe this should be fs.mkdirs(new Path(topDir)), correct?
Indeed it should be. I've didn't noticed it before due to the way hadoop internally works (the test was passing anyway).
- Jarek
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/17240/#review32622
-----------------------------------------------------------
On Jan. 23, 2014, 3:51 p.m., Jarek Cecho wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/17240/
> -----------------------------------------------------------
>
> (Updated Jan. 23, 2014, 3:51 p.m.)
>
>
> Review request for pig.
>
>
> Bugs: PIG-3717
> https://issues.apache.org/jira/browse/PIG-3717
>
>
> Repository: pig-git
>
>
> Description
> -------
>
> We do have PATH_FILTER correctly defined to filter out hidden files, but it's used only during recursion entry point. When doing directory listing inside the recursion, we did not used the filter. I've added the filter there.
>
>
> Diffs
> -----
>
> contrib/piggybank/java/src/main/java/org/apache/pig/piggybank/storage/avro/AvroStorageUtils.java 7235ab8df32a902dfb9b90622e02f16c10f6a6e4
> contrib/piggybank/java/src/test/java/org/apache/pig/piggybank/test/storage/avro/TestAvroStorageUtils.java 99bc43eb83fbf272806dc7cca66c904bcbf71577
>
> Diff: https://reviews.apache.org/r/17240/diff/
>
>
> Testing
> -------
>
> Added new test case covering this issue.
>
>
> Thanks,
>
> Jarek Cecho
>
>
Re: Review Request 17240: PIG-3717 AvroStorage is not filtering hidden files
on directory recursion
Posted by Miklos Christine <mw...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/17240/#review32622
-----------------------------------------------------------
contrib/piggybank/java/src/test/java/org/apache/pig/piggybank/test/storage/avro/TestAvroStorageUtils.java
<https://reviews.apache.org/r/17240/#comment61516>
I believe this should be fs.mkdirs(new Path(topDir)), correct?
- Miklos Christine
On Jan. 23, 2014, 3:51 p.m., Jarek Cecho wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/17240/
> -----------------------------------------------------------
>
> (Updated Jan. 23, 2014, 3:51 p.m.)
>
>
> Review request for pig.
>
>
> Bugs: PIG-3717
> https://issues.apache.org/jira/browse/PIG-3717
>
>
> Repository: pig-git
>
>
> Description
> -------
>
> We do have PATH_FILTER correctly defined to filter out hidden files, but it's used only during recursion entry point. When doing directory listing inside the recursion, we did not used the filter. I've added the filter there.
>
>
> Diffs
> -----
>
> contrib/piggybank/java/src/main/java/org/apache/pig/piggybank/storage/avro/AvroStorageUtils.java 7235ab8df32a902dfb9b90622e02f16c10f6a6e4
> contrib/piggybank/java/src/test/java/org/apache/pig/piggybank/test/storage/avro/TestAvroStorageUtils.java 99bc43eb83fbf272806dc7cca66c904bcbf71577
>
> Diff: https://reviews.apache.org/r/17240/diff/
>
>
> Testing
> -------
>
> Added new test case covering this issue.
>
>
> Thanks,
>
> Jarek Cecho
>
>
Re: Review Request 17240: PIG-3717 AvroStorage is not filtering hidden files
on directory recursion
Posted by Miklos Christine <mw...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/17240/#review32630
-----------------------------------------------------------
Ship it!
Ship It!
- Miklos Christine
On Jan. 23, 2014, 5:23 p.m., Jarek Cecho wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/17240/
> -----------------------------------------------------------
>
> (Updated Jan. 23, 2014, 5:23 p.m.)
>
>
> Review request for pig.
>
>
> Bugs: PIG-3717
> https://issues.apache.org/jira/browse/PIG-3717
>
>
> Repository: pig-git
>
>
> Description
> -------
>
> We do have PATH_FILTER correctly defined to filter out hidden files, but it's used only during recursion entry point. When doing directory listing inside the recursion, we did not used the filter. I've added the filter there.
>
>
> Diffs
> -----
>
> contrib/piggybank/java/src/main/java/org/apache/pig/piggybank/storage/avro/AvroStorageUtils.java 7235ab8df32a902dfb9b90622e02f16c10f6a6e4
> contrib/piggybank/java/src/test/java/org/apache/pig/piggybank/test/storage/avro/TestAvroStorageUtils.java 99bc43eb83fbf272806dc7cca66c904bcbf71577
>
> Diff: https://reviews.apache.org/r/17240/diff/
>
>
> Testing
> -------
>
> Added new test case covering this issue.
>
>
> Thanks,
>
> Jarek Cecho
>
>
Re: Review Request 17240: PIG-3717 AvroStorage is not filtering hidden files
on directory recursion
Posted by Jarek Cecho <ja...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/17240/
-----------------------------------------------------------
(Updated Jan. 23, 2014, 5:23 p.m.)
Review request for pig.
Changes
-------
Incorporated Miklos's feedback.
Bugs: PIG-3717
https://issues.apache.org/jira/browse/PIG-3717
Repository: pig-git
Description
-------
We do have PATH_FILTER correctly defined to filter out hidden files, but it's used only during recursion entry point. When doing directory listing inside the recursion, we did not used the filter. I've added the filter there.
Diffs (updated)
-----
contrib/piggybank/java/src/main/java/org/apache/pig/piggybank/storage/avro/AvroStorageUtils.java 7235ab8df32a902dfb9b90622e02f16c10f6a6e4
contrib/piggybank/java/src/test/java/org/apache/pig/piggybank/test/storage/avro/TestAvroStorageUtils.java 99bc43eb83fbf272806dc7cca66c904bcbf71577
Diff: https://reviews.apache.org/r/17240/diff/
Testing
-------
Added new test case covering this issue.
Thanks,
Jarek Cecho