Read an empty directory

[David Ortiz <dp...@gmail.com>]

Hello,

     Is there a way to get my pipeline to load an empty collection instead
of throwing an exception when I pass in a globbed path that doesn't match
anything?  It's part of a list of paths that otherwise match things.

Thanks,
     Dave

RE: Read an empty directory

[David Ortiz <do...@videologygroup.com>]

For the moment, seeing if I can do it with FileSystem.exists(Path) and just omit that particular input.  Not sure if it will like the globbed fs though.

From: Josh Wills [mailto:jwills@cloudera.com]
Sent: Monday, May 18, 2015 11:31 AM
To: user@crunch.apache.org
Subject: Re: Read an empty directory

Not a really clean way; one option would be to call getSize(Configuration) on the underlying Source you're reading from, and if it returns zero, substitute in a Pipeline.emptyPCollection() in the place where you would have called pipeline.read(Source).

J

On Mon, May 18, 2015 at 6:09 AM, David Ortiz <dp...@gmail.com>> wrote:
Hello,

     Is there a way to get my pipeline to load an empty collection instead of throwing an exception when I pass in a globbed path that doesn't match anything?  It's part of a list of paths that otherwise match things.

Thanks,
     Dave



--
Director of Data Science
Cloudera<http://www.cloudera.com>
Twitter: @josh_wills<http://twitter.com/josh_wills>
This email is intended only for the use of the individual(s) to whom it is addressed. If you have received this communication in error, please immediately notify the sender and delete the original email.

Re: Read an empty directory

[Josh Wills <jw...@cloudera.com>]

Not a really clean way; one option would be to call getSize(Configuration)
on the underlying Source you're reading from, and if it returns zero,
substitute in a Pipeline.emptyPCollection() in the place where you would
have called pipeline.read(Source).

J

On Mon, May 18, 2015 at 6:09 AM, David Ortiz <dp...@gmail.com> wrote:

> Hello,
>
>      Is there a way to get my pipeline to load an empty collection instead
> of throwing an exception when I pass in a globbed path that doesn't match
> anything?  It's part of a list of paths that otherwise match things.
>
> Thanks,
>      Dave
>



-- 
Director of Data Science
Cloudera <http://www.cloudera.com>
Twitter: @josh_wills <http://twitter.com/josh_wills>