Pig + xml ?

[Kayla Jay <ka...@yahoo.com>]

Hi

Can you use Pig with XML data files?  If so, does anyone have any examples?
I want to do something that would equate to an XPath query against the XML.

Thanks.



      

Re: Pig + xml ?

[Mark Chadwick <mc...@invitemedia.com>]

XML isn't particularly well suited for a Map/Reduce environment, however.
Hierarchical data is very tough to partition out to mappers.  On top of
that, because the data will be partitioned in 64M blocks (by default),
there's a very good chance that a random 64M chunk of a large XML file will
not even be parsable (opening tags in one block, closing tags in another).


On Tue, Jul 1, 2008 at 2:30 PM, Olga Natkovich <ol...@yahoo-inc.com> wrote:

> This can work but you would need to write a custom loader to parse the
> data: http://wiki.apache.org/pig/StorageFunction
>
> Olga
>
> > -----Original Message-----
> > From: Kayla Jay [mailto:kaylais30@yahoo.com]
> > Sent: Tuesday, July 01, 2008 11:24 AM
> > To: pig-user@incubator.apache.org
> > Subject: Pig + xml ?
> >
> > Hi
> >
> > Can you use Pig with XML data files?  If so, does anyone have
> > any examples?
> > I want to do something that would equate to an XPath query
> > against the XML.
> >
> > Thanks.
> >
> >
> >
> >
> >
>

RE: Pig + xml ?

[Olga Natkovich <ol...@yahoo-inc.com>]

This can work but you would need to write a custom loader to parse the
data: http://wiki.apache.org/pig/StorageFunction

Olga 

> -----Original Message-----
> From: Kayla Jay [mailto:kaylais30@yahoo.com] 
> Sent: Tuesday, July 01, 2008 11:24 AM
> To: pig-user@incubator.apache.org
> Subject: Pig + xml ?
> 
> Hi
> 
> Can you use Pig with XML data files?  If so, does anyone have 
> any examples?
> I want to do something that would equate to an XPath query 
> against the XML.
> 
> Thanks.
> 
> 
> 
>       
> 

Re: Pig + xml ?

[Mridul Muralidharan <mr...@yahoo-inc.com>]

Hi,

   What does the input look like ?
- A single file with multiple xml documents ?
- A directory containing a lot of individual xml files ?

If the former, then it might be slightly tricky since you will need to 
identify when an xml document "ends" and another "begins".
In simple cases, it could be an entire document in a single line - in 
which case, a simple line based loader will handle things fine.
If you have a special root node in your schema, you can use that for 
finding the start of a document/end of document and parse that out as a 
document/sax/etc.

Regards,
Mridul

Kayla Jay wrote:
> Hi
> 
> Can you use Pig with XML data files?  If so, does anyone have any examples?
> I want to do something that would equate to an XPath query against the XML.
> 
> Thanks.
> 
> 
> 
>