Apache Drill

[Daniel Ajo <Da...@abarcahealth.com>]

Hey there,

I was wondering if it is possible to query XML files using Apache Drill?

I see there are several formats, and maybe it would work using an xpath query of some sorts, but just wondering if it would work to directly query it using some sort of plug-in.

Well, let me know,

Daniel Ajo
********************************************************* CONFIDENTIALITY NOTE: This electronic transmission contains information belonging to Abarca Health LLC, which is confidential or legally privileged. If you are not the intended recipient, please immediately advise the sender by reply e-mail or telephone that this message has been inadvertently transmitted to you and delete this e-mail from your system. If you have received this transmission in error, you are hereby notified that any disclosure, copying, distribution or the taking of any action in reliance on the contents of the information is strictly prohibited.

Re: Apache Drill

[Julian Hyde <jh...@apache.org>]

IIRC, YAML has the same information content as JSON. So it could be
translated to the same events that a JSON parser would create. And so
Drill's JSON reader should be able to handle YAML files with very
little coercion.

As far as other formats go, let's see what adapters people contribute
and what performance they deliver.

On Sat, Oct 17, 2015 at 2:08 PM, Matt Burgess <ma...@gmail.com> wrote:
> Very fair, thanks! Should/can this approach be applied to other structured formats in the 1.x timeframe? YAML for example seems to have a manageable size in practice but there's nothing limiting its use in other domains and resulting in larger file(s).
>
> Sent from my iPhone
>
>> On Oct 17, 2015, at 4:51 PM, Julian Hyde <jh...@apache.org> wrote:
>>
>> Yes, frankly, performance is a concern. But there are also many
>> concerns about to fit a deep XML document into Drill's very
>> json-centric model. Building a good XML adapter is a very big task. My
>> hunch is that we should not let the perfect be the enemy of the good.
>> Build a version 1 XML adapter based on an XML-to-JSON converter and it
>> will give us plenty of ideas for what the "perfect" adapter in version
>> 2 should look like.
>>
>>> On Sat, Oct 17, 2015 at 1:43 PM, Matt Burgess <ma...@gmail.com> wrote:
>>> If the converter is clean and performant then I'm sure the community (including me) is interested :)
>>>
>>> However I wonder if Drill can afford to add a translation layer between data formats, could we be better served with similar parsing in Drill for XML as we do for JSON, or can it be pushed down far enough (to the parser) to not make a noticeable difference (which is what I think Julian is implying)?
>>>
>>> Sent from my iPhone
>>>
>>>> On Oct 17, 2015, at 1:41 PM, Magnus Pierre <mp...@maprtech.com> wrote:
>>>>
>>>> Hello,
>>>>
>>>> Just wrote a simple sax implementation that converts xml to json and that
>>>> is able to deal with decently complex xml's, that I currently use in Storm.
>>>> Takes attributes, and everything.
>>>>
>>>> I can share it with the community if interesting.
>>>>
>>>> /Magnus
>>>> Den 17 okt 2015 7:02 em skrev "Julian Hyde" <ju...@hydromatic.net>:
>>>>
>>>>> Seems to me the biggest problem is to make drill understand the nested
>>>>> structure of an xml document. That work has been done for json, so let's
>>>>> build on it. Suppose there was a translator that converted xml to json
>>>>> (adding attributes for things that json lacks, such as namespaces, text,
>>>>> element tags). Drill knows how to handle json, even if it is a bit verbose.
>>>>> The translator could be applied on the fly.
>>>>>
>>>>> Julian
>>>>>
>>>>>
>>>>>
>>>>> Sent from my iPad
>>>>>>> On Oct 16, 2015, at 2:31 PM, Stefán Baxter <st...@activitystream.com>
>>>>>> wrote:
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> It's not possible but there has been some talk here about supporting it.
>>>>>> If I remember correctly it's rather complicated and not really feasible.
>>>>>> (I'm just a newbie so don't take my words for it)
>>>>>>
>>>>>>
>>>>>> Regards,
>>>>>> -Stefan
>>>>>>
>>>>>> On Fri, Oct 16, 2015 at 8:54 PM, Daniel Ajo <Daniel.Ajo@abarcahealth.com
>>>>>>
>>>>>> wrote:
>>>>>>
>>>>>>> Hey there,
>>>>>>>
>>>>>>> I was wondering if it is possible to query XML files using Apache Drill?
>>>>>>>
>>>>>>> I see there are several formats, and maybe it would work using an xpath
>>>>>>> query of some sorts, but just wondering if it would work to directly
>>>>> query
>>>>>>> it using some sort of plug-in.
>>>>>>>
>>>>>>> Well, let me know,
>>>>>>>
>>>>>>> Daniel Ajo
>>>>>>> *********************************************************
>>>>> CONFIDENTIALITY
>>>>>>> NOTE: This electronic transmission contains information belonging to
>>>>> Abarca
>>>>>>> Health LLC, which is confidential or legally privileged. If you are not
>>>>> the
>>>>>>> intended recipient, please immediately advise the sender by reply
>>>>> e-mail or
>>>>>>> telephone that this message has been inadvertently transmitted to you
>>>>> and
>>>>>>> delete this e-mail from your system. If you have received this
>>>>> transmission
>>>>>>> in error, you are hereby notified that any disclosure, copying,
>>>>>>> distribution or the taking of any action in reliance on the contents of
>>>>> the
>>>>>>> information is strictly prohibited.
>>>>>

Re: Apache Drill

[Ted Dunning <te...@gmail.com>]

YAML is just a different surface syntax for JSON and thus should be
extremely easy to handle.  In many ways YAML is what JSON should have been
except for the lineage from javascript.



On Sat, Oct 17, 2015 at 2:08 PM, Matt Burgess <ma...@gmail.com> wrote:

> Very fair, thanks! Should/can this approach be applied to other structured
> formats in the 1.x timeframe? YAML for example seems to have a manageable
> size in practice but there's nothing limiting its use in other domains and
> resulting in larger file(s).
>
> Sent from my iPhone
>
> > On Oct 17, 2015, at 4:51 PM, Julian Hyde <jh...@apache.org> wrote:
> >
> > Yes, frankly, performance is a concern. But there are also many
> > concerns about to fit a deep XML document into Drill's very
> > json-centric model. Building a good XML adapter is a very big task. My
> > hunch is that we should not let the perfect be the enemy of the good.
> > Build a version 1 XML adapter based on an XML-to-JSON converter and it
> > will give us plenty of ideas for what the "perfect" adapter in version
> > 2 should look like.
> >
> >> On Sat, Oct 17, 2015 at 1:43 PM, Matt Burgess <ma...@gmail.com>
> wrote:
> >> If the converter is clean and performant then I'm sure the community
> (including me) is interested :)
> >>
> >> However I wonder if Drill can afford to add a translation layer between
> data formats, could we be better served with similar parsing in Drill for
> XML as we do for JSON, or can it be pushed down far enough (to the parser)
> to not make a noticeable difference (which is what I think Julian is
> implying)?
> >>
> >> Sent from my iPhone
> >>
> >>> On Oct 17, 2015, at 1:41 PM, Magnus Pierre <mp...@maprtech.com>
> wrote:
> >>>
> >>> Hello,
> >>>
> >>> Just wrote a simple sax implementation that converts xml to json and
> that
> >>> is able to deal with decently complex xml's, that I currently use in
> Storm.
> >>> Takes attributes, and everything.
> >>>
> >>> I can share it with the community if interesting.
> >>>
> >>> /Magnus
> >>> Den 17 okt 2015 7:02 em skrev "Julian Hyde" <ju...@hydromatic.net>:
> >>>
> >>>> Seems to me the biggest problem is to make drill understand the nested
> >>>> structure of an xml document. That work has been done for json, so
> let's
> >>>> build on it. Suppose there was a translator that converted xml to json
> >>>> (adding attributes for things that json lacks, such as namespaces,
> text,
> >>>> element tags). Drill knows how to handle json, even if it is a bit
> verbose.
> >>>> The translator could be applied on the fly.
> >>>>
> >>>> Julian
> >>>>
> >>>>
> >>>>
> >>>> Sent from my iPad
> >>>>>> On Oct 16, 2015, at 2:31 PM, Stefán Baxter <
> stefan@activitystream.com>
> >>>>> wrote:
> >>>>>
> >>>>> Hi,
> >>>>>
> >>>>> It's not possible but there has been some talk here about supporting
> it.
> >>>>> If I remember correctly it's rather complicated and not really
> feasible.
> >>>>> (I'm just a newbie so don't take my words for it)
> >>>>>
> >>>>>
> >>>>> Regards,
> >>>>> -Stefan
> >>>>>
> >>>>> On Fri, Oct 16, 2015 at 8:54 PM, Daniel Ajo <
> Daniel.Ajo@abarcahealth.com
> >>>>>
> >>>>> wrote:
> >>>>>
> >>>>>> Hey there,
> >>>>>>
> >>>>>> I was wondering if it is possible to query XML files using Apache
> Drill?
> >>>>>>
> >>>>>> I see there are several formats, and maybe it would work using an
> xpath
> >>>>>> query of some sorts, but just wondering if it would work to directly
> >>>> query
> >>>>>> it using some sort of plug-in.
> >>>>>>
> >>>>>> Well, let me know,
> >>>>>>
> >>>>>> Daniel Ajo
> >>>>>> *********************************************************
> >>>> CONFIDENTIALITY
> >>>>>> NOTE: This electronic transmission contains information belonging to
> >>>> Abarca
> >>>>>> Health LLC, which is confidential or legally privileged. If you are
> not
> >>>> the
> >>>>>> intended recipient, please immediately advise the sender by reply
> >>>> e-mail or
> >>>>>> telephone that this message has been inadvertently transmitted to
> you
> >>>> and
> >>>>>> delete this e-mail from your system. If you have received this
> >>>> transmission
> >>>>>> in error, you are hereby notified that any disclosure, copying,
> >>>>>> distribution or the taking of any action in reliance on the
> contents of
> >>>> the
> >>>>>> information is strictly prohibited.
> >>>>
>

Re: Apache Drill

[Matt Burgess <ma...@gmail.com>]

Very fair, thanks! Should/can this approach be applied to other structured formats in the 1.x timeframe? YAML for example seems to have a manageable size in practice but there's nothing limiting its use in other domains and resulting in larger file(s).

Sent from my iPhone

> On Oct 17, 2015, at 4:51 PM, Julian Hyde <jh...@apache.org> wrote:
> 
> Yes, frankly, performance is a concern. But there are also many
> concerns about to fit a deep XML document into Drill's very
> json-centric model. Building a good XML adapter is a very big task. My
> hunch is that we should not let the perfect be the enemy of the good.
> Build a version 1 XML adapter based on an XML-to-JSON converter and it
> will give us plenty of ideas for what the "perfect" adapter in version
> 2 should look like.
> 
>> On Sat, Oct 17, 2015 at 1:43 PM, Matt Burgess <ma...@gmail.com> wrote:
>> If the converter is clean and performant then I'm sure the community (including me) is interested :)
>> 
>> However I wonder if Drill can afford to add a translation layer between data formats, could we be better served with similar parsing in Drill for XML as we do for JSON, or can it be pushed down far enough (to the parser) to not make a noticeable difference (which is what I think Julian is implying)?
>> 
>> Sent from my iPhone
>> 
>>> On Oct 17, 2015, at 1:41 PM, Magnus Pierre <mp...@maprtech.com> wrote:
>>> 
>>> Hello,
>>> 
>>> Just wrote a simple sax implementation that converts xml to json and that
>>> is able to deal with decently complex xml's, that I currently use in Storm.
>>> Takes attributes, and everything.
>>> 
>>> I can share it with the community if interesting.
>>> 
>>> /Magnus
>>> Den 17 okt 2015 7:02 em skrev "Julian Hyde" <ju...@hydromatic.net>:
>>> 
>>>> Seems to me the biggest problem is to make drill understand the nested
>>>> structure of an xml document. That work has been done for json, so let's
>>>> build on it. Suppose there was a translator that converted xml to json
>>>> (adding attributes for things that json lacks, such as namespaces, text,
>>>> element tags). Drill knows how to handle json, even if it is a bit verbose.
>>>> The translator could be applied on the fly.
>>>> 
>>>> Julian
>>>> 
>>>> 
>>>> 
>>>> Sent from my iPad
>>>>>> On Oct 16, 2015, at 2:31 PM, Stefán Baxter <st...@activitystream.com>
>>>>> wrote:
>>>>> 
>>>>> Hi,
>>>>> 
>>>>> It's not possible but there has been some talk here about supporting it.
>>>>> If I remember correctly it's rather complicated and not really feasible.
>>>>> (I'm just a newbie so don't take my words for it)
>>>>> 
>>>>> 
>>>>> Regards,
>>>>> -Stefan
>>>>> 
>>>>> On Fri, Oct 16, 2015 at 8:54 PM, Daniel Ajo <Daniel.Ajo@abarcahealth.com
>>>>> 
>>>>> wrote:
>>>>> 
>>>>>> Hey there,
>>>>>> 
>>>>>> I was wondering if it is possible to query XML files using Apache Drill?
>>>>>> 
>>>>>> I see there are several formats, and maybe it would work using an xpath
>>>>>> query of some sorts, but just wondering if it would work to directly
>>>> query
>>>>>> it using some sort of plug-in.
>>>>>> 
>>>>>> Well, let me know,
>>>>>> 
>>>>>> Daniel Ajo
>>>>>> *********************************************************
>>>> CONFIDENTIALITY
>>>>>> NOTE: This electronic transmission contains information belonging to
>>>> Abarca
>>>>>> Health LLC, which is confidential or legally privileged. If you are not
>>>> the
>>>>>> intended recipient, please immediately advise the sender by reply
>>>> e-mail or
>>>>>> telephone that this message has been inadvertently transmitted to you
>>>> and
>>>>>> delete this e-mail from your system. If you have received this
>>>> transmission
>>>>>> in error, you are hereby notified that any disclosure, copying,
>>>>>> distribution or the taking of any action in reliance on the contents of
>>>> the
>>>>>> information is strictly prohibited.
>>>> 

Re: Apache Drill

[Julian Hyde <jh...@apache.org>]

Yes, frankly, performance is a concern. But there are also many
concerns about to fit a deep XML document into Drill's very
json-centric model. Building a good XML adapter is a very big task. My
hunch is that we should not let the perfect be the enemy of the good.
Build a version 1 XML adapter based on an XML-to-JSON converter and it
will give us plenty of ideas for what the "perfect" adapter in version
2 should look like.

On Sat, Oct 17, 2015 at 1:43 PM, Matt Burgess <ma...@gmail.com> wrote:
> If the converter is clean and performant then I'm sure the community (including me) is interested :)
>
> However I wonder if Drill can afford to add a translation layer between data formats, could we be better served with similar parsing in Drill for XML as we do for JSON, or can it be pushed down far enough (to the parser) to not make a noticeable difference (which is what I think Julian is implying)?
>
> Sent from my iPhone
>
>> On Oct 17, 2015, at 1:41 PM, Magnus Pierre <mp...@maprtech.com> wrote:
>>
>> Hello,
>>
>> Just wrote a simple sax implementation that converts xml to json and that
>> is able to deal with decently complex xml's, that I currently use in Storm.
>> Takes attributes, and everything.
>>
>> I can share it with the community if interesting.
>>
>> /Magnus
>> Den 17 okt 2015 7:02 em skrev "Julian Hyde" <ju...@hydromatic.net>:
>>
>>> Seems to me the biggest problem is to make drill understand the nested
>>> structure of an xml document. That work has been done for json, so let's
>>> build on it. Suppose there was a translator that converted xml to json
>>> (adding attributes for things that json lacks, such as namespaces, text,
>>> element tags). Drill knows how to handle json, even if it is a bit verbose.
>>> The translator could be applied on the fly.
>>>
>>> Julian
>>>
>>>
>>>
>>> Sent from my iPad
>>>>> On Oct 16, 2015, at 2:31 PM, Stefán Baxter <st...@activitystream.com>
>>>> wrote:
>>>>
>>>> Hi,
>>>>
>>>> It's not possible but there has been some talk here about supporting it.
>>>> If I remember correctly it's rather complicated and not really feasible.
>>>> (I'm just a newbie so don't take my words for it)
>>>>
>>>>
>>>> Regards,
>>>> -Stefan
>>>>
>>>> On Fri, Oct 16, 2015 at 8:54 PM, Daniel Ajo <Daniel.Ajo@abarcahealth.com
>>>>
>>>> wrote:
>>>>
>>>>> Hey there,
>>>>>
>>>>> I was wondering if it is possible to query XML files using Apache Drill?
>>>>>
>>>>> I see there are several formats, and maybe it would work using an xpath
>>>>> query of some sorts, but just wondering if it would work to directly
>>> query
>>>>> it using some sort of plug-in.
>>>>>
>>>>> Well, let me know,
>>>>>
>>>>> Daniel Ajo
>>>>> *********************************************************
>>> CONFIDENTIALITY
>>>>> NOTE: This electronic transmission contains information belonging to
>>> Abarca
>>>>> Health LLC, which is confidential or legally privileged. If you are not
>>> the
>>>>> intended recipient, please immediately advise the sender by reply
>>> e-mail or
>>>>> telephone that this message has been inadvertently transmitted to you
>>> and
>>>>> delete this e-mail from your system. If you have received this
>>> transmission
>>>>> in error, you are hereby notified that any disclosure, copying,
>>>>> distribution or the taking of any action in reliance on the contents of
>>> the
>>>>> information is strictly prohibited.
>>>

Re: Apache Drill

[Ted Dunning <te...@gmail.com>]

Inline

On Sun, Oct 18, 2015 at 11:37 AM, Julian Hyde <jh...@apache.org> wrote:

> ...
> My proposed “solution” — and I suspect you’re not going to like it — is to
> ignore, for now, harder XML problems and focus on the easier ones.


Hmm.... I think that this may or may not be easy. But it is real important.


> A lot of XML documents do not have repeating scalar values. They are
> collections of records, perhaps with nested records or nested collections
> of records.


The scalar-ness of my example was just a simplification. The same problem
occurs every time there is a list that sometimes contains 1 element.


> Whitespace can be safely thrown away. Namespaces are not used.


Fine.


> A lot of data is in XML format because XML was the only option considered,
> not because the data structure pushed the limits of what XML’s rich model
> can express.
>

True.


> I think 90% of cases can be handled using a simple XML-to-JSON mapper that
> takes hints such as that the “employee” tag is to become a list of JSON
> maps and the “salary” and “name” tags are to be treated as attributes.
>

Great.

The real question is whether or not the XML community already has such a
hinting mechanism.  Or is Drill about to reinvent that?


>
> I really think that if we focus on the harder cases we’ll end up with the
> wrong solution.
>

No doubt.  This isn't one of those.

Re: Apache Drill

[Julian Hyde <jh...@apache.org>]

Ted,

My proposed “solution” — and I suspect you’re not going to like it — is to ignore, for now, harder XML problems and focus on the easier ones. A lot of XML documents do not have repeating scalar values. They are collections of records, perhaps with nested records or nested collections of records. Whitespace can be safely thrown away. Namespaces are not used. A lot of data is in XML format because XML was the only option considered, not because the data structure pushed the limits of what XML’s rich model can express.

I think 90% of cases can be handled using a simple XML-to-JSON mapper that takes hints such as that the “employee” tag is to become a list of JSON maps and the “salary” and “name” tags are to be treated as attributes.

I really think that if we focus on the harder cases we’ll end up with the wrong solution.

Julian


> On Oct 18, 2015, at 11:24 AM, Ted Dunning <te...@gmail.com> wrote:
> 
> Kasper,
> 
> This might work.
> 
> One issue that I see is that Metamodel seems to take a very XML centric
> view of things while Drill takes a pretty JSON view of things.
> 
> The point at which I think that this might cause problems is that Drill
> currently has troubles when it sees a records like
> 
> <record><item>1</item></record>
> <record><item>2</item><item>3</item></record>
> 
> This is fine as far as XML is concerned, but if you think about it in terms
> of JSON, it is probably best to view these records as
> 
> {"item":[1]}
> {"item":[2,3]}
> 
> Unfortunately, from the first record, there is no way to tell that it
> should not be viewed as
> 
> {"item":1}
> 
> Do you have a suggestion that would help with this?
> 
> 
> On Sun, Oct 18, 2015 at 8:41 AM, Kasper Sørensen <
> i.am.kasper.sorensen@gmail.com> wrote:
> 
>> Hi there,
>> 
>> Sorry for barging in, but maybe this is a place where Drill and MetaModel
>> could benefit from each other? We've considered that before at least ...
>> 
>> MetaModel already has support for both DOM and SAX based XML querying. They
>> basically inherit some characteristics from DOM and SAX respectively:
>> 
>> - In the DOM variant we can infer a schema and all the user has to do is
>> select a XML file/resource anywhere.
>> - In the SAX variant the user has to specify which paths in the XML
>> document should represent logical "tables" and what paths represent their
>> columns.
>> 
>> See [1] for more info. Hope this might be of interest to integrate into
>> Drill?
>> 
>> Best regards,
>> Kasper Sørensen (from the MetaModel project)
>> 
>> [1] http://wiki.apache.org/metamodel/examples/XmlTableMapping
>> 
>> 2015-10-18 0:35 GMT+02:00 Magnus Pierre <mp...@maprtech.com>:
>> 
>>> Well, very few lines of code imho. And simple. Been able to parse pretty
>>> deep structures with no issues so far. Performance? 10-15 5mb xml's in
>> less
>>> than a second on my laptop but then I run it using Storm with some
>>> parallelism in place. Don't know if it's good or bad. I'll share the code
>>> next time I use computer. You don't need to use it, but it works at
>> least.
>>> 
>>> /M
>>> Den 17 okt 2015 10:43 em skrev "Matt Burgess" <ma...@gmail.com>:
>>> 
>>>> If the converter is clean and performant then I'm sure the community
>>>> (including me) is interested :)
>>>> 
>>>> However I wonder if Drill can afford to add a translation layer between
>>>> data formats, could we be better served with similar parsing in Drill
>> for
>>>> XML as we do for JSON, or can it be pushed down far enough (to the
>>> parser)
>>>> to not make a noticeable difference (which is what I think Julian is
>>>> implying)?
>>>> 
>>>> Sent from my iPhone
>>>> 
>>>>> On Oct 17, 2015, at 1:41 PM, Magnus Pierre <mp...@maprtech.com>
>>> wrote:
>>>>> 
>>>>> Hello,
>>>>> 
>>>>> Just wrote a simple sax implementation that converts xml to json and
>>> that
>>>>> is able to deal with decently complex xml's, that I currently use in
>>>> Storm.
>>>>> Takes attributes, and everything.
>>>>> 
>>>>> I can share it with the community if interesting.
>>>>> 
>>>>> /Magnus
>>>>> Den 17 okt 2015 7:02 em skrev "Julian Hyde" <ju...@hydromatic.net>:
>>>>> 
>>>>>> Seems to me the biggest problem is to make drill understand the
>> nested
>>>>>> structure of an xml document. That work has been done for json, so
>>> let's
>>>>>> build on it. Suppose there was a translator that converted xml to
>> json
>>>>>> (adding attributes for things that json lacks, such as namespaces,
>>> text,
>>>>>> element tags). Drill knows how to handle json, even if it is a bit
>>>> verbose.
>>>>>> The translator could be applied on the fly.
>>>>>> 
>>>>>> Julian
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> Sent from my iPad
>>>>>>>> On Oct 16, 2015, at 2:31 PM, Stefán Baxter <
>>> stefan@activitystream.com
>>>>> 
>>>>>>> wrote:
>>>>>>> 
>>>>>>> Hi,
>>>>>>> 
>>>>>>> It's not possible but there has been some talk here about
>> supporting
>>>> it.
>>>>>>> If I remember correctly it's rather complicated and not really
>>>> feasible.
>>>>>>> (I'm just a newbie so don't take my words for it)
>>>>>>> 
>>>>>>> 
>>>>>>> Regards,
>>>>>>> -Stefan
>>>>>>> 
>>>>>>> On Fri, Oct 16, 2015 at 8:54 PM, Daniel Ajo <
>>>> Daniel.Ajo@abarcahealth.com
>>>>>>> 
>>>>>>> wrote:
>>>>>>> 
>>>>>>>> Hey there,
>>>>>>>> 
>>>>>>>> I was wondering if it is possible to query XML files using Apache
>>>> Drill?
>>>>>>>> 
>>>>>>>> I see there are several formats, and maybe it would work using an
>>>> xpath
>>>>>>>> query of some sorts, but just wondering if it would work to
>> directly
>>>>>> query
>>>>>>>> it using some sort of plug-in.
>>>>>>>> 
>>>>>>>> Well, let me know,
>>>>>>>> 
>>>>>>>> Daniel Ajo
>>>>>>>> *********************************************************
>>>>>> CONFIDENTIALITY
>>>>>>>> NOTE: This electronic transmission contains information belonging
>> to
>>>>>> Abarca
>>>>>>>> Health LLC, which is confidential or legally privileged. If you
>> are
>>>> not
>>>>>> the
>>>>>>>> intended recipient, please immediately advise the sender by reply
>>>>>> e-mail or
>>>>>>>> telephone that this message has been inadvertently transmitted to
>>> you
>>>>>> and
>>>>>>>> delete this e-mail from your system. If you have received this
>>>>>> transmission
>>>>>>>> in error, you are hereby notified that any disclosure, copying,
>>>>>>>> distribution or the taking of any action in reliance on the
>> contents
>>>> of
>>>>>> the
>>>>>>>> information is strictly prohibited.
>>>>>> 
>>>> 
>>> 
>> 

Re: Apache Drill

[Jacques Nadeau <ja...@dremio.com>]

I actually generating thing this is a special version of object promotion.
Steven is already working on solving the same problem where it exists when
we have the JSON file:

{"item":1}
{"item":[2,3]}

The main difference isn't until we see the "3" value in XML that we need to
promote the "2" value to be the first element in an array. In Drill this is
simply a value copy (not something that is currently exposed in
ComplexWriter but exists at the vector level already). We could also do the
promotion with lookahead and a markable/rewindable event stream.

In general though, I agree with Julian. Let's get a basic reader working
first.

Does someone want to create a JIRA and propose a basic design. I'm more
than happy to help people through the format plugin definitions as
necessary since documentation there is lean.



--
Jacques Nadeau
CTO and Co-Founder, Dremio

On Sun, Oct 18, 2015 at 11:24 AM, Ted Dunning <te...@gmail.com> wrote:

> Kasper,
>
> This might work.
>
> One issue that I see is that Metamodel seems to take a very XML centric
> view of things while Drill takes a pretty JSON view of things.
>
> The point at which I think that this might cause problems is that Drill
> currently has troubles when it sees a records like
>
> <record><item>1</item></record>
> <record><item>2</item><item>3</item></record>
>
> This is fine as far as XML is concerned, but if you think about it in terms
> of JSON, it is probably best to view these records as
>
> {"item":[1]}
> {"item":[2,3]}
>
> Unfortunately, from the first record, there is no way to tell that it
> should not be viewed as
>
> {"item":1}
>
> Do you have a suggestion that would help with this?
>
>
> On Sun, Oct 18, 2015 at 8:41 AM, Kasper Sørensen <
> i.am.kasper.sorensen@gmail.com> wrote:
>
> > Hi there,
> >
> > Sorry for barging in, but maybe this is a place where Drill and MetaModel
> > could benefit from each other? We've considered that before at least ...
> >
> > MetaModel already has support for both DOM and SAX based XML querying.
> They
> > basically inherit some characteristics from DOM and SAX respectively:
> >
> >  - In the DOM variant we can infer a schema and all the user has to do is
> > select a XML file/resource anywhere.
> >  - In the SAX variant the user has to specify which paths in the XML
> > document should represent logical "tables" and what paths represent their
> > columns.
> >
> > See [1] for more info. Hope this might be of interest to integrate into
> > Drill?
> >
> > Best regards,
> > Kasper Sørensen (from the MetaModel project)
> >
> > [1] http://wiki.apache.org/metamodel/examples/XmlTableMapping
> >
> > 2015-10-18 0:35 GMT+02:00 Magnus Pierre <mp...@maprtech.com>:
> >
> > > Well, very few lines of code imho. And simple. Been able to parse
> pretty
> > > deep structures with no issues so far. Performance? 10-15 5mb xml's in
> > less
> > > than a second on my laptop but then I run it using Storm with some
> > > parallelism in place. Don't know if it's good or bad. I'll share the
> code
> > > next time I use computer. You don't need to use it, but it works at
> > least.
> > >
> > > /M
> > > Den 17 okt 2015 10:43 em skrev "Matt Burgess" <ma...@gmail.com>:
> > >
> > > > If the converter is clean and performant then I'm sure the community
> > > > (including me) is interested :)
> > > >
> > > > However I wonder if Drill can afford to add a translation layer
> between
> > > > data formats, could we be better served with similar parsing in Drill
> > for
> > > > XML as we do for JSON, or can it be pushed down far enough (to the
> > > parser)
> > > > to not make a noticeable difference (which is what I think Julian is
> > > > implying)?
> > > >
> > > > Sent from my iPhone
> > > >
> > > > > On Oct 17, 2015, at 1:41 PM, Magnus Pierre <mp...@maprtech.com>
> > > wrote:
> > > > >
> > > > > Hello,
> > > > >
> > > > > Just wrote a simple sax implementation that converts xml to json
> and
> > > that
> > > > > is able to deal with decently complex xml's, that I currently use
> in
> > > > Storm.
> > > > > Takes attributes, and everything.
> > > > >
> > > > > I can share it with the community if interesting.
> > > > >
> > > > > /Magnus
> > > > > Den 17 okt 2015 7:02 em skrev "Julian Hyde" <julian@hydromatic.net
> >:
> > > > >
> > > > >> Seems to me the biggest problem is to make drill understand the
> > nested
> > > > >> structure of an xml document. That work has been done for json, so
> > > let's
> > > > >> build on it. Suppose there was a translator that converted xml to
> > json
> > > > >> (adding attributes for things that json lacks, such as namespaces,
> > > text,
> > > > >> element tags). Drill knows how to handle json, even if it is a bit
> > > > verbose.
> > > > >> The translator could be applied on the fly.
> > > > >>
> > > > >> Julian
> > > > >>
> > > > >>
> > > > >>
> > > > >> Sent from my iPad
> > > > >>>> On Oct 16, 2015, at 2:31 PM, Stefán Baxter <
> > > stefan@activitystream.com
> > > > >
> > > > >>> wrote:
> > > > >>>
> > > > >>> Hi,
> > > > >>>
> > > > >>> It's not possible but there has been some talk here about
> > supporting
> > > > it.
> > > > >>> If I remember correctly it's rather complicated and not really
> > > > feasible.
> > > > >>> (I'm just a newbie so don't take my words for it)
> > > > >>>
> > > > >>>
> > > > >>> Regards,
> > > > >>> -Stefan
> > > > >>>
> > > > >>> On Fri, Oct 16, 2015 at 8:54 PM, Daniel Ajo <
> > > > Daniel.Ajo@abarcahealth.com
> > > > >>>
> > > > >>> wrote:
> > > > >>>
> > > > >>>> Hey there,
> > > > >>>>
> > > > >>>> I was wondering if it is possible to query XML files using
> Apache
> > > > Drill?
> > > > >>>>
> > > > >>>> I see there are several formats, and maybe it would work using
> an
> > > > xpath
> > > > >>>> query of some sorts, but just wondering if it would work to
> > directly
> > > > >> query
> > > > >>>> it using some sort of plug-in.
> > > > >>>>
> > > > >>>> Well, let me know,
> > > > >>>>
> > > > >>>> Daniel Ajo
> > > > >>>> *********************************************************
> > > > >> CONFIDENTIALITY
> > > > >>>> NOTE: This electronic transmission contains information
> belonging
> > to
> > > > >> Abarca
> > > > >>>> Health LLC, which is confidential or legally privileged. If you
> > are
> > > > not
> > > > >> the
> > > > >>>> intended recipient, please immediately advise the sender by
> reply
> > > > >> e-mail or
> > > > >>>> telephone that this message has been inadvertently transmitted
> to
> > > you
> > > > >> and
> > > > >>>> delete this e-mail from your system. If you have received this
> > > > >> transmission
> > > > >>>> in error, you are hereby notified that any disclosure, copying,
> > > > >>>> distribution or the taking of any action in reliance on the
> > contents
> > > > of
> > > > >> the
> > > > >>>> information is strictly prohibited.
> > > > >>
> > > >
> > >
> >
>

Re: Apache Drill

[Kasper Sørensen <i....@gmail.com>]

@Jacques - I might be interested, but I am new to Drill - just saw the demo
at Apache: Big Data EU and felt that what we do in MetaModel and what you
do in Drill is hugely complementary. But I need someone to help me get
started and thought this might be a good spot for it.

@Ted - The mapping is specified as you can see it in the wiki page linked
to [1]. The first argument of the XmlSaxTableDef instantiation defines the
scope of a record. I agree that every <record> in your example should be a
record in the dataset as well. Today we would then not have a function in
our mapping to support mapping a field that represent the list of <item>s,
but that's a function we could add IMO.

What I mean by mapping two tables is to have two XmlSaxTableDef instances,
one for <record> and one for <item>. Then you could essentially join them a
la:

  SELECT ... FROM /record r INNER JOIN /record/item i ON r.row_id =
i.index(/record)

But I think this approach is probably less desirable given the way you guys
think about XML documents as a single table/source. In MetaModel we allow
for both use cases, but I guess most users would also find it more natural
to just have a single table mapped.

Yes, MetaModel's way of coping with this kind of stuff is to map towards a
relational model. We do offer richer data types such as list and key-value
map to allow nested structures within a record. But MetaModel's row format
is bound to a set of columns, not a document or such. Basically that has
been our way of enabling SQL queries to fit well with the underlying
datastore.

[1] http://wiki.apache.org/metamodel/examples/XmlTableMapping

2015-10-19 2:56 GMT+02:00 Ted Dunning <te...@gmail.com>:

> Kasper,
>
> How is the mapping you suggest specified?
>
> In my example, I meant for there to be many records in a file and each
> record element to be a record insofar as Drill is concerned.  I also didn't
> include other information that presumably would make it more interesting to
> talk about a record element as a unit.
>
> Your suggestion (1) is essentially to denest the records, but that loses
> the nice hierarchical structure expressed in the original that so easily
> could be expressed in the JSON data model.
>
> For your option (2), what do you mean by map 2 tables?  Does MetaModel
> inherently assume that all output is purely relational?
>
>
>
>
> On Sun, Oct 18, 2015 at 1:18 PM, Kasper Sørensen <
> i.am.kasper.sorensen@gmail.com> wrote:
>
> > Hi Ted,
> >
> > Actually in MetaModel you then have two choices with your mapping to
> table
> > format.
> >
> > 1) Either map the "item" as the granularity of a record. That way you
> will
> > get three rows - one for each item. On the last of the two rows you would
> > have the same values for any element that is registered at the <record>
> > scope.
> >
> > 2) You can also map 2 tables instead - one for <record> and one for
> <item>
> > and then join them as you like.
> >
> >
> > 2015-10-18 20:24 GMT+02:00 Ted Dunning <te...@gmail.com>:
> >
> > > Kasper,
> > >
> > > This might work.
> > >
> > > One issue that I see is that Metamodel seems to take a very XML centric
> > > view of things while Drill takes a pretty JSON view of things.
> > >
> > > The point at which I think that this might cause problems is that Drill
> > > currently has troubles when it sees a records like
> > >
> > > <record><item>1</item></record>
> > > <record><item>2</item><item>3</item></record>
> > >
> > > This is fine as far as XML is concerned, but if you think about it in
> > terms
> > > of JSON, it is probably best to view these records as
> > >
> > > {"item":[1]}
> > > {"item":[2,3]}
> > >
> > > Unfortunately, from the first record, there is no way to tell that it
> > > should not be viewed as
> > >
> > > {"item":1}
> > >
> > > Do you have a suggestion that would help with this?
> > >
> > >
> > > On Sun, Oct 18, 2015 at 8:41 AM, Kasper Sørensen <
> > > i.am.kasper.sorensen@gmail.com> wrote:
> > >
> > > > Hi there,
> > > >
> > > > Sorry for barging in, but maybe this is a place where Drill and
> > MetaModel
> > > > could benefit from each other? We've considered that before at least
> > ...
> > > >
> > > > MetaModel already has support for both DOM and SAX based XML
> querying.
> > > They
> > > > basically inherit some characteristics from DOM and SAX respectively:
> > > >
> > > >  - In the DOM variant we can infer a schema and all the user has to
> do
> > is
> > > > select a XML file/resource anywhere.
> > > >  - In the SAX variant the user has to specify which paths in the XML
> > > > document should represent logical "tables" and what paths represent
> > their
> > > > columns.
> > > >
> > > > See [1] for more info. Hope this might be of interest to integrate
> into
> > > > Drill?
> > > >
> > > > Best regards,
> > > > Kasper Sørensen (from the MetaModel project)
> > > >
> > > > [1] http://wiki.apache.org/metamodel/examples/XmlTableMapping
> > > >
> > > > 2015-10-18 0:35 GMT+02:00 Magnus Pierre <mp...@maprtech.com>:
> > > >
> > > > > Well, very few lines of code imho. And simple. Been able to parse
> > > pretty
> > > > > deep structures with no issues so far. Performance? 10-15 5mb xml's
> > in
> > > > less
> > > > > than a second on my laptop but then I run it using Storm with some
> > > > > parallelism in place. Don't know if it's good or bad. I'll share
> the
> > > code
> > > > > next time I use computer. You don't need to use it, but it works at
> > > > least.
> > > > >
> > > > > /M
> > > > > Den 17 okt 2015 10:43 em skrev "Matt Burgess" <mattyb149@gmail.com
> >:
> > > > >
> > > > > > If the converter is clean and performant then I'm sure the
> > community
> > > > > > (including me) is interested :)
> > > > > >
> > > > > > However I wonder if Drill can afford to add a translation layer
> > > between
> > > > > > data formats, could we be better served with similar parsing in
> > Drill
> > > > for
> > > > > > XML as we do for JSON, or can it be pushed down far enough (to
> the
> > > > > parser)
> > > > > > to not make a noticeable difference (which is what I think Julian
> > is
> > > > > > implying)?
> > > > > >
> > > > > > Sent from my iPhone
> > > > > >
> > > > > > > On Oct 17, 2015, at 1:41 PM, Magnus Pierre <
> mpierre@maprtech.com
> > >
> > > > > wrote:
> > > > > > >
> > > > > > > Hello,
> > > > > > >
> > > > > > > Just wrote a simple sax implementation that converts xml to
> json
> > > and
> > > > > that
> > > > > > > is able to deal with decently complex xml's, that I currently
> use
> > > in
> > > > > > Storm.
> > > > > > > Takes attributes, and everything.
> > > > > > >
> > > > > > > I can share it with the community if interesting.
> > > > > > >
> > > > > > > /Magnus
> > > > > > > Den 17 okt 2015 7:02 em skrev "Julian Hyde" <
> > julian@hydromatic.net
> > > >:
> > > > > > >
> > > > > > >> Seems to me the biggest problem is to make drill understand
> the
> > > > nested
> > > > > > >> structure of an xml document. That work has been done for
> json,
> > so
> > > > > let's
> > > > > > >> build on it. Suppose there was a translator that converted xml
> > to
> > > > json
> > > > > > >> (adding attributes for things that json lacks, such as
> > namespaces,
> > > > > text,
> > > > > > >> element tags). Drill knows how to handle json, even if it is a
> > bit
> > > > > > verbose.
> > > > > > >> The translator could be applied on the fly.
> > > > > > >>
> > > > > > >> Julian
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >> Sent from my iPad
> > > > > > >>>> On Oct 16, 2015, at 2:31 PM, Stefán Baxter <
> > > > > stefan@activitystream.com
> > > > > > >
> > > > > > >>> wrote:
> > > > > > >>>
> > > > > > >>> Hi,
> > > > > > >>>
> > > > > > >>> It's not possible but there has been some talk here about
> > > > supporting
> > > > > > it.
> > > > > > >>> If I remember correctly it's rather complicated and not
> really
> > > > > > feasible.
> > > > > > >>> (I'm just a newbie so don't take my words for it)
> > > > > > >>>
> > > > > > >>>
> > > > > > >>> Regards,
> > > > > > >>> -Stefan
> > > > > > >>>
> > > > > > >>> On Fri, Oct 16, 2015 at 8:54 PM, Daniel Ajo <
> > > > > > Daniel.Ajo@abarcahealth.com
> > > > > > >>>
> > > > > > >>> wrote:
> > > > > > >>>
> > > > > > >>>> Hey there,
> > > > > > >>>>
> > > > > > >>>> I was wondering if it is possible to query XML files using
> > > Apache
> > > > > > Drill?
> > > > > > >>>>
> > > > > > >>>> I see there are several formats, and maybe it would work
> using
> > > an
> > > > > > xpath
> > > > > > >>>> query of some sorts, but just wondering if it would work to
> > > > directly
> > > > > > >> query
> > > > > > >>>> it using some sort of plug-in.
> > > > > > >>>>
> > > > > > >>>> Well, let me know,
> > > > > > >>>>
> > > > > > >>>> Daniel Ajo
> > > > > > >>>> *********************************************************
> > > > > > >> CONFIDENTIALITY
> > > > > > >>>> NOTE: This electronic transmission contains information
> > > belonging
> > > > to
> > > > > > >> Abarca
> > > > > > >>>> Health LLC, which is confidential or legally privileged. If
> > you
> > > > are
> > > > > > not
> > > > > > >> the
> > > > > > >>>> intended recipient, please immediately advise the sender by
> > > reply
> > > > > > >> e-mail or
> > > > > > >>>> telephone that this message has been inadvertently
> transmitted
> > > to
> > > > > you
> > > > > > >> and
> > > > > > >>>> delete this e-mail from your system. If you have received
> this
> > > > > > >> transmission
> > > > > > >>>> in error, you are hereby notified that any disclosure,
> > copying,
> > > > > > >>>> distribution or the taking of any action in reliance on the
> > > > contents
> > > > > > of
> > > > > > >> the
> > > > > > >>>> information is strictly prohibited.
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Apache Drill

[Ted Dunning <te...@gmail.com>]

Kasper,

How is the mapping you suggest specified?

In my example, I meant for there to be many records in a file and each
record element to be a record insofar as Drill is concerned.  I also didn't
include other information that presumably would make it more interesting to
talk about a record element as a unit.

Your suggestion (1) is essentially to denest the records, but that loses
the nice hierarchical structure expressed in the original that so easily
could be expressed in the JSON data model.

For your option (2), what do you mean by map 2 tables?  Does MetaModel
inherently assume that all output is purely relational?




On Sun, Oct 18, 2015 at 1:18 PM, Kasper Sørensen <
i.am.kasper.sorensen@gmail.com> wrote:

> Hi Ted,
>
> Actually in MetaModel you then have two choices with your mapping to table
> format.
>
> 1) Either map the "item" as the granularity of a record. That way you will
> get three rows - one for each item. On the last of the two rows you would
> have the same values for any element that is registered at the <record>
> scope.
>
> 2) You can also map 2 tables instead - one for <record> and one for <item>
> and then join them as you like.
>
>
> 2015-10-18 20:24 GMT+02:00 Ted Dunning <te...@gmail.com>:
>
> > Kasper,
> >
> > This might work.
> >
> > One issue that I see is that Metamodel seems to take a very XML centric
> > view of things while Drill takes a pretty JSON view of things.
> >
> > The point at which I think that this might cause problems is that Drill
> > currently has troubles when it sees a records like
> >
> > <record><item>1</item></record>
> > <record><item>2</item><item>3</item></record>
> >
> > This is fine as far as XML is concerned, but if you think about it in
> terms
> > of JSON, it is probably best to view these records as
> >
> > {"item":[1]}
> > {"item":[2,3]}
> >
> > Unfortunately, from the first record, there is no way to tell that it
> > should not be viewed as
> >
> > {"item":1}
> >
> > Do you have a suggestion that would help with this?
> >
> >
> > On Sun, Oct 18, 2015 at 8:41 AM, Kasper Sørensen <
> > i.am.kasper.sorensen@gmail.com> wrote:
> >
> > > Hi there,
> > >
> > > Sorry for barging in, but maybe this is a place where Drill and
> MetaModel
> > > could benefit from each other? We've considered that before at least
> ...
> > >
> > > MetaModel already has support for both DOM and SAX based XML querying.
> > They
> > > basically inherit some characteristics from DOM and SAX respectively:
> > >
> > >  - In the DOM variant we can infer a schema and all the user has to do
> is
> > > select a XML file/resource anywhere.
> > >  - In the SAX variant the user has to specify which paths in the XML
> > > document should represent logical "tables" and what paths represent
> their
> > > columns.
> > >
> > > See [1] for more info. Hope this might be of interest to integrate into
> > > Drill?
> > >
> > > Best regards,
> > > Kasper Sørensen (from the MetaModel project)
> > >
> > > [1] http://wiki.apache.org/metamodel/examples/XmlTableMapping
> > >
> > > 2015-10-18 0:35 GMT+02:00 Magnus Pierre <mp...@maprtech.com>:
> > >
> > > > Well, very few lines of code imho. And simple. Been able to parse
> > pretty
> > > > deep structures with no issues so far. Performance? 10-15 5mb xml's
> in
> > > less
> > > > than a second on my laptop but then I run it using Storm with some
> > > > parallelism in place. Don't know if it's good or bad. I'll share the
> > code
> > > > next time I use computer. You don't need to use it, but it works at
> > > least.
> > > >
> > > > /M
> > > > Den 17 okt 2015 10:43 em skrev "Matt Burgess" <ma...@gmail.com>:
> > > >
> > > > > If the converter is clean and performant then I'm sure the
> community
> > > > > (including me) is interested :)
> > > > >
> > > > > However I wonder if Drill can afford to add a translation layer
> > between
> > > > > data formats, could we be better served with similar parsing in
> Drill
> > > for
> > > > > XML as we do for JSON, or can it be pushed down far enough (to the
> > > > parser)
> > > > > to not make a noticeable difference (which is what I think Julian
> is
> > > > > implying)?
> > > > >
> > > > > Sent from my iPhone
> > > > >
> > > > > > On Oct 17, 2015, at 1:41 PM, Magnus Pierre <mpierre@maprtech.com
> >
> > > > wrote:
> > > > > >
> > > > > > Hello,
> > > > > >
> > > > > > Just wrote a simple sax implementation that converts xml to json
> > and
> > > > that
> > > > > > is able to deal with decently complex xml's, that I currently use
> > in
> > > > > Storm.
> > > > > > Takes attributes, and everything.
> > > > > >
> > > > > > I can share it with the community if interesting.
> > > > > >
> > > > > > /Magnus
> > > > > > Den 17 okt 2015 7:02 em skrev "Julian Hyde" <
> julian@hydromatic.net
> > >:
> > > > > >
> > > > > >> Seems to me the biggest problem is to make drill understand the
> > > nested
> > > > > >> structure of an xml document. That work has been done for json,
> so
> > > > let's
> > > > > >> build on it. Suppose there was a translator that converted xml
> to
> > > json
> > > > > >> (adding attributes for things that json lacks, such as
> namespaces,
> > > > text,
> > > > > >> element tags). Drill knows how to handle json, even if it is a
> bit
> > > > > verbose.
> > > > > >> The translator could be applied on the fly.
> > > > > >>
> > > > > >> Julian
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >> Sent from my iPad
> > > > > >>>> On Oct 16, 2015, at 2:31 PM, Stefán Baxter <
> > > > stefan@activitystream.com
> > > > > >
> > > > > >>> wrote:
> > > > > >>>
> > > > > >>> Hi,
> > > > > >>>
> > > > > >>> It's not possible but there has been some talk here about
> > > supporting
> > > > > it.
> > > > > >>> If I remember correctly it's rather complicated and not really
> > > > > feasible.
> > > > > >>> (I'm just a newbie so don't take my words for it)
> > > > > >>>
> > > > > >>>
> > > > > >>> Regards,
> > > > > >>> -Stefan
> > > > > >>>
> > > > > >>> On Fri, Oct 16, 2015 at 8:54 PM, Daniel Ajo <
> > > > > Daniel.Ajo@abarcahealth.com
> > > > > >>>
> > > > > >>> wrote:
> > > > > >>>
> > > > > >>>> Hey there,
> > > > > >>>>
> > > > > >>>> I was wondering if it is possible to query XML files using
> > Apache
> > > > > Drill?
> > > > > >>>>
> > > > > >>>> I see there are several formats, and maybe it would work using
> > an
> > > > > xpath
> > > > > >>>> query of some sorts, but just wondering if it would work to
> > > directly
> > > > > >> query
> > > > > >>>> it using some sort of plug-in.
> > > > > >>>>
> > > > > >>>> Well, let me know,
> > > > > >>>>
> > > > > >>>> Daniel Ajo
> > > > > >>>> *********************************************************
> > > > > >> CONFIDENTIALITY
> > > > > >>>> NOTE: This electronic transmission contains information
> > belonging
> > > to
> > > > > >> Abarca
> > > > > >>>> Health LLC, which is confidential or legally privileged. If
> you
> > > are
> > > > > not
> > > > > >> the
> > > > > >>>> intended recipient, please immediately advise the sender by
> > reply
> > > > > >> e-mail or
> > > > > >>>> telephone that this message has been inadvertently transmitted
> > to
> > > > you
> > > > > >> and
> > > > > >>>> delete this e-mail from your system. If you have received this
> > > > > >> transmission
> > > > > >>>> in error, you are hereby notified that any disclosure,
> copying,
> > > > > >>>> distribution or the taking of any action in reliance on the
> > > contents
> > > > > of
> > > > > >> the
> > > > > >>>> information is strictly prohibited.
> > > > > >>
> > > > >
> > > >
> > >
> >
>

Re: Apache Drill

[Jacques Nadeau <ja...@dremio.com>]

Kasper,

Are you interested in working on a Drill metamodel format plugin? That way,
anything that Metamodel exposes would be available in Drill. It seems like
this would add great value to many users.

Jacques

--
Jacques Nadeau
CTO and Co-Founder, Dremio

On Sun, Oct 18, 2015 at 1:18 PM, Kasper Sørensen <
i.am.kasper.sorensen@gmail.com> wrote:

> Hi Ted,
>
> Actually in MetaModel you then have two choices with your mapping to table
> format.
>
> 1) Either map the "item" as the granularity of a record. That way you will
> get three rows - one for each item. On the last of the two rows you would
> have the same values for any element that is registered at the <record>
> scope.
>
> 2) You can also map 2 tables instead - one for <record> and one for <item>
> and then join them as you like.
>
>
> 2015-10-18 20:24 GMT+02:00 Ted Dunning <te...@gmail.com>:
>
> > Kasper,
> >
> > This might work.
> >
> > One issue that I see is that Metamodel seems to take a very XML centric
> > view of things while Drill takes a pretty JSON view of things.
> >
> > The point at which I think that this might cause problems is that Drill
> > currently has troubles when it sees a records like
> >
> > <record><item>1</item></record>
> > <record><item>2</item><item>3</item></record>
> >
> > This is fine as far as XML is concerned, but if you think about it in
> terms
> > of JSON, it is probably best to view these records as
> >
> > {"item":[1]}
> > {"item":[2,3]}
> >
> > Unfortunately, from the first record, there is no way to tell that it
> > should not be viewed as
> >
> > {"item":1}
> >
> > Do you have a suggestion that would help with this?
> >
> >
> > On Sun, Oct 18, 2015 at 8:41 AM, Kasper Sørensen <
> > i.am.kasper.sorensen@gmail.com> wrote:
> >
> > > Hi there,
> > >
> > > Sorry for barging in, but maybe this is a place where Drill and
> MetaModel
> > > could benefit from each other? We've considered that before at least
> ...
> > >
> > > MetaModel already has support for both DOM and SAX based XML querying.
> > They
> > > basically inherit some characteristics from DOM and SAX respectively:
> > >
> > >  - In the DOM variant we can infer a schema and all the user has to do
> is
> > > select a XML file/resource anywhere.
> > >  - In the SAX variant the user has to specify which paths in the XML
> > > document should represent logical "tables" and what paths represent
> their
> > > columns.
> > >
> > > See [1] for more info. Hope this might be of interest to integrate into
> > > Drill?
> > >
> > > Best regards,
> > > Kasper Sørensen (from the MetaModel project)
> > >
> > > [1] http://wiki.apache.org/metamodel/examples/XmlTableMapping
> > >
> > > 2015-10-18 0:35 GMT+02:00 Magnus Pierre <mp...@maprtech.com>:
> > >
> > > > Well, very few lines of code imho. And simple. Been able to parse
> > pretty
> > > > deep structures with no issues so far. Performance? 10-15 5mb xml's
> in
> > > less
> > > > than a second on my laptop but then I run it using Storm with some
> > > > parallelism in place. Don't know if it's good or bad. I'll share the
> > code
> > > > next time I use computer. You don't need to use it, but it works at
> > > least.
> > > >
> > > > /M
> > > > Den 17 okt 2015 10:43 em skrev "Matt Burgess" <ma...@gmail.com>:
> > > >
> > > > > If the converter is clean and performant then I'm sure the
> community
> > > > > (including me) is interested :)
> > > > >
> > > > > However I wonder if Drill can afford to add a translation layer
> > between
> > > > > data formats, could we be better served with similar parsing in
> Drill
> > > for
> > > > > XML as we do for JSON, or can it be pushed down far enough (to the
> > > > parser)
> > > > > to not make a noticeable difference (which is what I think Julian
> is
> > > > > implying)?
> > > > >
> > > > > Sent from my iPhone
> > > > >
> > > > > > On Oct 17, 2015, at 1:41 PM, Magnus Pierre <mpierre@maprtech.com
> >
> > > > wrote:
> > > > > >
> > > > > > Hello,
> > > > > >
> > > > > > Just wrote a simple sax implementation that converts xml to json
> > and
> > > > that
> > > > > > is able to deal with decently complex xml's, that I currently use
> > in
> > > > > Storm.
> > > > > > Takes attributes, and everything.
> > > > > >
> > > > > > I can share it with the community if interesting.
> > > > > >
> > > > > > /Magnus
> > > > > > Den 17 okt 2015 7:02 em skrev "Julian Hyde" <
> julian@hydromatic.net
> > >:
> > > > > >
> > > > > >> Seems to me the biggest problem is to make drill understand the
> > > nested
> > > > > >> structure of an xml document. That work has been done for json,
> so
> > > > let's
> > > > > >> build on it. Suppose there was a translator that converted xml
> to
> > > json
> > > > > >> (adding attributes for things that json lacks, such as
> namespaces,
> > > > text,
> > > > > >> element tags). Drill knows how to handle json, even if it is a
> bit
> > > > > verbose.
> > > > > >> The translator could be applied on the fly.
> > > > > >>
> > > > > >> Julian
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >> Sent from my iPad
> > > > > >>>> On Oct 16, 2015, at 2:31 PM, Stefán Baxter <
> > > > stefan@activitystream.com
> > > > > >
> > > > > >>> wrote:
> > > > > >>>
> > > > > >>> Hi,
> > > > > >>>
> > > > > >>> It's not possible but there has been some talk here about
> > > supporting
> > > > > it.
> > > > > >>> If I remember correctly it's rather complicated and not really
> > > > > feasible.
> > > > > >>> (I'm just a newbie so don't take my words for it)
> > > > > >>>
> > > > > >>>
> > > > > >>> Regards,
> > > > > >>> -Stefan
> > > > > >>>
> > > > > >>> On Fri, Oct 16, 2015 at 8:54 PM, Daniel Ajo <
> > > > > Daniel.Ajo@abarcahealth.com
> > > > > >>>
> > > > > >>> wrote:
> > > > > >>>
> > > > > >>>> Hey there,
> > > > > >>>>
> > > > > >>>> I was wondering if it is possible to query XML files using
> > Apache
> > > > > Drill?
> > > > > >>>>
> > > > > >>>> I see there are several formats, and maybe it would work using
> > an
> > > > > xpath
> > > > > >>>> query of some sorts, but just wondering if it would work to
> > > directly
> > > > > >> query
> > > > > >>>> it using some sort of plug-in.
> > > > > >>>>
> > > > > >>>> Well, let me know,
> > > > > >>>>
> > > > > >>>> Daniel Ajo
> > > > > >>>> *********************************************************
> > > > > >> CONFIDENTIALITY
> > > > > >>>> NOTE: This electronic transmission contains information
> > belonging
> > > to
> > > > > >> Abarca
> > > > > >>>> Health LLC, which is confidential or legally privileged. If
> you
> > > are
> > > > > not
> > > > > >> the
> > > > > >>>> intended recipient, please immediately advise the sender by
> > reply
> > > > > >> e-mail or
> > > > > >>>> telephone that this message has been inadvertently transmitted
> > to
> > > > you
> > > > > >> and
> > > > > >>>> delete this e-mail from your system. If you have received this
> > > > > >> transmission
> > > > > >>>> in error, you are hereby notified that any disclosure,
> copying,
> > > > > >>>> distribution or the taking of any action in reliance on the
> > > contents
> > > > > of
> > > > > >> the
> > > > > >>>> information is strictly prohibited.
> > > > > >>
> > > > >
> > > >
> > >
> >
>

Re: Apache Drill

[Kasper Sørensen <i....@gmail.com>]

Hi Ted,

Actually in MetaModel you then have two choices with your mapping to table
format.

1) Either map the "item" as the granularity of a record. That way you will
get three rows - one for each item. On the last of the two rows you would
have the same values for any element that is registered at the <record>
scope.

2) You can also map 2 tables instead - one for <record> and one for <item>
and then join them as you like.


2015-10-18 20:24 GMT+02:00 Ted Dunning <te...@gmail.com>:

> Kasper,
>
> This might work.
>
> One issue that I see is that Metamodel seems to take a very XML centric
> view of things while Drill takes a pretty JSON view of things.
>
> The point at which I think that this might cause problems is that Drill
> currently has troubles when it sees a records like
>
> <record><item>1</item></record>
> <record><item>2</item><item>3</item></record>
>
> This is fine as far as XML is concerned, but if you think about it in terms
> of JSON, it is probably best to view these records as
>
> {"item":[1]}
> {"item":[2,3]}
>
> Unfortunately, from the first record, there is no way to tell that it
> should not be viewed as
>
> {"item":1}
>
> Do you have a suggestion that would help with this?
>
>
> On Sun, Oct 18, 2015 at 8:41 AM, Kasper Sørensen <
> i.am.kasper.sorensen@gmail.com> wrote:
>
> > Hi there,
> >
> > Sorry for barging in, but maybe this is a place where Drill and MetaModel
> > could benefit from each other? We've considered that before at least ...
> >
> > MetaModel already has support for both DOM and SAX based XML querying.
> They
> > basically inherit some characteristics from DOM and SAX respectively:
> >
> >  - In the DOM variant we can infer a schema and all the user has to do is
> > select a XML file/resource anywhere.
> >  - In the SAX variant the user has to specify which paths in the XML
> > document should represent logical "tables" and what paths represent their
> > columns.
> >
> > See [1] for more info. Hope this might be of interest to integrate into
> > Drill?
> >
> > Best regards,
> > Kasper Sørensen (from the MetaModel project)
> >
> > [1] http://wiki.apache.org/metamodel/examples/XmlTableMapping
> >
> > 2015-10-18 0:35 GMT+02:00 Magnus Pierre <mp...@maprtech.com>:
> >
> > > Well, very few lines of code imho. And simple. Been able to parse
> pretty
> > > deep structures with no issues so far. Performance? 10-15 5mb xml's in
> > less
> > > than a second on my laptop but then I run it using Storm with some
> > > parallelism in place. Don't know if it's good or bad. I'll share the
> code
> > > next time I use computer. You don't need to use it, but it works at
> > least.
> > >
> > > /M
> > > Den 17 okt 2015 10:43 em skrev "Matt Burgess" <ma...@gmail.com>:
> > >
> > > > If the converter is clean and performant then I'm sure the community
> > > > (including me) is interested :)
> > > >
> > > > However I wonder if Drill can afford to add a translation layer
> between
> > > > data formats, could we be better served with similar parsing in Drill
> > for
> > > > XML as we do for JSON, or can it be pushed down far enough (to the
> > > parser)
> > > > to not make a noticeable difference (which is what I think Julian is
> > > > implying)?
> > > >
> > > > Sent from my iPhone
> > > >
> > > > > On Oct 17, 2015, at 1:41 PM, Magnus Pierre <mp...@maprtech.com>
> > > wrote:
> > > > >
> > > > > Hello,
> > > > >
> > > > > Just wrote a simple sax implementation that converts xml to json
> and
> > > that
> > > > > is able to deal with decently complex xml's, that I currently use
> in
> > > > Storm.
> > > > > Takes attributes, and everything.
> > > > >
> > > > > I can share it with the community if interesting.
> > > > >
> > > > > /Magnus
> > > > > Den 17 okt 2015 7:02 em skrev "Julian Hyde" <julian@hydromatic.net
> >:
> > > > >
> > > > >> Seems to me the biggest problem is to make drill understand the
> > nested
> > > > >> structure of an xml document. That work has been done for json, so
> > > let's
> > > > >> build on it. Suppose there was a translator that converted xml to
> > json
> > > > >> (adding attributes for things that json lacks, such as namespaces,
> > > text,
> > > > >> element tags). Drill knows how to handle json, even if it is a bit
> > > > verbose.
> > > > >> The translator could be applied on the fly.
> > > > >>
> > > > >> Julian
> > > > >>
> > > > >>
> > > > >>
> > > > >> Sent from my iPad
> > > > >>>> On Oct 16, 2015, at 2:31 PM, Stefán Baxter <
> > > stefan@activitystream.com
> > > > >
> > > > >>> wrote:
> > > > >>>
> > > > >>> Hi,
> > > > >>>
> > > > >>> It's not possible but there has been some talk here about
> > supporting
> > > > it.
> > > > >>> If I remember correctly it's rather complicated and not really
> > > > feasible.
> > > > >>> (I'm just a newbie so don't take my words for it)
> > > > >>>
> > > > >>>
> > > > >>> Regards,
> > > > >>> -Stefan
> > > > >>>
> > > > >>> On Fri, Oct 16, 2015 at 8:54 PM, Daniel Ajo <
> > > > Daniel.Ajo@abarcahealth.com
> > > > >>>
> > > > >>> wrote:
> > > > >>>
> > > > >>>> Hey there,
> > > > >>>>
> > > > >>>> I was wondering if it is possible to query XML files using
> Apache
> > > > Drill?
> > > > >>>>
> > > > >>>> I see there are several formats, and maybe it would work using
> an
> > > > xpath
> > > > >>>> query of some sorts, but just wondering if it would work to
> > directly
> > > > >> query
> > > > >>>> it using some sort of plug-in.
> > > > >>>>
> > > > >>>> Well, let me know,
> > > > >>>>
> > > > >>>> Daniel Ajo
> > > > >>>> *********************************************************
> > > > >> CONFIDENTIALITY
> > > > >>>> NOTE: This electronic transmission contains information
> belonging
> > to
> > > > >> Abarca
> > > > >>>> Health LLC, which is confidential or legally privileged. If you
> > are
> > > > not
> > > > >> the
> > > > >>>> intended recipient, please immediately advise the sender by
> reply
> > > > >> e-mail or
> > > > >>>> telephone that this message has been inadvertently transmitted
> to
> > > you
> > > > >> and
> > > > >>>> delete this e-mail from your system. If you have received this
> > > > >> transmission
> > > > >>>> in error, you are hereby notified that any disclosure, copying,
> > > > >>>> distribution or the taking of any action in reliance on the
> > contents
> > > > of
> > > > >> the
> > > > >>>> information is strictly prohibited.
> > > > >>
> > > >
> > >
> >
>

Re: Apache Drill

[Ted Dunning <te...@gmail.com>]

Kasper,

This might work.

One issue that I see is that Metamodel seems to take a very XML centric
view of things while Drill takes a pretty JSON view of things.

The point at which I think that this might cause problems is that Drill
currently has troubles when it sees a records like

<record><item>1</item></record>
<record><item>2</item><item>3</item></record>

This is fine as far as XML is concerned, but if you think about it in terms
of JSON, it is probably best to view these records as

{"item":[1]}
{"item":[2,3]}

Unfortunately, from the first record, there is no way to tell that it
should not be viewed as

{"item":1}

Do you have a suggestion that would help with this?


On Sun, Oct 18, 2015 at 8:41 AM, Kasper Sørensen <
i.am.kasper.sorensen@gmail.com> wrote:

> Hi there,
>
> Sorry for barging in, but maybe this is a place where Drill and MetaModel
> could benefit from each other? We've considered that before at least ...
>
> MetaModel already has support for both DOM and SAX based XML querying. They
> basically inherit some characteristics from DOM and SAX respectively:
>
>  - In the DOM variant we can infer a schema and all the user has to do is
> select a XML file/resource anywhere.
>  - In the SAX variant the user has to specify which paths in the XML
> document should represent logical "tables" and what paths represent their
> columns.
>
> See [1] for more info. Hope this might be of interest to integrate into
> Drill?
>
> Best regards,
> Kasper Sørensen (from the MetaModel project)
>
> [1] http://wiki.apache.org/metamodel/examples/XmlTableMapping
>
> 2015-10-18 0:35 GMT+02:00 Magnus Pierre <mp...@maprtech.com>:
>
> > Well, very few lines of code imho. And simple. Been able to parse pretty
> > deep structures with no issues so far. Performance? 10-15 5mb xml's in
> less
> > than a second on my laptop but then I run it using Storm with some
> > parallelism in place. Don't know if it's good or bad. I'll share the code
> > next time I use computer. You don't need to use it, but it works at
> least.
> >
> > /M
> > Den 17 okt 2015 10:43 em skrev "Matt Burgess" <ma...@gmail.com>:
> >
> > > If the converter is clean and performant then I'm sure the community
> > > (including me) is interested :)
> > >
> > > However I wonder if Drill can afford to add a translation layer between
> > > data formats, could we be better served with similar parsing in Drill
> for
> > > XML as we do for JSON, or can it be pushed down far enough (to the
> > parser)
> > > to not make a noticeable difference (which is what I think Julian is
> > > implying)?
> > >
> > > Sent from my iPhone
> > >
> > > > On Oct 17, 2015, at 1:41 PM, Magnus Pierre <mp...@maprtech.com>
> > wrote:
> > > >
> > > > Hello,
> > > >
> > > > Just wrote a simple sax implementation that converts xml to json and
> > that
> > > > is able to deal with decently complex xml's, that I currently use in
> > > Storm.
> > > > Takes attributes, and everything.
> > > >
> > > > I can share it with the community if interesting.
> > > >
> > > > /Magnus
> > > > Den 17 okt 2015 7:02 em skrev "Julian Hyde" <ju...@hydromatic.net>:
> > > >
> > > >> Seems to me the biggest problem is to make drill understand the
> nested
> > > >> structure of an xml document. That work has been done for json, so
> > let's
> > > >> build on it. Suppose there was a translator that converted xml to
> json
> > > >> (adding attributes for things that json lacks, such as namespaces,
> > text,
> > > >> element tags). Drill knows how to handle json, even if it is a bit
> > > verbose.
> > > >> The translator could be applied on the fly.
> > > >>
> > > >> Julian
> > > >>
> > > >>
> > > >>
> > > >> Sent from my iPad
> > > >>>> On Oct 16, 2015, at 2:31 PM, Stefán Baxter <
> > stefan@activitystream.com
> > > >
> > > >>> wrote:
> > > >>>
> > > >>> Hi,
> > > >>>
> > > >>> It's not possible but there has been some talk here about
> supporting
> > > it.
> > > >>> If I remember correctly it's rather complicated and not really
> > > feasible.
> > > >>> (I'm just a newbie so don't take my words for it)
> > > >>>
> > > >>>
> > > >>> Regards,
> > > >>> -Stefan
> > > >>>
> > > >>> On Fri, Oct 16, 2015 at 8:54 PM, Daniel Ajo <
> > > Daniel.Ajo@abarcahealth.com
> > > >>>
> > > >>> wrote:
> > > >>>
> > > >>>> Hey there,
> > > >>>>
> > > >>>> I was wondering if it is possible to query XML files using Apache
> > > Drill?
> > > >>>>
> > > >>>> I see there are several formats, and maybe it would work using an
> > > xpath
> > > >>>> query of some sorts, but just wondering if it would work to
> directly
> > > >> query
> > > >>>> it using some sort of plug-in.
> > > >>>>
> > > >>>> Well, let me know,
> > > >>>>
> > > >>>> Daniel Ajo
> > > >>>> *********************************************************
> > > >> CONFIDENTIALITY
> > > >>>> NOTE: This electronic transmission contains information belonging
> to
> > > >> Abarca
> > > >>>> Health LLC, which is confidential or legally privileged. If you
> are
> > > not
> > > >> the
> > > >>>> intended recipient, please immediately advise the sender by reply
> > > >> e-mail or
> > > >>>> telephone that this message has been inadvertently transmitted to
> > you
> > > >> and
> > > >>>> delete this e-mail from your system. If you have received this
> > > >> transmission
> > > >>>> in error, you are hereby notified that any disclosure, copying,
> > > >>>> distribution or the taking of any action in reliance on the
> contents
> > > of
> > > >> the
> > > >>>> information is strictly prohibited.
> > > >>
> > >
> >
>

Re: Apache Drill

[Kasper Sørensen <i....@gmail.com>]

Hi there,

Sorry for barging in, but maybe this is a place where Drill and MetaModel
could benefit from each other? We've considered that before at least ...

MetaModel already has support for both DOM and SAX based XML querying. They
basically inherit some characteristics from DOM and SAX respectively:

 - In the DOM variant we can infer a schema and all the user has to do is
select a XML file/resource anywhere.
 - In the SAX variant the user has to specify which paths in the XML
document should represent logical "tables" and what paths represent their
columns.

See [1] for more info. Hope this might be of interest to integrate into
Drill?

Best regards,
Kasper Sørensen (from the MetaModel project)

[1] http://wiki.apache.org/metamodel/examples/XmlTableMapping

2015-10-18 0:35 GMT+02:00 Magnus Pierre <mp...@maprtech.com>:

> Well, very few lines of code imho. And simple. Been able to parse pretty
> deep structures with no issues so far. Performance? 10-15 5mb xml's in less
> than a second on my laptop but then I run it using Storm with some
> parallelism in place. Don't know if it's good or bad. I'll share the code
> next time I use computer. You don't need to use it, but it works at least.
>
> /M
> Den 17 okt 2015 10:43 em skrev "Matt Burgess" <ma...@gmail.com>:
>
> > If the converter is clean and performant then I'm sure the community
> > (including me) is interested :)
> >
> > However I wonder if Drill can afford to add a translation layer between
> > data formats, could we be better served with similar parsing in Drill for
> > XML as we do for JSON, or can it be pushed down far enough (to the
> parser)
> > to not make a noticeable difference (which is what I think Julian is
> > implying)?
> >
> > Sent from my iPhone
> >
> > > On Oct 17, 2015, at 1:41 PM, Magnus Pierre <mp...@maprtech.com>
> wrote:
> > >
> > > Hello,
> > >
> > > Just wrote a simple sax implementation that converts xml to json and
> that
> > > is able to deal with decently complex xml's, that I currently use in
> > Storm.
> > > Takes attributes, and everything.
> > >
> > > I can share it with the community if interesting.
> > >
> > > /Magnus
> > > Den 17 okt 2015 7:02 em skrev "Julian Hyde" <ju...@hydromatic.net>:
> > >
> > >> Seems to me the biggest problem is to make drill understand the nested
> > >> structure of an xml document. That work has been done for json, so
> let's
> > >> build on it. Suppose there was a translator that converted xml to json
> > >> (adding attributes for things that json lacks, such as namespaces,
> text,
> > >> element tags). Drill knows how to handle json, even if it is a bit
> > verbose.
> > >> The translator could be applied on the fly.
> > >>
> > >> Julian
> > >>
> > >>
> > >>
> > >> Sent from my iPad
> > >>>> On Oct 16, 2015, at 2:31 PM, Stefán Baxter <
> stefan@activitystream.com
> > >
> > >>> wrote:
> > >>>
> > >>> Hi,
> > >>>
> > >>> It's not possible but there has been some talk here about supporting
> > it.
> > >>> If I remember correctly it's rather complicated and not really
> > feasible.
> > >>> (I'm just a newbie so don't take my words for it)
> > >>>
> > >>>
> > >>> Regards,
> > >>> -Stefan
> > >>>
> > >>> On Fri, Oct 16, 2015 at 8:54 PM, Daniel Ajo <
> > Daniel.Ajo@abarcahealth.com
> > >>>
> > >>> wrote:
> > >>>
> > >>>> Hey there,
> > >>>>
> > >>>> I was wondering if it is possible to query XML files using Apache
> > Drill?
> > >>>>
> > >>>> I see there are several formats, and maybe it would work using an
> > xpath
> > >>>> query of some sorts, but just wondering if it would work to directly
> > >> query
> > >>>> it using some sort of plug-in.
> > >>>>
> > >>>> Well, let me know,
> > >>>>
> > >>>> Daniel Ajo
> > >>>> *********************************************************
> > >> CONFIDENTIALITY
> > >>>> NOTE: This electronic transmission contains information belonging to
> > >> Abarca
> > >>>> Health LLC, which is confidential or legally privileged. If you are
> > not
> > >> the
> > >>>> intended recipient, please immediately advise the sender by reply
> > >> e-mail or
> > >>>> telephone that this message has been inadvertently transmitted to
> you
> > >> and
> > >>>> delete this e-mail from your system. If you have received this
> > >> transmission
> > >>>> in error, you are hereby notified that any disclosure, copying,
> > >>>> distribution or the taking of any action in reliance on the contents
> > of
> > >> the
> > >>>> information is strictly prohibited.
> > >>
> >
>

Re: Apache Drill

[Magnus Pierre <mp...@maprtech.com>]

Well, very few lines of code imho. And simple. Been able to parse pretty
deep structures with no issues so far. Performance? 10-15 5mb xml's in less
than a second on my laptop but then I run it using Storm with some
parallelism in place. Don't know if it's good or bad. I'll share the code
next time I use computer. You don't need to use it, but it works at least.

/M
Den 17 okt 2015 10:43 em skrev "Matt Burgess" <ma...@gmail.com>:

> If the converter is clean and performant then I'm sure the community
> (including me) is interested :)
>
> However I wonder if Drill can afford to add a translation layer between
> data formats, could we be better served with similar parsing in Drill for
> XML as we do for JSON, or can it be pushed down far enough (to the parser)
> to not make a noticeable difference (which is what I think Julian is
> implying)?
>
> Sent from my iPhone
>
> > On Oct 17, 2015, at 1:41 PM, Magnus Pierre <mp...@maprtech.com> wrote:
> >
> > Hello,
> >
> > Just wrote a simple sax implementation that converts xml to json and that
> > is able to deal with decently complex xml's, that I currently use in
> Storm.
> > Takes attributes, and everything.
> >
> > I can share it with the community if interesting.
> >
> > /Magnus
> > Den 17 okt 2015 7:02 em skrev "Julian Hyde" <ju...@hydromatic.net>:
> >
> >> Seems to me the biggest problem is to make drill understand the nested
> >> structure of an xml document. That work has been done for json, so let's
> >> build on it. Suppose there was a translator that converted xml to json
> >> (adding attributes for things that json lacks, such as namespaces, text,
> >> element tags). Drill knows how to handle json, even if it is a bit
> verbose.
> >> The translator could be applied on the fly.
> >>
> >> Julian
> >>
> >>
> >>
> >> Sent from my iPad
> >>>> On Oct 16, 2015, at 2:31 PM, Stefán Baxter <stefan@activitystream.com
> >
> >>> wrote:
> >>>
> >>> Hi,
> >>>
> >>> It's not possible but there has been some talk here about supporting
> it.
> >>> If I remember correctly it's rather complicated and not really
> feasible.
> >>> (I'm just a newbie so don't take my words for it)
> >>>
> >>>
> >>> Regards,
> >>> -Stefan
> >>>
> >>> On Fri, Oct 16, 2015 at 8:54 PM, Daniel Ajo <
> Daniel.Ajo@abarcahealth.com
> >>>
> >>> wrote:
> >>>
> >>>> Hey there,
> >>>>
> >>>> I was wondering if it is possible to query XML files using Apache
> Drill?
> >>>>
> >>>> I see there are several formats, and maybe it would work using an
> xpath
> >>>> query of some sorts, but just wondering if it would work to directly
> >> query
> >>>> it using some sort of plug-in.
> >>>>
> >>>> Well, let me know,
> >>>>
> >>>> Daniel Ajo
> >>>> *********************************************************
> >> CONFIDENTIALITY
> >>>> NOTE: This electronic transmission contains information belonging to
> >> Abarca
> >>>> Health LLC, which is confidential or legally privileged. If you are
> not
> >> the
> >>>> intended recipient, please immediately advise the sender by reply
> >> e-mail or
> >>>> telephone that this message has been inadvertently transmitted to you
> >> and
> >>>> delete this e-mail from your system. If you have received this
> >> transmission
> >>>> in error, you are hereby notified that any disclosure, copying,
> >>>> distribution or the taking of any action in reliance on the contents
> of
> >> the
> >>>> information is strictly prohibited.
> >>
>

Re: Apache Drill

[Ted Dunning <te...@gmail.com>]

The real problems for XML with Drill are

a) distinguishing an element from a list with a single member

b) seeing empty lists

c) determining when intervening whitespace can be discarded

A schema can help a lot in deciding these issues.


On Sat, Oct 17, 2015 at 1:43 PM, Matt Burgess <ma...@gmail.com> wrote:

> If the converter is clean and performant then I'm sure the community
> (including me) is interested :)
>
> However I wonder if Drill can afford to add a translation layer between
> data formats, could we be better served with similar parsing in Drill for
> XML as we do for JSON, or can it be pushed down far enough (to the parser)
> to not make a noticeable difference (which is what I think Julian is
> implying)?
>
> Sent from my iPhone
>
> > On Oct 17, 2015, at 1:41 PM, Magnus Pierre <mp...@maprtech.com> wrote:
> >
> > Hello,
> >
> > Just wrote a simple sax implementation that converts xml to json and that
> > is able to deal with decently complex xml's, that I currently use in
> Storm.
> > Takes attributes, and everything.
> >
> > I can share it with the community if interesting.
> >
> > /Magnus
> > Den 17 okt 2015 7:02 em skrev "Julian Hyde" <ju...@hydromatic.net>:
> >
> >> Seems to me the biggest problem is to make drill understand the nested
> >> structure of an xml document. That work has been done for json, so let's
> >> build on it. Suppose there was a translator that converted xml to json
> >> (adding attributes for things that json lacks, such as namespaces, text,
> >> element tags). Drill knows how to handle json, even if it is a bit
> verbose.
> >> The translator could be applied on the fly.
> >>
> >> Julian
> >>
> >>
> >>
> >> Sent from my iPad
> >>>> On Oct 16, 2015, at 2:31 PM, Stefán Baxter <stefan@activitystream.com
> >
> >>> wrote:
> >>>
> >>> Hi,
> >>>
> >>> It's not possible but there has been some talk here about supporting
> it.
> >>> If I remember correctly it's rather complicated and not really
> feasible.
> >>> (I'm just a newbie so don't take my words for it)
> >>>
> >>>
> >>> Regards,
> >>> -Stefan
> >>>
> >>> On Fri, Oct 16, 2015 at 8:54 PM, Daniel Ajo <
> Daniel.Ajo@abarcahealth.com
> >>>
> >>> wrote:
> >>>
> >>>> Hey there,
> >>>>
> >>>> I was wondering if it is possible to query XML files using Apache
> Drill?
> >>>>
> >>>> I see there are several formats, and maybe it would work using an
> xpath
> >>>> query of some sorts, but just wondering if it would work to directly
> >> query
> >>>> it using some sort of plug-in.
> >>>>
> >>>> Well, let me know,
> >>>>
> >>>> Daniel Ajo
> >>>> *********************************************************
> >> CONFIDENTIALITY
> >>>> NOTE: This electronic transmission contains information belonging to
> >> Abarca
> >>>> Health LLC, which is confidential or legally privileged. If you are
> not
> >> the
> >>>> intended recipient, please immediately advise the sender by reply
> >> e-mail or
> >>>> telephone that this message has been inadvertently transmitted to you
> >> and
> >>>> delete this e-mail from your system. If you have received this
> >> transmission
> >>>> in error, you are hereby notified that any disclosure, copying,
> >>>> distribution or the taking of any action in reliance on the contents
> of
> >> the
> >>>> information is strictly prohibited.
> >>
>

Re: Apache Drill

[Matt Burgess <ma...@gmail.com>]

If the converter is clean and performant then I'm sure the community (including me) is interested :)

However I wonder if Drill can afford to add a translation layer between data formats, could we be better served with similar parsing in Drill for XML as we do for JSON, or can it be pushed down far enough (to the parser) to not make a noticeable difference (which is what I think Julian is implying)?

Sent from my iPhone

> On Oct 17, 2015, at 1:41 PM, Magnus Pierre <mp...@maprtech.com> wrote:
> 
> Hello,
> 
> Just wrote a simple sax implementation that converts xml to json and that
> is able to deal with decently complex xml's, that I currently use in Storm.
> Takes attributes, and everything.
> 
> I can share it with the community if interesting.
> 
> /Magnus
> Den 17 okt 2015 7:02 em skrev "Julian Hyde" <ju...@hydromatic.net>:
> 
>> Seems to me the biggest problem is to make drill understand the nested
>> structure of an xml document. That work has been done for json, so let's
>> build on it. Suppose there was a translator that converted xml to json
>> (adding attributes for things that json lacks, such as namespaces, text,
>> element tags). Drill knows how to handle json, even if it is a bit verbose.
>> The translator could be applied on the fly.
>> 
>> Julian
>> 
>> 
>> 
>> Sent from my iPad
>>>> On Oct 16, 2015, at 2:31 PM, Stefán Baxter <st...@activitystream.com>
>>> wrote:
>>> 
>>> Hi,
>>> 
>>> It's not possible but there has been some talk here about supporting it.
>>> If I remember correctly it's rather complicated and not really feasible.
>>> (I'm just a newbie so don't take my words for it)
>>> 
>>> 
>>> Regards,
>>> -Stefan
>>> 
>>> On Fri, Oct 16, 2015 at 8:54 PM, Daniel Ajo <Daniel.Ajo@abarcahealth.com
>>> 
>>> wrote:
>>> 
>>>> Hey there,
>>>> 
>>>> I was wondering if it is possible to query XML files using Apache Drill?
>>>> 
>>>> I see there are several formats, and maybe it would work using an xpath
>>>> query of some sorts, but just wondering if it would work to directly
>> query
>>>> it using some sort of plug-in.
>>>> 
>>>> Well, let me know,
>>>> 
>>>> Daniel Ajo
>>>> *********************************************************
>> CONFIDENTIALITY
>>>> NOTE: This electronic transmission contains information belonging to
>> Abarca
>>>> Health LLC, which is confidential or legally privileged. If you are not
>> the
>>>> intended recipient, please immediately advise the sender by reply
>> e-mail or
>>>> telephone that this message has been inadvertently transmitted to you
>> and
>>>> delete this e-mail from your system. If you have received this
>> transmission
>>>> in error, you are hereby notified that any disclosure, copying,
>>>> distribution or the taking of any action in reliance on the contents of
>> the
>>>> information is strictly prohibited.
>> 

Re: Apache Drill

[Magnus Pierre <mp...@maprtech.com>]

Hello,

Just wrote a simple sax implementation that converts xml to json and that
is able to deal with decently complex xml's, that I currently use in Storm.
Takes attributes, and everything.

I can share it with the community if interesting.

/Magnus
Den 17 okt 2015 7:02 em skrev "Julian Hyde" <ju...@hydromatic.net>:

> Seems to me the biggest problem is to make drill understand the nested
> structure of an xml document. That work has been done for json, so let's
> build on it. Suppose there was a translator that converted xml to json
> (adding attributes for things that json lacks, such as namespaces, text,
> element tags). Drill knows how to handle json, even if it is a bit verbose.
> The translator could be applied on the fly.
>
> Julian
>
>
>
> Sent from my iPad
> > On Oct 16, 2015, at 2:31 PM, Stefán Baxter <st...@activitystream.com>
> wrote:
> >
> > Hi,
> >
> > It's not possible but there has been some talk here about supporting it.
> > If I remember correctly it's rather complicated and not really feasible.
> > (I'm just a newbie so don't take my words for it)
> >
> >
> > Regards,
> > -Stefan
> >
> > On Fri, Oct 16, 2015 at 8:54 PM, Daniel Ajo <Daniel.Ajo@abarcahealth.com
> >
> > wrote:
> >
> >> Hey there,
> >>
> >> I was wondering if it is possible to query XML files using Apache Drill?
> >>
> >> I see there are several formats, and maybe it would work using an xpath
> >> query of some sorts, but just wondering if it would work to directly
> query
> >> it using some sort of plug-in.
> >>
> >> Well, let me know,
> >>
> >> Daniel Ajo
> >> *********************************************************
> CONFIDENTIALITY
> >> NOTE: This electronic transmission contains information belonging to
> Abarca
> >> Health LLC, which is confidential or legally privileged. If you are not
> the
> >> intended recipient, please immediately advise the sender by reply
> e-mail or
> >> telephone that this message has been inadvertently transmitted to you
> and
> >> delete this e-mail from your system. If you have received this
> transmission
> >> in error, you are hereby notified that any disclosure, copying,
> >> distribution or the taking of any action in reliance on the contents of
> the
> >> information is strictly prohibited.
> >>
>

Re: Apache Drill

[Julian Hyde <ju...@hydromatic.net>]

Seems to me the biggest problem is to make drill understand the nested structure of an xml document. That work has been done for json, so let's build on it. Suppose there was a translator that converted xml to json (adding attributes for things that json lacks, such as namespaces, text, element tags). Drill knows how to handle json, even if it is a bit verbose. The translator could be applied on the fly. 

Julian



Sent from my iPad
> On Oct 16, 2015, at 2:31 PM, Stefán Baxter <st...@activitystream.com> wrote:
> 
> Hi,
> 
> It's not possible but there has been some talk here about supporting it.
> If I remember correctly it's rather complicated and not really feasible.
> (I'm just a newbie so don't take my words for it)
> 
> 
> Regards,
> -Stefan
> 
> On Fri, Oct 16, 2015 at 8:54 PM, Daniel Ajo <Da...@abarcahealth.com>
> wrote:
> 
>> Hey there,
>> 
>> I was wondering if it is possible to query XML files using Apache Drill?
>> 
>> I see there are several formats, and maybe it would work using an xpath
>> query of some sorts, but just wondering if it would work to directly query
>> it using some sort of plug-in.
>> 
>> Well, let me know,
>> 
>> Daniel Ajo
>> ********************************************************* CONFIDENTIALITY
>> NOTE: This electronic transmission contains information belonging to Abarca
>> Health LLC, which is confidential or legally privileged. If you are not the
>> intended recipient, please immediately advise the sender by reply e-mail or
>> telephone that this message has been inadvertently transmitted to you and
>> delete this e-mail from your system. If you have received this transmission
>> in error, you are hereby notified that any disclosure, copying,
>> distribution or the taking of any action in reliance on the contents of the
>> information is strictly prohibited.
>> 

Re: Apache Drill

[Stefán Baxter <st...@activitystream.com>]

Hi,

It's not possible but there has been some talk here about supporting it.
If I remember correctly it's rather complicated and not really feasible.
(I'm just a newbie so don't take my words for it)


Regards,
 -Stefan

On Fri, Oct 16, 2015 at 8:54 PM, Daniel Ajo <Da...@abarcahealth.com>
wrote:

> Hey there,
>
> I was wondering if it is possible to query XML files using Apache Drill?
>
> I see there are several formats, and maybe it would work using an xpath
> query of some sorts, but just wondering if it would work to directly query
> it using some sort of plug-in.
>
> Well, let me know,
>
> Daniel Ajo
> ********************************************************* CONFIDENTIALITY
> NOTE: This electronic transmission contains information belonging to Abarca
> Health LLC, which is confidential or legally privileged. If you are not the
> intended recipient, please immediately advise the sender by reply e-mail or
> telephone that this message has been inadvertently transmitted to you and
> delete this e-mail from your system. If you have received this transmission
> in error, you are hereby notified that any disclosure, copying,
> distribution or the taking of any action in reliance on the contents of the
> information is strictly prohibited.
>