You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@metron.apache.org by Nick Allen <ni...@nickallen.org> on 2017/02/24 15:12:19 UTC
JSONMapParser Normalizer aka Flattener
Per Otto's advice, I am looking to reuse the normalizer/flattener mechanism
that currently exists in JSONMapParser. It looks like the mechanism is
built into the class, so I will have to extract it. It looks like landing
it in JSONUtils is a logical place.
It appears that the mechanism only handles maps, not lists. Is that true?
I will need to add similar functionality for lists to reuse this for
METRON-686.
Re: JSONMapParser Normalizer aka Flattener
Posted by Matt Foley <ma...@apache.org>.
I view “flattening” as moving from a highly functional format (JSON) to a less functional format (flat, text-like).
Many of the less functional formats we see, that do not permit hierarchical Maps, nevertheless permit List<String> and List<Number>. (No, not List<Map>, obviously.)
A typical, and workable, flattening would be comma-separated substrings (the below illustrates a possible escape):
"list": ["e1", "e2,foo", "e3"]
->
list: "e1,e2\,foo,e3" or " e1,e2,foo,e3"
A list of numbers, with period decimal mark, also does well as a comma-separated string.
I would suggest that only List<String> and List<Number> be flattened this way.
This allows us to express the kind of formats that are typical in flat configuration parameter files used by many other systems.
List<Object> could be reasonably flattened with dotted indexes, as you suggest, or left for a later implementation.
--Matt
On 2/24/17, 7:34 AM, "Nick Allen" <ni...@nickallen.org> wrote:
And sorry this should be...
"list": ["e1", "e2", "e3"]
->
list.0: e1
list.1: e2
list.2: e3
On Fri, Feb 24, 2017 at 10:26 AM, Nick Allen <ni...@nickallen.org> wrote:
> So I don't need to unfold lists, but I do maps? I thought the commentary
> on METRON-686 was that Solr cannot handle "complex types". I took that to
> mean both maps and lists.
>
> Yes, Otto. The only reasonable way to unfold a list would be using the
> index of the element.
>
> "list": ["e1", "e2", "e3"]
> ->
> list.0.e1
> list.1.e2
> list.2.e3
>
>
> I don't want to unfold lists or maps. :)
>
>
>
>
> On Fri, Feb 24, 2017 at 10:17 AM, Otto Fowler <ot...@gmail.com>
> wrote:
>
>> No, I don’t think it does. I am not sure how you would do that, other
>> than putting a number at the end of the unwrapped array item?
>>
>>
>>
>> On February 24, 2017 at 10:12:26, Nick Allen (nick@nickallen.org) wrote:
>>
>> Per Otto's advice, I am looking to reuse the normalizer/flattener
>> mechanism
>> that currently exists in JSONMapParser. It looks like the mechanism is
>> built into the class, so I will have to extract it. It looks like landing
>> it in JSONUtils is a logical place.
>>
>> It appears that the mechanism only handles maps, not lists. Is that true?
>> I will need to add similar functionality for lists to reuse this for
>> METRON-686.
>>
>>
>
Re: JSONMapParser Normalizer aka Flattener
Posted by Nick Allen <ni...@nickallen.org>.
And sorry this should be...
"list": ["e1", "e2", "e3"]
->
list.0: e1
list.1: e2
list.2: e3
On Fri, Feb 24, 2017 at 10:26 AM, Nick Allen <ni...@nickallen.org> wrote:
> So I don't need to unfold lists, but I do maps? I thought the commentary
> on METRON-686 was that Solr cannot handle "complex types". I took that to
> mean both maps and lists.
>
> Yes, Otto. The only reasonable way to unfold a list would be using the
> index of the element.
>
> "list": ["e1", "e2", "e3"]
> ->
> list.0.e1
> list.1.e2
> list.2.e3
>
>
> I don't want to unfold lists or maps. :)
>
>
>
>
> On Fri, Feb 24, 2017 at 10:17 AM, Otto Fowler <ot...@gmail.com>
> wrote:
>
>> No, I don’t think it does. I am not sure how you would do that, other
>> than putting a number at the end of the unwrapped array item?
>>
>>
>>
>> On February 24, 2017 at 10:12:26, Nick Allen (nick@nickallen.org) wrote:
>>
>> Per Otto's advice, I am looking to reuse the normalizer/flattener
>> mechanism
>> that currently exists in JSONMapParser. It looks like the mechanism is
>> built into the class, so I will have to extract it. It looks like landing
>> it in JSONUtils is a logical place.
>>
>> It appears that the mechanism only handles maps, not lists. Is that true?
>> I will need to add similar functionality for lists to reuse this for
>> METRON-686.
>>
>>
>
Re: JSONMapParser Normalizer aka Flattener
Posted by Otto Fowler <ot...@gmail.com>.
+1
On February 24, 2017 at 11:05:21, Nick Allen (nick@nickallen.org) wrote:
I'll just flatten maps like we do in the JSONMapParser and leave it at
that. I will not do anything with lists. We can clean this up as part of
METRON-735.
On Fri, Feb 24, 2017 at 11:00 AM, Otto Fowler <ot...@gmail.com>
wrote:
> Tells you how much solr stuff we have going on.
> This is all yucky. Then again I voted for 1.
>
>
>
> On February 24, 2017 at 10:26:52, Nick Allen (nick@nickallen.org) wrote:
>
> So I don't need to unfold lists, but I do maps? I thought the commentary
> on METRON-686 was that Solr cannot handle "complex types". I took that to
> mean both maps and lists.
>
> Yes, Otto. The only reasonable way to unfold a list would be using the
> index of the element.
>
> "list": ["e1", "e2", "e3"]
> ->
> list.0.e1
> list.1.e2
> list.2.e3
>
>
> I don't want to unfold lists or maps. :)
>
>
>
>
> On Fri, Feb 24, 2017 at 10:17 AM, Otto Fowler <ot...@gmail.com>
> wrote:
>
>> No, I don’t think it does. I am not sure how you would do that, other
>> than putting a number at the end of the unwrapped array item?
>>
>>
>>
>> On February 24, 2017 at 10:12:26, Nick Allen (nick@nickallen.org) wrote:
>>
>> Per Otto's advice, I am looking to reuse the normalizer/flattener
>> mechanism
>> that currently exists in JSONMapParser. It looks like the mechanism is
>> built into the class, so I will have to extract it. It looks like landing
>> it in JSONUtils is a logical place.
>>
>> It appears that the mechanism only handles maps, not lists. Is that true?
>> I will need to add similar functionality for lists to reuse this for
>> METRON-686.
>>
>>
>
Re: JSONMapParser Normalizer aka Flattener
Posted by Nick Allen <ni...@nickallen.org>.
I'll just flatten maps like we do in the JSONMapParser and leave it at
that. I will not do anything with lists. We can clean this up as part of
METRON-735.
On Fri, Feb 24, 2017 at 11:00 AM, Otto Fowler <ot...@gmail.com>
wrote:
> Tells you how much solr stuff we have going on.
> This is all yucky. Then again I voted for 1.
>
>
>
> On February 24, 2017 at 10:26:52, Nick Allen (nick@nickallen.org) wrote:
>
> So I don't need to unfold lists, but I do maps? I thought the commentary
> on METRON-686 was that Solr cannot handle "complex types". I took that to
> mean both maps and lists.
>
> Yes, Otto. The only reasonable way to unfold a list would be using the
> index of the element.
>
> "list": ["e1", "e2", "e3"]
> ->
> list.0.e1
> list.1.e2
> list.2.e3
>
>
> I don't want to unfold lists or maps. :)
>
>
>
>
> On Fri, Feb 24, 2017 at 10:17 AM, Otto Fowler <ot...@gmail.com>
> wrote:
>
>> No, I don’t think it does. I am not sure how you would do that, other
>> than putting a number at the end of the unwrapped array item?
>>
>>
>>
>> On February 24, 2017 at 10:12:26, Nick Allen (nick@nickallen.org) wrote:
>>
>> Per Otto's advice, I am looking to reuse the normalizer/flattener
>> mechanism
>> that currently exists in JSONMapParser. It looks like the mechanism is
>> built into the class, so I will have to extract it. It looks like landing
>> it in JSONUtils is a logical place.
>>
>> It appears that the mechanism only handles maps, not lists. Is that true?
>> I will need to add similar functionality for lists to reuse this for
>> METRON-686.
>>
>>
>
Re: JSONMapParser Normalizer aka Flattener
Posted by Otto Fowler <ot...@gmail.com>.
Tells you how much solr stuff we have going on.
This is all yucky. Then again I voted for 1.
On February 24, 2017 at 10:26:52, Nick Allen (nick@nickallen.org) wrote:
So I don't need to unfold lists, but I do maps? I thought the commentary
on METRON-686 was that Solr cannot handle "complex types". I took that to
mean both maps and lists.
Yes, Otto. The only reasonable way to unfold a list would be using the
index of the element.
"list": ["e1", "e2", "e3"]
->
list.0.e1
list.1.e2
list.2.e3
I don't want to unfold lists or maps. :)
On Fri, Feb 24, 2017 at 10:17 AM, Otto Fowler <ot...@gmail.com>
wrote:
> No, I don’t think it does. I am not sure how you would do that, other
> than putting a number at the end of the unwrapped array item?
>
>
>
> On February 24, 2017 at 10:12:26, Nick Allen (nick@nickallen.org) wrote:
>
> Per Otto's advice, I am looking to reuse the normalizer/flattener mechanism
> that currently exists in JSONMapParser. It looks like the mechanism is
> built into the class, so I will have to extract it. It looks like landing
> it in JSONUtils is a logical place.
>
> It appears that the mechanism only handles maps, not lists. Is that true?
> I will need to add similar functionality for lists to reuse this for
> METRON-686.
>
>
Re: JSONMapParser Normalizer aka Flattener
Posted by Nick Allen <ni...@nickallen.org>.
So I don't need to unfold lists, but I do maps? I thought the commentary
on METRON-686 was that Solr cannot handle "complex types". I took that to
mean both maps and lists.
Yes, Otto. The only reasonable way to unfold a list would be using the
index of the element.
"list": ["e1", "e2", "e3"]
->
list.0.e1
list.1.e2
list.2.e3
I don't want to unfold lists or maps. :)
On Fri, Feb 24, 2017 at 10:17 AM, Otto Fowler <ot...@gmail.com>
wrote:
> No, I don’t think it does. I am not sure how you would do that, other
> than putting a number at the end of the unwrapped array item?
>
>
>
> On February 24, 2017 at 10:12:26, Nick Allen (nick@nickallen.org) wrote:
>
> Per Otto's advice, I am looking to reuse the normalizer/flattener
> mechanism
> that currently exists in JSONMapParser. It looks like the mechanism is
> built into the class, so I will have to extract it. It looks like landing
> it in JSONUtils is a logical place.
>
> It appears that the mechanism only handles maps, not lists. Is that true?
> I will need to add similar functionality for lists to reuse this for
> METRON-686.
>
>
Re: JSONMapParser Normalizer aka Flattener
Posted by Otto Fowler <ot...@gmail.com>.
No, I don’t think it does. I am not sure how you would do that, other than
putting a number at the end of the unwrapped array item?
On February 24, 2017 at 10:12:26, Nick Allen (nick@nickallen.org) wrote:
Per Otto's advice, I am looking to reuse the normalizer/flattener mechanism
that currently exists in JSONMapParser. It looks like the mechanism is
built into the class, so I will have to extract it. It looks like landing
it in JSONUtils is a logical place.
It appears that the mechanism only handles maps, not lists. Is that true?
I will need to add similar functionality for lists to reuse this for
METRON-686.
Re: JSONMapParser Normalizer aka Flattener
Posted by Nick Allen <ni...@nickallen.org>.
Again, I don't want to do it, if we don't have to, but it is definitely
possible, no?
A map embedded in a list like...
"list": [
"e1",
{
"k1": "v1",
"k2": "v2"
},
"e3"
]
would translate to..
list.0: e1
list.1.k1: v1
list.1.k2: v2
list.2: e3
On Fri, Feb 24, 2017 at 10:26 AM, Otto Fowler <ot...@gmail.com>
wrote:
> I agree with this, flattening the map doesn’t change it’s meaning - it is
> not arbitrary.
> Finding an item still makes sense, what I mean is the flattened name is
> meaningful.
>
> What you would have to do for an array would make it not meaningful, since
> it would just be an
> arbitrary positional number in the name.
>
> Now, how should we handle lists that contain maps? Don’t flatten the list,
> but flatten the maps inside?
>
>
> On February 24, 2017 at 10:15:02, Casey Stella (cestella@gmail.com) wrote:
>
> I don't know, I think I'm ok with lists, but I might be biased. I think
> it's the nested maps that were the issue. Flattening lists seems...wrong
> to me. Maybe that's wrong-headed, but there it is. ;)
>
> On Fri, Feb 24, 2017 at 10:12 AM, Nick Allen <ni...@nickallen.org> wrote:
>
> > Per Otto's advice, I am looking to reuse the normalizer/flattener
> mechanism
> > that currently exists in JSONMapParser. It looks like the mechanism is
> > built into the class, so I will have to extract it. It looks like landing
> > it in JSONUtils is a logical place.
> >
> > It appears that the mechanism only handles maps, not lists. Is that true?
> > I will need to add similar functionality for lists to reuse this for
> > METRON-686.
> >
>
Re: JSONMapParser Normalizer aka Flattener
Posted by Otto Fowler <ot...@gmail.com>.
I agree with this, flattening the map doesn’t change it’s meaning - it is
not arbitrary.
Finding an item still makes sense, what I mean is the flattened name is
meaningful.
What you would have to do for an array would make it not meaningful, since
it would just be an
arbitrary positional number in the name.
Now, how should we handle lists that contain maps? Don’t flatten the list,
but flatten the maps inside?
On February 24, 2017 at 10:15:02, Casey Stella (cestella@gmail.com) wrote:
I don't know, I think I'm ok with lists, but I might be biased. I think
it's the nested maps that were the issue. Flattening lists seems...wrong
to me. Maybe that's wrong-headed, but there it is. ;)
On Fri, Feb 24, 2017 at 10:12 AM, Nick Allen <ni...@nickallen.org> wrote:
> Per Otto's advice, I am looking to reuse the normalizer/flattener
mechanism
> that currently exists in JSONMapParser. It looks like the mechanism is
> built into the class, so I will have to extract it. It looks like landing
> it in JSONUtils is a logical place.
>
> It appears that the mechanism only handles maps, not lists. Is that true?
> I will need to add similar functionality for lists to reuse this for
> METRON-686.
>
Re: JSONMapParser Normalizer aka Flattener
Posted by Casey Stella <ce...@gmail.com>.
I don't know, I think I'm ok with lists, but I might be biased. I think
it's the nested maps that were the issue. Flattening lists seems...wrong
to me. Maybe that's wrong-headed, but there it is. ;)
On Fri, Feb 24, 2017 at 10:12 AM, Nick Allen <ni...@nickallen.org> wrote:
> Per Otto's advice, I am looking to reuse the normalizer/flattener mechanism
> that currently exists in JSONMapParser. It looks like the mechanism is
> built into the class, so I will have to extract it. It looks like landing
> it in JSONUtils is a logical place.
>
> It appears that the mechanism only handles maps, not lists. Is that true?
> I will need to add similar functionality for lists to reuse this for
> METRON-686.
>