You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@drill.apache.org by Edmon Begoli <eb...@gmail.com> on 2015/09/16 23:17:53 UTC
The meaning/intent of the following methods - JSON storage format -
part 1
I am studying the JSON Storage plugin.
Can someone please answer what is the meaning (or intent) of the following
methods:
# Question 1:
# This is from inside the getter for RecordWriter. What is major/minor
fragment id?
String fragmentId = String.format("%d_%d", handle.getMajorFragmentId(),
handle.getMinorFragmentId());
# Question 2:
# Speficially, what is Operator Type, and what is JSON specific about it?
public int getReaderOperatorType() {
return CoreOperatorType.JSON_SUB_SCAN_VALUE;
}
# Question 3:
# same, but for writer
@Override
public int getWriterOperatorType() {
throw new UnsupportedOperationException();
}
# Question 4:
# I generally understand the concept of predicate pushdown, but is this
about predicate push down or something else?
# How does one implement a push down for storage format?
@Override
public boolean supportsPushDown() {
# Question 5:
# Is this int value of 31 purely randomly selected, or is there an index
somewhere?
@Override
public int hashCode() {
return 31;
}
Re: The meaning/intent of the following methods - JSON storage format
- part 1
Posted by Abdel Hakim Deneche <ad...@maprtech.com>.
Hi Edmon,
About Question 1, you can read the following page, that gives a pretty good
explanation about major and minor fragments:
https://drill.apache.org/docs/drill-query-execution/
On Wed, Sep 16, 2015 at 2:17 PM, Edmon Begoli <eb...@gmail.com> wrote:
> I am studying the JSON Storage plugin.
>
> Can someone please answer what is the meaning (or intent) of the following
> methods:
>
> # Question 1:
> # This is from inside the getter for RecordWriter. What is major/minor
> fragment id?
> String fragmentId = String.format("%d_%d", handle.getMajorFragmentId(),
> handle.getMinorFragmentId());
>
>
> # Question 2:
> # Speficially, what is Operator Type, and what is JSON specific about it?
> public int getReaderOperatorType() {
> return CoreOperatorType.JSON_SUB_SCAN_VALUE;
> }
>
> # Question 3:
> # same, but for writer
> @Override
> public int getWriterOperatorType() {
> throw new UnsupportedOperationException();
> }
>
> # Question 4:
> # I generally understand the concept of predicate pushdown, but is this
> about predicate push down or something else?
> # How does one implement a push down for storage format?
> @Override
> public boolean supportsPushDown() {
>
> # Question 5:
> # Is this int value of 31 purely randomly selected, or is there an index
> somewhere?
> @Override
> public int hashCode() {
> return 31;
> }
>
--
Abdelhakim Deneche
Software Engineer
<http://www.mapr.com/>
Now Available - Free Hadoop On-Demand Training
<http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available>
Re: The meaning/intent of the following methods - JSON storage format
- part 1
Posted by Jacques Nadeau <ja...@dremio.com>.
31:
I think that is the default behavior of Eclipse when it generates a
hashcode for a class that has no properties.
--
Jacques Nadeau
CTO and Co-Founder, Dremio
On Tue, Sep 22, 2015 at 5:00 PM, Ted Dunning <te...@gmail.com> wrote:
> Actually, returning a constant from a hashCode function is pretty goofy.
>
> In the String implementation 31 is a *factor*.
>
>
>
> On Tue, Sep 22, 2015 at 4:25 PM, Jinfeng Ni <ji...@gmail.com> wrote:
>
> > For question 5, this link [1] probably explains why 31 is used:
> >
> >
> > [1]
> >
> http://stackoverflow.com/questions/299304/why-does-javas-hashcode-in-string-use-31-as-a-multiplier
> >
> >
> > On Tue, Sep 22, 2015 at 10:05 AM, Abdel Hakim Deneche
> > <ad...@maprtech.com> wrote:
> > > Question 3:
> > > I would assume getWriterOperatorType() has a similar meaning to
> > > getReaderOperatorType() but it's not used anywhere in Drill. That would
> > > explain why JsonFormatPlugin throws an UnsupportedOperationException
> > >
> > > Question 4:
> > > supportsPushDown() is about "projection" push down and not filter push
> > down
> > >
> > > Question 5:
> > > I would say, purely random.
> > >
> > > On Fri, Sep 18, 2015 at 11:48 AM, Edmon Begoli <eb...@gmail.com>
> > wrote:
> > >
> > >> Thanks, Abdel.
> > >> I will use your answers to guide my development, but I will also
> > contribute
> > >> them back as a javadoc.
> > >>
> > >> On Fri, Sep 18, 2015 at 2:39 PM, Abdel Hakim Deneche <
> > >> adeneche@maprtech.com>
> > >> wrote:
> > >>
> > >> > Question 2:
> > >> > to my knowledge (what I've found digging through the code). Each
> > Operator
> > >> > has a unique OperatorType that is used when writing the operator's
> > stats
> > >> in
> > >> > the profile. So in this case, Each FormatPlugin implementation
> should
> > >> have
> > >> > a corresponding unique OperatorType added to CoreOperatorType.
> > >> >
> > >> > I will get back to you after I get more information about the
> > remaining
> > >> > questions.
> > >> >
> > >> > Thanks
> > >> >
> > >> > On Thu, Sep 17, 2015 at 8:14 PM, Edmon Begoli <eb...@gmail.com>
> > wrote:
> > >> >
> > >> > > Hello all,
> > >> > >
> > >> > > Could some please with answering my questions 2-5 below?
> > >> > >
> > >> > > Thank you,
> > >> > > Edmon
> > >> > >
> > >> > > On Wednesday, September 16, 2015, Edmon Begoli <ebegoli@gmail.com
> >
> > >> > wrote:
> > >> > >
> > >> > > > I am studying the JSON Storage plugin.
> > >> > > >
> > >> > > > Can someone please answer what is the meaning (or intent) of the
> > >> > > following
> > >> > > > methods:
> > >> > > >
> > >> > > > # Question 1:
> > >> > > > # This is from inside the getter for RecordWriter. What is
> > >> major/minor
> > >> > > > fragment id?
> > >> > > > String fragmentId = String.format("%d_%d",
> > >> handle.getMajorFragmentId(),
> > >> > > > handle.getMinorFragmentId());
> > >> > > >
> > >> > > >
> > >> > > > # Question 2:
> > >> > > > # Speficially, what is Operator Type, and what is JSON specific
> > about
> > >> > it?
> > >> > > > public int getReaderOperatorType() {
> > >> > > > return CoreOperatorType.JSON_SUB_SCAN_VALUE;
> > >> > > > }
> > >> > > >
> > >> > > > # Question 3:
> > >> > > > # same, but for writer
> > >> > > > @Override
> > >> > > > public int getWriterOperatorType() {
> > >> > > > throw new UnsupportedOperationException();
> > >> > > > }
> > >> > > >
> > >> > > > # Question 4:
> > >> > > > # I generally understand the concept of predicate pushdown, but
> is
> > >> this
> > >> > > > about predicate push down or something else?
> > >> > > > # How does one implement a push down for storage format?
> > >> > > > @Override
> > >> > > > public boolean supportsPushDown() {
> > >> > > >
> > >> > > > # Question 5:
> > >> > > > # Is this int value of 31 purely randomly selected, or is there
> an
> > >> > index
> > >> > > > somewhere?
> > >> > > > @Override
> > >> > > > public int hashCode() {
> > >> > > > return 31;
> > >> > > > }
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > >
> > >> >
> > >> >
> > >> >
> > >> > --
> > >> >
> > >> > Abdelhakim Deneche
> > >> >
> > >> > Software Engineer
> > >> >
> > >> > <http://www.mapr.com/>
> > >> >
> > >> >
> > >> > Now Available - Free Hadoop On-Demand Training
> > >> > <
> > >> >
> > >>
> >
> http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available
> > >> > >
> > >> >
> > >>
> > >
> > >
> > >
> > > --
> > >
> > > Abdelhakim Deneche
> > >
> > > Software Engineer
> > >
> > > <http://www.mapr.com/>
> > >
> > >
> > > Now Available - Free Hadoop On-Demand Training
> > > <
> >
> http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available
> > >
> >
>
Re: The meaning/intent of the following methods - JSON storage format
- part 1
Posted by Ted Dunning <te...@gmail.com>.
Actually, returning a constant from a hashCode function is pretty goofy.
In the String implementation 31 is a *factor*.
On Tue, Sep 22, 2015 at 4:25 PM, Jinfeng Ni <ji...@gmail.com> wrote:
> For question 5, this link [1] probably explains why 31 is used:
>
>
> [1]
> http://stackoverflow.com/questions/299304/why-does-javas-hashcode-in-string-use-31-as-a-multiplier
>
>
> On Tue, Sep 22, 2015 at 10:05 AM, Abdel Hakim Deneche
> <ad...@maprtech.com> wrote:
> > Question 3:
> > I would assume getWriterOperatorType() has a similar meaning to
> > getReaderOperatorType() but it's not used anywhere in Drill. That would
> > explain why JsonFormatPlugin throws an UnsupportedOperationException
> >
> > Question 4:
> > supportsPushDown() is about "projection" push down and not filter push
> down
> >
> > Question 5:
> > I would say, purely random.
> >
> > On Fri, Sep 18, 2015 at 11:48 AM, Edmon Begoli <eb...@gmail.com>
> wrote:
> >
> >> Thanks, Abdel.
> >> I will use your answers to guide my development, but I will also
> contribute
> >> them back as a javadoc.
> >>
> >> On Fri, Sep 18, 2015 at 2:39 PM, Abdel Hakim Deneche <
> >> adeneche@maprtech.com>
> >> wrote:
> >>
> >> > Question 2:
> >> > to my knowledge (what I've found digging through the code). Each
> Operator
> >> > has a unique OperatorType that is used when writing the operator's
> stats
> >> in
> >> > the profile. So in this case, Each FormatPlugin implementation should
> >> have
> >> > a corresponding unique OperatorType added to CoreOperatorType.
> >> >
> >> > I will get back to you after I get more information about the
> remaining
> >> > questions.
> >> >
> >> > Thanks
> >> >
> >> > On Thu, Sep 17, 2015 at 8:14 PM, Edmon Begoli <eb...@gmail.com>
> wrote:
> >> >
> >> > > Hello all,
> >> > >
> >> > > Could some please with answering my questions 2-5 below?
> >> > >
> >> > > Thank you,
> >> > > Edmon
> >> > >
> >> > > On Wednesday, September 16, 2015, Edmon Begoli <eb...@gmail.com>
> >> > wrote:
> >> > >
> >> > > > I am studying the JSON Storage plugin.
> >> > > >
> >> > > > Can someone please answer what is the meaning (or intent) of the
> >> > > following
> >> > > > methods:
> >> > > >
> >> > > > # Question 1:
> >> > > > # This is from inside the getter for RecordWriter. What is
> >> major/minor
> >> > > > fragment id?
> >> > > > String fragmentId = String.format("%d_%d",
> >> handle.getMajorFragmentId(),
> >> > > > handle.getMinorFragmentId());
> >> > > >
> >> > > >
> >> > > > # Question 2:
> >> > > > # Speficially, what is Operator Type, and what is JSON specific
> about
> >> > it?
> >> > > > public int getReaderOperatorType() {
> >> > > > return CoreOperatorType.JSON_SUB_SCAN_VALUE;
> >> > > > }
> >> > > >
> >> > > > # Question 3:
> >> > > > # same, but for writer
> >> > > > @Override
> >> > > > public int getWriterOperatorType() {
> >> > > > throw new UnsupportedOperationException();
> >> > > > }
> >> > > >
> >> > > > # Question 4:
> >> > > > # I generally understand the concept of predicate pushdown, but is
> >> this
> >> > > > about predicate push down or something else?
> >> > > > # How does one implement a push down for storage format?
> >> > > > @Override
> >> > > > public boolean supportsPushDown() {
> >> > > >
> >> > > > # Question 5:
> >> > > > # Is this int value of 31 purely randomly selected, or is there an
> >> > index
> >> > > > somewhere?
> >> > > > @Override
> >> > > > public int hashCode() {
> >> > > > return 31;
> >> > > > }
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > >
> >> >
> >> >
> >> >
> >> > --
> >> >
> >> > Abdelhakim Deneche
> >> >
> >> > Software Engineer
> >> >
> >> > <http://www.mapr.com/>
> >> >
> >> >
> >> > Now Available - Free Hadoop On-Demand Training
> >> > <
> >> >
> >>
> http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available
> >> > >
> >> >
> >>
> >
> >
> >
> > --
> >
> > Abdelhakim Deneche
> >
> > Software Engineer
> >
> > <http://www.mapr.com/>
> >
> >
> > Now Available - Free Hadoop On-Demand Training
> > <
> http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available
> >
>
Re: The meaning/intent of the following methods - JSON storage format
- part 1
Posted by Jinfeng Ni <ji...@gmail.com>.
For question 5, this link [1] probably explains why 31 is used:
[1] http://stackoverflow.com/questions/299304/why-does-javas-hashcode-in-string-use-31-as-a-multiplier
On Tue, Sep 22, 2015 at 10:05 AM, Abdel Hakim Deneche
<ad...@maprtech.com> wrote:
> Question 3:
> I would assume getWriterOperatorType() has a similar meaning to
> getReaderOperatorType() but it's not used anywhere in Drill. That would
> explain why JsonFormatPlugin throws an UnsupportedOperationException
>
> Question 4:
> supportsPushDown() is about "projection" push down and not filter push down
>
> Question 5:
> I would say, purely random.
>
> On Fri, Sep 18, 2015 at 11:48 AM, Edmon Begoli <eb...@gmail.com> wrote:
>
>> Thanks, Abdel.
>> I will use your answers to guide my development, but I will also contribute
>> them back as a javadoc.
>>
>> On Fri, Sep 18, 2015 at 2:39 PM, Abdel Hakim Deneche <
>> adeneche@maprtech.com>
>> wrote:
>>
>> > Question 2:
>> > to my knowledge (what I've found digging through the code). Each Operator
>> > has a unique OperatorType that is used when writing the operator's stats
>> in
>> > the profile. So in this case, Each FormatPlugin implementation should
>> have
>> > a corresponding unique OperatorType added to CoreOperatorType.
>> >
>> > I will get back to you after I get more information about the remaining
>> > questions.
>> >
>> > Thanks
>> >
>> > On Thu, Sep 17, 2015 at 8:14 PM, Edmon Begoli <eb...@gmail.com> wrote:
>> >
>> > > Hello all,
>> > >
>> > > Could some please with answering my questions 2-5 below?
>> > >
>> > > Thank you,
>> > > Edmon
>> > >
>> > > On Wednesday, September 16, 2015, Edmon Begoli <eb...@gmail.com>
>> > wrote:
>> > >
>> > > > I am studying the JSON Storage plugin.
>> > > >
>> > > > Can someone please answer what is the meaning (or intent) of the
>> > > following
>> > > > methods:
>> > > >
>> > > > # Question 1:
>> > > > # This is from inside the getter for RecordWriter. What is
>> major/minor
>> > > > fragment id?
>> > > > String fragmentId = String.format("%d_%d",
>> handle.getMajorFragmentId(),
>> > > > handle.getMinorFragmentId());
>> > > >
>> > > >
>> > > > # Question 2:
>> > > > # Speficially, what is Operator Type, and what is JSON specific about
>> > it?
>> > > > public int getReaderOperatorType() {
>> > > > return CoreOperatorType.JSON_SUB_SCAN_VALUE;
>> > > > }
>> > > >
>> > > > # Question 3:
>> > > > # same, but for writer
>> > > > @Override
>> > > > public int getWriterOperatorType() {
>> > > > throw new UnsupportedOperationException();
>> > > > }
>> > > >
>> > > > # Question 4:
>> > > > # I generally understand the concept of predicate pushdown, but is
>> this
>> > > > about predicate push down or something else?
>> > > > # How does one implement a push down for storage format?
>> > > > @Override
>> > > > public boolean supportsPushDown() {
>> > > >
>> > > > # Question 5:
>> > > > # Is this int value of 31 purely randomly selected, or is there an
>> > index
>> > > > somewhere?
>> > > > @Override
>> > > > public int hashCode() {
>> > > > return 31;
>> > > > }
>> > > >
>> > > >
>> > > >
>> > > >
>> > >
>> >
>> >
>> >
>> > --
>> >
>> > Abdelhakim Deneche
>> >
>> > Software Engineer
>> >
>> > <http://www.mapr.com/>
>> >
>> >
>> > Now Available - Free Hadoop On-Demand Training
>> > <
>> >
>> http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available
>> > >
>> >
>>
>
>
>
> --
>
> Abdelhakim Deneche
>
> Software Engineer
>
> <http://www.mapr.com/>
>
>
> Now Available - Free Hadoop On-Demand Training
> <http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available>
Re: The meaning/intent of the following methods - JSON storage format
- part 1
Posted by Abdel Hakim Deneche <ad...@maprtech.com>.
Question 3:
I would assume getWriterOperatorType() has a similar meaning to
getReaderOperatorType() but it's not used anywhere in Drill. That would
explain why JsonFormatPlugin throws an UnsupportedOperationException
Question 4:
supportsPushDown() is about "projection" push down and not filter push down
Question 5:
I would say, purely random.
On Fri, Sep 18, 2015 at 11:48 AM, Edmon Begoli <eb...@gmail.com> wrote:
> Thanks, Abdel.
> I will use your answers to guide my development, but I will also contribute
> them back as a javadoc.
>
> On Fri, Sep 18, 2015 at 2:39 PM, Abdel Hakim Deneche <
> adeneche@maprtech.com>
> wrote:
>
> > Question 2:
> > to my knowledge (what I've found digging through the code). Each Operator
> > has a unique OperatorType that is used when writing the operator's stats
> in
> > the profile. So in this case, Each FormatPlugin implementation should
> have
> > a corresponding unique OperatorType added to CoreOperatorType.
> >
> > I will get back to you after I get more information about the remaining
> > questions.
> >
> > Thanks
> >
> > On Thu, Sep 17, 2015 at 8:14 PM, Edmon Begoli <eb...@gmail.com> wrote:
> >
> > > Hello all,
> > >
> > > Could some please with answering my questions 2-5 below?
> > >
> > > Thank you,
> > > Edmon
> > >
> > > On Wednesday, September 16, 2015, Edmon Begoli <eb...@gmail.com>
> > wrote:
> > >
> > > > I am studying the JSON Storage plugin.
> > > >
> > > > Can someone please answer what is the meaning (or intent) of the
> > > following
> > > > methods:
> > > >
> > > > # Question 1:
> > > > # This is from inside the getter for RecordWriter. What is
> major/minor
> > > > fragment id?
> > > > String fragmentId = String.format("%d_%d",
> handle.getMajorFragmentId(),
> > > > handle.getMinorFragmentId());
> > > >
> > > >
> > > > # Question 2:
> > > > # Speficially, what is Operator Type, and what is JSON specific about
> > it?
> > > > public int getReaderOperatorType() {
> > > > return CoreOperatorType.JSON_SUB_SCAN_VALUE;
> > > > }
> > > >
> > > > # Question 3:
> > > > # same, but for writer
> > > > @Override
> > > > public int getWriterOperatorType() {
> > > > throw new UnsupportedOperationException();
> > > > }
> > > >
> > > > # Question 4:
> > > > # I generally understand the concept of predicate pushdown, but is
> this
> > > > about predicate push down or something else?
> > > > # How does one implement a push down for storage format?
> > > > @Override
> > > > public boolean supportsPushDown() {
> > > >
> > > > # Question 5:
> > > > # Is this int value of 31 purely randomly selected, or is there an
> > index
> > > > somewhere?
> > > > @Override
> > > > public int hashCode() {
> > > > return 31;
> > > > }
> > > >
> > > >
> > > >
> > > >
> > >
> >
> >
> >
> > --
> >
> > Abdelhakim Deneche
> >
> > Software Engineer
> >
> > <http://www.mapr.com/>
> >
> >
> > Now Available - Free Hadoop On-Demand Training
> > <
> >
> http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available
> > >
> >
>
--
Abdelhakim Deneche
Software Engineer
<http://www.mapr.com/>
Now Available - Free Hadoop On-Demand Training
<http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available>
Re: The meaning/intent of the following methods - JSON storage format
- part 1
Posted by Edmon Begoli <eb...@gmail.com>.
Thanks, Abdel.
I will use your answers to guide my development, but I will also contribute
them back as a javadoc.
On Fri, Sep 18, 2015 at 2:39 PM, Abdel Hakim Deneche <ad...@maprtech.com>
wrote:
> Question 2:
> to my knowledge (what I've found digging through the code). Each Operator
> has a unique OperatorType that is used when writing the operator's stats in
> the profile. So in this case, Each FormatPlugin implementation should have
> a corresponding unique OperatorType added to CoreOperatorType.
>
> I will get back to you after I get more information about the remaining
> questions.
>
> Thanks
>
> On Thu, Sep 17, 2015 at 8:14 PM, Edmon Begoli <eb...@gmail.com> wrote:
>
> > Hello all,
> >
> > Could some please with answering my questions 2-5 below?
> >
> > Thank you,
> > Edmon
> >
> > On Wednesday, September 16, 2015, Edmon Begoli <eb...@gmail.com>
> wrote:
> >
> > > I am studying the JSON Storage plugin.
> > >
> > > Can someone please answer what is the meaning (or intent) of the
> > following
> > > methods:
> > >
> > > # Question 1:
> > > # This is from inside the getter for RecordWriter. What is major/minor
> > > fragment id?
> > > String fragmentId = String.format("%d_%d", handle.getMajorFragmentId(),
> > > handle.getMinorFragmentId());
> > >
> > >
> > > # Question 2:
> > > # Speficially, what is Operator Type, and what is JSON specific about
> it?
> > > public int getReaderOperatorType() {
> > > return CoreOperatorType.JSON_SUB_SCAN_VALUE;
> > > }
> > >
> > > # Question 3:
> > > # same, but for writer
> > > @Override
> > > public int getWriterOperatorType() {
> > > throw new UnsupportedOperationException();
> > > }
> > >
> > > # Question 4:
> > > # I generally understand the concept of predicate pushdown, but is this
> > > about predicate push down or something else?
> > > # How does one implement a push down for storage format?
> > > @Override
> > > public boolean supportsPushDown() {
> > >
> > > # Question 5:
> > > # Is this int value of 31 purely randomly selected, or is there an
> index
> > > somewhere?
> > > @Override
> > > public int hashCode() {
> > > return 31;
> > > }
> > >
> > >
> > >
> > >
> >
>
>
>
> --
>
> Abdelhakim Deneche
>
> Software Engineer
>
> <http://www.mapr.com/>
>
>
> Now Available - Free Hadoop On-Demand Training
> <
> http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available
> >
>
Re: The meaning/intent of the following methods - JSON storage format
- part 1
Posted by Abdel Hakim Deneche <ad...@maprtech.com>.
Question 2:
to my knowledge (what I've found digging through the code). Each Operator
has a unique OperatorType that is used when writing the operator's stats in
the profile. So in this case, Each FormatPlugin implementation should have
a corresponding unique OperatorType added to CoreOperatorType.
I will get back to you after I get more information about the remaining
questions.
Thanks
On Thu, Sep 17, 2015 at 8:14 PM, Edmon Begoli <eb...@gmail.com> wrote:
> Hello all,
>
> Could some please with answering my questions 2-5 below?
>
> Thank you,
> Edmon
>
> On Wednesday, September 16, 2015, Edmon Begoli <eb...@gmail.com> wrote:
>
> > I am studying the JSON Storage plugin.
> >
> > Can someone please answer what is the meaning (or intent) of the
> following
> > methods:
> >
> > # Question 1:
> > # This is from inside the getter for RecordWriter. What is major/minor
> > fragment id?
> > String fragmentId = String.format("%d_%d", handle.getMajorFragmentId(),
> > handle.getMinorFragmentId());
> >
> >
> > # Question 2:
> > # Speficially, what is Operator Type, and what is JSON specific about it?
> > public int getReaderOperatorType() {
> > return CoreOperatorType.JSON_SUB_SCAN_VALUE;
> > }
> >
> > # Question 3:
> > # same, but for writer
> > @Override
> > public int getWriterOperatorType() {
> > throw new UnsupportedOperationException();
> > }
> >
> > # Question 4:
> > # I generally understand the concept of predicate pushdown, but is this
> > about predicate push down or something else?
> > # How does one implement a push down for storage format?
> > @Override
> > public boolean supportsPushDown() {
> >
> > # Question 5:
> > # Is this int value of 31 purely randomly selected, or is there an index
> > somewhere?
> > @Override
> > public int hashCode() {
> > return 31;
> > }
> >
> >
> >
> >
>
--
Abdelhakim Deneche
Software Engineer
<http://www.mapr.com/>
Now Available - Free Hadoop On-Demand Training
<http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available>
Re: The meaning/intent of the following methods - JSON storage format
- part 1
Posted by Edmon Begoli <eb...@gmail.com>.
Hello all,
Could some please with answering my questions 2-5 below?
Thank you,
Edmon
On Wednesday, September 16, 2015, Edmon Begoli <eb...@gmail.com> wrote:
> I am studying the JSON Storage plugin.
>
> Can someone please answer what is the meaning (or intent) of the following
> methods:
>
> # Question 1:
> # This is from inside the getter for RecordWriter. What is major/minor
> fragment id?
> String fragmentId = String.format("%d_%d", handle.getMajorFragmentId(),
> handle.getMinorFragmentId());
>
>
> # Question 2:
> # Speficially, what is Operator Type, and what is JSON specific about it?
> public int getReaderOperatorType() {
> return CoreOperatorType.JSON_SUB_SCAN_VALUE;
> }
>
> # Question 3:
> # same, but for writer
> @Override
> public int getWriterOperatorType() {
> throw new UnsupportedOperationException();
> }
>
> # Question 4:
> # I generally understand the concept of predicate pushdown, but is this
> about predicate push down or something else?
> # How does one implement a push down for storage format?
> @Override
> public boolean supportsPushDown() {
>
> # Question 5:
> # Is this int value of 31 purely randomly selected, or is there an index
> somewhere?
> @Override
> public int hashCode() {
> return 31;
> }
>
>
>
>