AccumuloInputFormat getters

[Josh Elser <jo...@gmail.com>]

Why are all of the getters on the AccumuloInputFormat protected (really, 
InputFormatBase) instead of public?

This has repeatedly infuriated me as it makes it impossible for me to 
verify that the Configuration actually has the data in it as needed.

It seems intentional so I figured I would ask before making a ticket and 
changing it.

- Josh

Re: AccumuloInputFormat getters

[Josh Elser <jo...@gmail.com>]

(sorry for the bifurcation of the thread -- had another thought I wanted 
to write)

Yeah, I can respect the difficulty in a user being unable to get back 
the same object that was serialized. At a glance, I would think that the 
getters would best mirror the "current" way that we recommend the API is 
interacted with (as in, not a deprecated method). I don't know if that's 
possible across all of the methods, but for some of these, I think there 
is much worth.

For context, this limitation has bitten me working in both Hive and Pig. 
I don't have control over the Configuration object that I'm handed from 
the framework, and I have no way to determine whether or not the proper 
configuration still exists within the current Configuration. I want to 
be able to see if the proper Accumulo configuration items are still 
present in the Configuration (to debug an issue), but I have no way to 
do it easily.

Subclassing AIF/AOF seems unnecessarily complex to me. I'll see if I can 
come up with something simple to add.

On 7/16/14, 9:59 PM, Christopher wrote:
> Well, you can subclass to introspect. And, if you feel the API can be
> improved by offering stronger getter/setter support with the stability
> guarantees that we care about for public API, go ahead. (It probably
> wouldn't change much anyway, since we now treat protected as public API,
> too, I think). I won't object to the improvements... just explaining why
> it's like that. My concern if you were to do this would be whether this
> would actually add too much bloat or not to consumers of the API who don't
> need to subclass, and the lack of 1-to-1 in many cases... but if you can
> address those things sufficiently, I wouldn't object.
>
>
> --
> Christopher L Tubbs II
> http://gravatar.com/ctubbsii
>
>
> On Wed, Jul 16, 2014 at 9:55 PM, Josh Elser <jo...@gmail.com> wrote:
>
>> Ultimately, I feel like there's a big problem when I, an "experienced
>> Accumulo developer", am getting frustrated with the API.
>>
>> As it stands right now, I have no way to introspect the contents of a
>> Configuration to ensure that the state is as I expect it to be. I'm stuck
>> dumping the entire configuration, and grep'ing it to see if the values I
>> expect are in there with *some* key. If so, I then have to try to unravel
>> what exactly is the appropriate key that the value should be paired with.
>>
>> I can understand the complexity in the storage of relevant data within the
>> Configuration, but this seems unnecessarily complicated to me.
>>
>>
>> On 7/16/14, 9:48 PM, Josh Elser wrote:
>>
>>> The value of the name of the table that the AccumuloInputFormat is going
>>> to read is subject to change? Isn't the point of a getter that it can
>>> unwrap the specifics of the serialization within the configuration and
>>> present the high-level constructs (username, AuthenticationToken, table
>>> name, IteratorSettings, etc) that users expect?
>>>
>>> On 7/16/14, 9:46 PM, Christopher wrote:
>>>
>>>> Because those things represent internals of the configuration that are
>>>> subject to change, and we don't want end users becoming dependent on
>>>> them.
>>>> They are protected, because they may be needed for subclassing, where the
>>>> subclass assumes some greater risk than an end user of the API.
>>>>
>>>>
>>>> --
>>>> Christopher L Tubbs II
>>>> http://gravatar.com/ctubbsii
>>>>
>>>>
>>>> On Wed, Jul 16, 2014 at 9:43 PM, Josh Elser <jo...@gmail.com>
>>>> wrote:
>>>>
>>>>   Why are all of the getters on the AccumuloInputFormat protected (really,
>>>>> InputFormatBase) instead of public?
>>>>>
>>>>> This has repeatedly infuriated me as it makes it impossible for me to
>>>>> verify that the Configuration actually has the data in it as needed.
>>>>>
>>>>> It seems intentional so I figured I would ask before making a ticket and
>>>>> changing it.
>>>>>
>>>>> - Josh
>>>>>
>>>>>
>>>>
>

Re: AccumuloInputFormat getters

[Sean Busbey <bu...@cloudera.com>]

If we're going to make an interface, we should get some of the new client
API together and try to make the configuration interface the same.

-- 
Sean
That could work. It also lines up with the idea of having "one expected
return class" for certain methods.

Maybe have some pojo or interface to keep us honest would make the API
aspect easier to manage?

On 7/16/14, 10:09 PM, William Slacum wrote:

> It's dubious to say it's "internals", when it gets hamjammed into a map of
> strings to other strings that's going to be passed around to many
> processes.
>
> Maybe we can make our own serializable pojo that implements some interface
> for consumers to use. That would at least let us hide internals and have a
> single entry point into the Hadoop configuration.
>
>
> On Wed, Jul 16, 2014 at 9:59 PM, Christopher <ct...@apache.org> wrote:
>
>  Well, you can subclass to introspect. And, if you feel the API can be
>> improved by offering stronger getter/setter support with the stability
>> guarantees that we care about for public API, go ahead. (It probably
>> wouldn't change much anyway, since we now treat protected as public API,
>> too, I think). I won't object to the improvements... just explaining why
>> it's like that. My concern if you were to do this would be whether this
>> would actually add too much bloat or not to consumers of the API who don't
>> need to subclass, and the lack of 1-to-1 in many cases... but if you can
>> address those things sufficiently, I wouldn't object.
>>
>>
>> --
>> Christopher L Tubbs II
>> http://gravatar.com/ctubbsii
>>
>>
>> On Wed, Jul 16, 2014 at 9:55 PM, Josh Elser <jo...@gmail.com> wrote:
>>
>>  Ultimately, I feel like there's a big problem when I, an "experienced
>>> Accumulo developer", am getting frustrated with the API.
>>>
>>> As it stands right now, I have no way to introspect the contents of a
>>> Configuration to ensure that the state is as I expect it to be. I'm stuck
>>> dumping the entire configuration, and grep'ing it to see if the values I
>>> expect are in there with *some* key. If so, I then have to try to unravel
>>> what exactly is the appropriate key that the value should be paired with.
>>>
>>> I can understand the complexity in the storage of relevant data within
>>>
>> the
>>
>>> Configuration, but this seems unnecessarily complicated to me.
>>>
>>>
>>> On 7/16/14, 9:48 PM, Josh Elser wrote:
>>>
>>>  The value of the name of the table that the AccumuloInputFormat is going
>>>> to read is subject to change? Isn't the point of a getter that it can
>>>> unwrap the specifics of the serialization within the configuration and
>>>> present the high-level constructs (username, AuthenticationToken, table
>>>> name, IteratorSettings, etc) that users expect?
>>>>
>>>> On 7/16/14, 9:46 PM, Christopher wrote:
>>>>
>>>>  Because those things represent internals of the configuration that are
>>>>> subject to change, and we don't want end users becoming dependent on
>>>>> them.
>>>>> They are protected, because they may be needed for subclassing, where
>>>>>
>>>> the
>>
>>> subclass assumes some greater risk than an end user of the API.
>>>>>
>>>>>
>>>>> --
>>>>> Christopher L Tubbs II
>>>>> http://gravatar.com/ctubbsii
>>>>>
>>>>>
>>>>> On Wed, Jul 16, 2014 at 9:43 PM, Josh Elser <jo...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>   Why are all of the getters on the AccumuloInputFormat protected
>>>>>
>>>> (really,
>>
>>> InputFormatBase) instead of public?
>>>>>>
>>>>>> This has repeatedly infuriated me as it makes it impossible for me to
>>>>>> verify that the Configuration actually has the data in it as needed.
>>>>>>
>>>>>> It seems intentional so I figured I would ask before making a ticket
>>>>>>
>>>>> and
>>
>>> changing it.
>>>>>>
>>>>>> - Josh
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>
>

Re: AccumuloInputFormat getters

[Josh Elser <jo...@gmail.com>]

That could work. It also lines up with the idea of having "one expected 
return class" for certain methods.

Maybe have some pojo or interface to keep us honest would make the API 
aspect easier to manage?

On 7/16/14, 10:09 PM, William Slacum wrote:
> It's dubious to say it's "internals", when it gets hamjammed into a map of
> strings to other strings that's going to be passed around to many processes.
>
> Maybe we can make our own serializable pojo that implements some interface
> for consumers to use. That would at least let us hide internals and have a
> single entry point into the Hadoop configuration.
>
>
> On Wed, Jul 16, 2014 at 9:59 PM, Christopher <ct...@apache.org> wrote:
>
>> Well, you can subclass to introspect. And, if you feel the API can be
>> improved by offering stronger getter/setter support with the stability
>> guarantees that we care about for public API, go ahead. (It probably
>> wouldn't change much anyway, since we now treat protected as public API,
>> too, I think). I won't object to the improvements... just explaining why
>> it's like that. My concern if you were to do this would be whether this
>> would actually add too much bloat or not to consumers of the API who don't
>> need to subclass, and the lack of 1-to-1 in many cases... but if you can
>> address those things sufficiently, I wouldn't object.
>>
>>
>> --
>> Christopher L Tubbs II
>> http://gravatar.com/ctubbsii
>>
>>
>> On Wed, Jul 16, 2014 at 9:55 PM, Josh Elser <jo...@gmail.com> wrote:
>>
>>> Ultimately, I feel like there's a big problem when I, an "experienced
>>> Accumulo developer", am getting frustrated with the API.
>>>
>>> As it stands right now, I have no way to introspect the contents of a
>>> Configuration to ensure that the state is as I expect it to be. I'm stuck
>>> dumping the entire configuration, and grep'ing it to see if the values I
>>> expect are in there with *some* key. If so, I then have to try to unravel
>>> what exactly is the appropriate key that the value should be paired with.
>>>
>>> I can understand the complexity in the storage of relevant data within
>> the
>>> Configuration, but this seems unnecessarily complicated to me.
>>>
>>>
>>> On 7/16/14, 9:48 PM, Josh Elser wrote:
>>>
>>>> The value of the name of the table that the AccumuloInputFormat is going
>>>> to read is subject to change? Isn't the point of a getter that it can
>>>> unwrap the specifics of the serialization within the configuration and
>>>> present the high-level constructs (username, AuthenticationToken, table
>>>> name, IteratorSettings, etc) that users expect?
>>>>
>>>> On 7/16/14, 9:46 PM, Christopher wrote:
>>>>
>>>>> Because those things represent internals of the configuration that are
>>>>> subject to change, and we don't want end users becoming dependent on
>>>>> them.
>>>>> They are protected, because they may be needed for subclassing, where
>> the
>>>>> subclass assumes some greater risk than an end user of the API.
>>>>>
>>>>>
>>>>> --
>>>>> Christopher L Tubbs II
>>>>> http://gravatar.com/ctubbsii
>>>>>
>>>>>
>>>>> On Wed, Jul 16, 2014 at 9:43 PM, Josh Elser <jo...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>   Why are all of the getters on the AccumuloInputFormat protected
>> (really,
>>>>>> InputFormatBase) instead of public?
>>>>>>
>>>>>> This has repeatedly infuriated me as it makes it impossible for me to
>>>>>> verify that the Configuration actually has the data in it as needed.
>>>>>>
>>>>>> It seems intentional so I figured I would ask before making a ticket
>> and
>>>>>> changing it.
>>>>>>
>>>>>> - Josh
>>>>>>
>>>>>>
>>>>>
>>
>

Re: AccumuloInputFormat getters

[William Slacum <wi...@accumulo.net>]

It's dubious to say it's "internals", when it gets hamjammed into a map of
strings to other strings that's going to be passed around to many processes.

Maybe we can make our own serializable pojo that implements some interface
for consumers to use. That would at least let us hide internals and have a
single entry point into the Hadoop configuration.


On Wed, Jul 16, 2014 at 9:59 PM, Christopher <ct...@apache.org> wrote:

> Well, you can subclass to introspect. And, if you feel the API can be
> improved by offering stronger getter/setter support with the stability
> guarantees that we care about for public API, go ahead. (It probably
> wouldn't change much anyway, since we now treat protected as public API,
> too, I think). I won't object to the improvements... just explaining why
> it's like that. My concern if you were to do this would be whether this
> would actually add too much bloat or not to consumers of the API who don't
> need to subclass, and the lack of 1-to-1 in many cases... but if you can
> address those things sufficiently, I wouldn't object.
>
>
> --
> Christopher L Tubbs II
> http://gravatar.com/ctubbsii
>
>
> On Wed, Jul 16, 2014 at 9:55 PM, Josh Elser <jo...@gmail.com> wrote:
>
> > Ultimately, I feel like there's a big problem when I, an "experienced
> > Accumulo developer", am getting frustrated with the API.
> >
> > As it stands right now, I have no way to introspect the contents of a
> > Configuration to ensure that the state is as I expect it to be. I'm stuck
> > dumping the entire configuration, and grep'ing it to see if the values I
> > expect are in there with *some* key. If so, I then have to try to unravel
> > what exactly is the appropriate key that the value should be paired with.
> >
> > I can understand the complexity in the storage of relevant data within
> the
> > Configuration, but this seems unnecessarily complicated to me.
> >
> >
> > On 7/16/14, 9:48 PM, Josh Elser wrote:
> >
> >> The value of the name of the table that the AccumuloInputFormat is going
> >> to read is subject to change? Isn't the point of a getter that it can
> >> unwrap the specifics of the serialization within the configuration and
> >> present the high-level constructs (username, AuthenticationToken, table
> >> name, IteratorSettings, etc) that users expect?
> >>
> >> On 7/16/14, 9:46 PM, Christopher wrote:
> >>
> >>> Because those things represent internals of the configuration that are
> >>> subject to change, and we don't want end users becoming dependent on
> >>> them.
> >>> They are protected, because they may be needed for subclassing, where
> the
> >>> subclass assumes some greater risk than an end user of the API.
> >>>
> >>>
> >>> --
> >>> Christopher L Tubbs II
> >>> http://gravatar.com/ctubbsii
> >>>
> >>>
> >>> On Wed, Jul 16, 2014 at 9:43 PM, Josh Elser <jo...@gmail.com>
> >>> wrote:
> >>>
> >>>  Why are all of the getters on the AccumuloInputFormat protected
> (really,
> >>>> InputFormatBase) instead of public?
> >>>>
> >>>> This has repeatedly infuriated me as it makes it impossible for me to
> >>>> verify that the Configuration actually has the data in it as needed.
> >>>>
> >>>> It seems intentional so I figured I would ask before making a ticket
> and
> >>>> changing it.
> >>>>
> >>>> - Josh
> >>>>
> >>>>
> >>>
>

Re: AccumuloInputFormat getters

[Christopher <ct...@apache.org>]

Well, you can subclass to introspect. And, if you feel the API can be
improved by offering stronger getter/setter support with the stability
guarantees that we care about for public API, go ahead. (It probably
wouldn't change much anyway, since we now treat protected as public API,
too, I think). I won't object to the improvements... just explaining why
it's like that. My concern if you were to do this would be whether this
would actually add too much bloat or not to consumers of the API who don't
need to subclass, and the lack of 1-to-1 in many cases... but if you can
address those things sufficiently, I wouldn't object.


--
Christopher L Tubbs II
http://gravatar.com/ctubbsii


On Wed, Jul 16, 2014 at 9:55 PM, Josh Elser <jo...@gmail.com> wrote:

> Ultimately, I feel like there's a big problem when I, an "experienced
> Accumulo developer", am getting frustrated with the API.
>
> As it stands right now, I have no way to introspect the contents of a
> Configuration to ensure that the state is as I expect it to be. I'm stuck
> dumping the entire configuration, and grep'ing it to see if the values I
> expect are in there with *some* key. If so, I then have to try to unravel
> what exactly is the appropriate key that the value should be paired with.
>
> I can understand the complexity in the storage of relevant data within the
> Configuration, but this seems unnecessarily complicated to me.
>
>
> On 7/16/14, 9:48 PM, Josh Elser wrote:
>
>> The value of the name of the table that the AccumuloInputFormat is going
>> to read is subject to change? Isn't the point of a getter that it can
>> unwrap the specifics of the serialization within the configuration and
>> present the high-level constructs (username, AuthenticationToken, table
>> name, IteratorSettings, etc) that users expect?
>>
>> On 7/16/14, 9:46 PM, Christopher wrote:
>>
>>> Because those things represent internals of the configuration that are
>>> subject to change, and we don't want end users becoming dependent on
>>> them.
>>> They are protected, because they may be needed for subclassing, where the
>>> subclass assumes some greater risk than an end user of the API.
>>>
>>>
>>> --
>>> Christopher L Tubbs II
>>> http://gravatar.com/ctubbsii
>>>
>>>
>>> On Wed, Jul 16, 2014 at 9:43 PM, Josh Elser <jo...@gmail.com>
>>> wrote:
>>>
>>>  Why are all of the getters on the AccumuloInputFormat protected (really,
>>>> InputFormatBase) instead of public?
>>>>
>>>> This has repeatedly infuriated me as it makes it impossible for me to
>>>> verify that the Configuration actually has the data in it as needed.
>>>>
>>>> It seems intentional so I figured I would ask before making a ticket and
>>>> changing it.
>>>>
>>>> - Josh
>>>>
>>>>
>>>

Re: AccumuloInputFormat getters

[Josh Elser <jo...@gmail.com>]

Ultimately, I feel like there's a big problem when I, an "experienced 
Accumulo developer", am getting frustrated with the API.

As it stands right now, I have no way to introspect the contents of a 
Configuration to ensure that the state is as I expect it to be. I'm 
stuck dumping the entire configuration, and grep'ing it to see if the 
values I expect are in there with *some* key. If so, I then have to try 
to unravel what exactly is the appropriate key that the value should be 
paired with.

I can understand the complexity in the storage of relevant data within 
the Configuration, but this seems unnecessarily complicated to me.

On 7/16/14, 9:48 PM, Josh Elser wrote:
> The value of the name of the table that the AccumuloInputFormat is going
> to read is subject to change? Isn't the point of a getter that it can
> unwrap the specifics of the serialization within the configuration and
> present the high-level constructs (username, AuthenticationToken, table
> name, IteratorSettings, etc) that users expect?
>
> On 7/16/14, 9:46 PM, Christopher wrote:
>> Because those things represent internals of the configuration that are
>> subject to change, and we don't want end users becoming dependent on
>> them.
>> They are protected, because they may be needed for subclassing, where the
>> subclass assumes some greater risk than an end user of the API.
>>
>>
>> --
>> Christopher L Tubbs II
>> http://gravatar.com/ctubbsii
>>
>>
>> On Wed, Jul 16, 2014 at 9:43 PM, Josh Elser <jo...@gmail.com> wrote:
>>
>>> Why are all of the getters on the AccumuloInputFormat protected (really,
>>> InputFormatBase) instead of public?
>>>
>>> This has repeatedly infuriated me as it makes it impossible for me to
>>> verify that the Configuration actually has the data in it as needed.
>>>
>>> It seems intentional so I figured I would ask before making a ticket and
>>> changing it.
>>>
>>> - Josh
>>>
>>

Re: AccumuloInputFormat getters

[Christopher <ct...@apache.org>]

The problem is that our setters/getters were never 1-to-1. Some setters set
a group of related config, but the getters need to read them individually,
to perform different functions. For instance, setting the connector
information... you'd pass username and password information, but the code
inside (or a subclass) may need just the username, or the instanceName, or
(in the past) the default table name, etc. Or, for example, when we've
deprecated some setters in favor of others, the corresponding getters no
longer made sense. For instance, when we added the ability to specify a
password file, the entire concept of getting a password back wouldn't make
sense... you never gave a password in the first place. It's now no longer
just deserialization, but you'd have to add new functionality to read the
password file and return a password, that has nothing to do with needs
internal to the class... but would just be providing a complete
setter/getter API for the sake of doing it.

I'm not going to argue that a particular field is or isn't going to
change... I'm just explaining the history of why it's like that, and the
kinds of changes that have occurred in the past. Personally, I don't think
people should be depending on our serialization. They launched the job,
they should know what they put in it. If they need to pass state to
different components of their code, they shouldn't depend on our
serialization to carry that state, even if it seems like there's some
overlap.


--
Christopher L Tubbs II
http://gravatar.com/ctubbsii


On Wed, Jul 16, 2014 at 9:48 PM, Josh Elser <jo...@gmail.com> wrote:

> The value of the name of the table that the AccumuloInputFormat is going
> to read is subject to change? Isn't the point of a getter that it can
> unwrap the specifics of the serialization within the configuration and
> present the high-level constructs (username, AuthenticationToken, table
> name, IteratorSettings, etc) that users expect?
>
>
> On 7/16/14, 9:46 PM, Christopher wrote:
>
>> Because those things represent internals of the configuration that are
>> subject to change, and we don't want end users becoming dependent on them.
>> They are protected, because they may be needed for subclassing, where the
>> subclass assumes some greater risk than an end user of the API.
>>
>>
>> --
>> Christopher L Tubbs II
>> http://gravatar.com/ctubbsii
>>
>>
>> On Wed, Jul 16, 2014 at 9:43 PM, Josh Elser <jo...@gmail.com> wrote:
>>
>>  Why are all of the getters on the AccumuloInputFormat protected (really,
>>> InputFormatBase) instead of public?
>>>
>>> This has repeatedly infuriated me as it makes it impossible for me to
>>> verify that the Configuration actually has the data in it as needed.
>>>
>>> It seems intentional so I figured I would ask before making a ticket and
>>> changing it.
>>>
>>> - Josh
>>>
>>>
>>

Re: AccumuloInputFormat getters

[Josh Elser <jo...@gmail.com>]

The value of the name of the table that the AccumuloInputFormat is going 
to read is subject to change? Isn't the point of a getter that it can 
unwrap the specifics of the serialization within the configuration and 
present the high-level constructs (username, AuthenticationToken, table 
name, IteratorSettings, etc) that users expect?

On 7/16/14, 9:46 PM, Christopher wrote:
> Because those things represent internals of the configuration that are
> subject to change, and we don't want end users becoming dependent on them.
> They are protected, because they may be needed for subclassing, where the
> subclass assumes some greater risk than an end user of the API.
>
>
> --
> Christopher L Tubbs II
> http://gravatar.com/ctubbsii
>
>
> On Wed, Jul 16, 2014 at 9:43 PM, Josh Elser <jo...@gmail.com> wrote:
>
>> Why are all of the getters on the AccumuloInputFormat protected (really,
>> InputFormatBase) instead of public?
>>
>> This has repeatedly infuriated me as it makes it impossible for me to
>> verify that the Configuration actually has the data in it as needed.
>>
>> It seems intentional so I figured I would ask before making a ticket and
>> changing it.
>>
>> - Josh
>>
>

Re: AccumuloInputFormat getters

[Christopher <ct...@apache.org>]

Because those things represent internals of the configuration that are
subject to change, and we don't want end users becoming dependent on them.
They are protected, because they may be needed for subclassing, where the
subclass assumes some greater risk than an end user of the API.


--
Christopher L Tubbs II
http://gravatar.com/ctubbsii


On Wed, Jul 16, 2014 at 9:43 PM, Josh Elser <jo...@gmail.com> wrote:

> Why are all of the getters on the AccumuloInputFormat protected (really,
> InputFormatBase) instead of public?
>
> This has repeatedly infuriated me as it makes it impossible for me to
> verify that the Configuration actually has the data in it as needed.
>
> It seems intentional so I figured I would ask before making a ticket and
> changing it.
>
> - Josh
>