About Properties

[Ferdy Galema <fe...@kalooga.com>]

Hi,

While Lewis and I were discussing over NUTCH-1205, we identified
the Properties object as the major source of trouble/confusion when
configuring datastores. First and foremost, it's makes no sense that we
have 2 ways to configure a store, namely via Configuration and Properties.
Besides this there seems to be some trouble with the serialization and
initialization of the stores. (Sometimes runtime Properties settings are
not correctly used). We have a few ways to solve this problem:

-Stop supporting the adding of dynamic properties (runtime settings) and
only support the static gora.properties file. People wanting to use runtime
props shall use Configuration somehow.
-Completely remove the Properties object from Gora altogether. Migrate
existing properties to Configuration.
-Trying to make both Properties and Configuration work (that's what the
current direction seems to be). Difficult it seems.
-Something else?

I think the second option is best. The advantage of Configuration is that
it inheritently works with mapreduce because it is automatically
(de)serialized and available in the mappers and reducers. What do you think?

Ferdy.

Re: About Properties

[Renato Marroquín Mogrovejo <re...@gmail.com>]

I think most hadoop related projects do prefer configuration, so I
think it's a good idea that Gora does this as well.
Pig prefers configuration as well, but I think studying on how to make
Gora more compatible with other projects should be discussed on
separate threads. Just my 2 cents (:

Renato M.

2012/5/2 Lewis John Mcgibbney <le...@gmail.com>:
> Hello Everyone,
>
> Just to add to this thread, it became plain obvious to me that
> improvements/simplifications could be made here when I ran into
> problems when trying to address the issue with gora-cassandra tests.
> HBase store seems to use Configuration and hbase-site.xml, Cassandra
> store needs to use properties and cassandra.yaml, Sql store needs to
> use properties etc.
>
> I see Ferdy's suggestion to deprecate the properties object moving
> forward, instead making the Gora API prefer Configuration. Although my
> knowledge regarding Pig etc is limited at this stage, I can only
> assume that this also ties in with our future vision to improve the
> analysis aspect of the Gora API.
>
> Thanks
>
> On Tue, May 1, 2012 at 12:16 PM, Ferdy Galema <fe...@kalooga.com> wrote:
>> Hi,
>>
>> While Lewis and I were discussing over NUTCH-1205, we identified
>> the Properties object as the major source of trouble/confusion when
>> configuring datastores. First and foremost, it's makes no sense that we
>> have 2 ways to configure a store, namely via Configuration and Properties.
>> Besides this there seems to be some trouble with the serialization and
>> initialization of the stores. (Sometimes runtime Properties settings are
>> not correctly used). We have a few ways to solve this problem:
>>
>> -Stop supporting the adding of dynamic properties (runtime settings) and
>> only support the static gora.properties file. People wanting to use runtime
>> props shall use Configuration somehow.
>> -Completely remove the Properties object from Gora altogether. Migrate
>> existing properties to Configuration.
>> -Trying to make both Properties and Configuration work (that's what the
>> current direction seems to be). Difficult it seems.
>> -Something else?
>>
>> I think the second option is best. The advantage of Configuration is that
>> it inheritently works with mapreduce because it is automatically
>> (de)serialized and available in the mappers and reducers. What do you think?
>>
>> Ferdy.
>
>
>
> --
> Lewis

Re: About Properties

[Lewis John Mcgibbney <le...@gmail.com>]

Hello Everyone,

Just to add to this thread, it became plain obvious to me that
improvements/simplifications could be made here when I ran into
problems when trying to address the issue with gora-cassandra tests.
HBase store seems to use Configuration and hbase-site.xml, Cassandra
store needs to use properties and cassandra.yaml, Sql store needs to
use properties etc.

I see Ferdy's suggestion to deprecate the properties object moving
forward, instead making the Gora API prefer Configuration. Although my
knowledge regarding Pig etc is limited at this stage, I can only
assume that this also ties in with our future vision to improve the
analysis aspect of the Gora API.

Thanks

On Tue, May 1, 2012 at 12:16 PM, Ferdy Galema <fe...@kalooga.com> wrote:
> Hi,
>
> While Lewis and I were discussing over NUTCH-1205, we identified
> the Properties object as the major source of trouble/confusion when
> configuring datastores. First and foremost, it's makes no sense that we
> have 2 ways to configure a store, namely via Configuration and Properties.
> Besides this there seems to be some trouble with the serialization and
> initialization of the stores. (Sometimes runtime Properties settings are
> not correctly used). We have a few ways to solve this problem:
>
> -Stop supporting the adding of dynamic properties (runtime settings) and
> only support the static gora.properties file. People wanting to use runtime
> props shall use Configuration somehow.
> -Completely remove the Properties object from Gora altogether. Migrate
> existing properties to Configuration.
> -Trying to make both Properties and Configuration work (that's what the
> current direction seems to be). Difficult it seems.
> -Something else?
>
> I think the second option is best. The advantage of Configuration is that
> it inheritently works with mapreduce because it is automatically
> (de)serialized and available in the mappers and reducers. What do you think?
>
> Ferdy.



-- 
Lewis