You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@gora.apache.org by Lewis John Mcgibbney <le...@gmail.com> on 2011/11/02 20:20:05 UTC
Re: Setting properties in gora.properties
Hi,
(I'm keeping gora-dev's in on this one as well)
A quick update to this one, having submitted a work in progress patch for
NUTCH-1189, so that properties are easier for the gora-cassandra module to
find I'm trying to get the configuration to work as follows
# Cassandra store properties
# gora.CassandraMapping.keyspace=
# gora.CassandraMapping.name=
# gora.CassandraMapping.class=
# gora.CassandraMapping.qualifier=
# gora.CassandraMapping.family=
# gora.CassandraMapping.type=
# gora.CassandraMapping.cluster=
gora.CassandraMapping.host=localhost://127.0.0.1:9160
However I'm still getting
Caused by: java.io.IOException: Property with base name "servers" could not
be found, make sure to include this property in gora.properties file
This is caused by a null value being returned for baseKey parameter in
DataStoreFactory.
I think it's important to note, that in order to get Nutchgora to build
with the gora-cassandra properties uncommented in NUTCHGORA/ivy/ivy.xml it
was necessary to include the following exclusions
<dependency org="org.apache.gora" name="gora-cassandra"
rev="0.1.1-incubating" conf="*->default">
<exclude org="org.apache.thrift" />
<exclude org="org.apache.cassandra" />
</dependency>
Firstly, does anyone have a solution regarding the removing or substituting
the exclusions? Enis has mentioned that they should be included within
gora-cassandra/ivy/ivy.xml, however as this has not been loaded to maven
repos it is not currently a possibility therefore they need to be included
in the Nutchgora side of things.
Secondly, does anyone know how the illusive baseKey parameter should be
included within the gora.properties?
Presumably something similar to
# gora.CassandraMapping.servers=somethingSomething.com
This is nearly sorted (I hope) but there are a couple of areas that need
attention, therefore anyone with some insight would be great.
Thanks, and sorry for ridiculous thread
Lewis
On Tue, Nov 1, 2011 at 2:01 PM, Lewis John Mcgibbney <
lewis.mcgibbney@gmail.com> wrote:
> Hi,
>
> I'm currently trying to complete NUTCH-902 and GORA-39 and kill two birds
> with the one stone, however I've uprooted some more nasties which I'm now
> trying to address. When configuring Nutchgora with Cassandra I'm getting
> the following
>
> lewis@lewis-01:~/ASF/nutchgora/runtime/local$ bin/nutch inject urls
> crawldb
> InjectorJob: starting
> InjectorJob: urlDir: urls
> InjectorJob: org.apache.gora.util.GoraException: java.io.IOException:
> java.io.IOException: Property with base name "servers" could not be found,
> make sure to include this property in gora.properties file
> at
> org.apache.gora.store.DataStoreFactory.createDataStore(DataStoreFactory.java:110)
> at
> org.apache.gora.store.DataStoreFactory.createDataStore(DataStoreFactory.java:93)
> at
> org.apache.nutch.storage.StorageUtils.createWebStore(StorageUtils.java:59)
> at org.apache.nutch.crawl.InjectorJob.run(InjectorJob.java:243)
> at org.apache.nutch.crawl.InjectorJob.inject(InjectorJob.java:268)
> at org.apache.nutch.crawl.InjectorJob.run(InjectorJob.java:282)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> at org.apache.nutch.crawl.InjectorJob.main(InjectorJob.java:292)
> Caused by: java.io.IOException: java.io.IOException: Property with base
> name "servers" could not be found, make sure to include this property in
> gora.properties file
> at
> org.apache.gora.cassandra.store.CassandraStore.readMapping(CassandraStore.java:462)
> at
> org.apache.gora.cassandra.store.CassandraStore.initialize(CassandraStore.java:91)
> at
> org.apache.gora.store.DataStoreFactory.initializeDataStore(DataStoreFactory.java:81)
> at
> org.apache.gora.store.DataStoreFactory.createDataStore(DataStoreFactory.java:104)
> ... 7 more
> Caused by: java.io.IOException: Property with base name "servers" could
> not be found, make sure to include this property in gora.properties file
> at
> org.apache.gora.store.DataStoreFactory.findPropertyOrDie(DataStoreFactory.java:254)
> at
> org.apache.gora.cassandra.store.CassandraStore.createClient(CassandraStore.java:394)
> at
> org.apache.gora.cassandra.store.CassandraStore.readMapping(CassandraStore.java:425)
> ... 10 more
>
> Can someone please explain a bit about what kind of properties we
> can/should add to gora.properties for cassandra setup. I've tried editing
> gora.properties as follows with no luck
>
> #gora.sqlstore.jdbc.driver=org.hsqldb.jdbcDriver
> #gora.sqlstore.jdbc.url=jdbc:hsqldb:hsql://localhost/nutchtest
> servers=localhost/127.0.0.1:9160
>
> If there are any resources people are aware of on the net then I'll begin
> getting my head around them.
>
> Thanks in advance
>
> Lewis
>
>
> --
> *Lewis*
>
>
--
*Lewis*
Re: Setting properties in gora.properties
Posted by Enis Söztutar <en...@gmail.com>.
Hi Lewis,
I guess in gora-cassandra/src/test/conf/gora.properties, the servers are
listed as:
gora.cassandrastore.servers=localhost:9160
In setting the properties for gora data stores, you have to supply the data
store that it applies to. The documentation at
https://incubator.apache.org/gora/docs/current/gora-conf.html
seems fair enough, but we(as gora-devs) have to fill out data-store
specific configuration documentation
https://incubator.apache.org/gora/docs/current/gora-hbase.html
https://incubator.apache.org/gora/docs/current/gora-cassandra.html
Thanks,
Enis
On Wed, Nov 2, 2011 at 12:20 PM, Lewis John Mcgibbney <
lewis.mcgibbney@gmail.com> wrote:
> Hi,
>
> (I'm keeping gora-dev's in on this one as well)
>
> A quick update to this one, having submitted a work in progress patch for
> NUTCH-1189, so that properties are easier for the gora-cassandra module to
> find I'm trying to get the configuration to work as follows
>
> # Cassandra store properties
> # gora.CassandraMapping.keyspace=
> # gora.CassandraMapping.name=
> # gora.CassandraMapping.class=
> # gora.CassandraMapping.qualifier=
> # gora.CassandraMapping.family=
> # gora.CassandraMapping.type=
> # gora.CassandraMapping.cluster=
> gora.CassandraMapping.host=localhost://127.0.0.1:9160
>
> However I'm still getting
>
> Caused by: java.io.IOException: Property with base name "servers" could not
> be found, make sure to include this property in gora.properties file
>
> This is caused by a null value being returned for baseKey parameter in
> DataStoreFactory.
>
> I think it's important to note, that in order to get Nutchgora to build
> with the gora-cassandra properties uncommented in NUTCHGORA/ivy/ivy.xml it
> was necessary to include the following exclusions
>
> <dependency org="org.apache.gora" name="gora-cassandra"
> rev="0.1.1-incubating" conf="*->default">
> <exclude org="org.apache.thrift" />
> <exclude org="org.apache.cassandra" />
> </dependency>
>
> Firstly, does anyone have a solution regarding the removing or substituting
> the exclusions? Enis has mentioned that they should be included within
> gora-cassandra/ivy/ivy.xml, however as this has not been loaded to maven
> repos it is not currently a possibility therefore they need to be included
> in the Nutchgora side of things.
> Secondly, does anyone know how the illusive baseKey parameter should be
> included within the gora.properties?
> Presumably something similar to
>
> # gora.CassandraMapping.servers=somethingSomething.com
>
> This is nearly sorted (I hope) but there are a couple of areas that need
> attention, therefore anyone with some insight would be great.
>
> Thanks, and sorry for ridiculous thread
>
> Lewis
>
> On Tue, Nov 1, 2011 at 2:01 PM, Lewis John Mcgibbney <
> lewis.mcgibbney@gmail.com> wrote:
>
> > Hi,
> >
> > I'm currently trying to complete NUTCH-902 and GORA-39 and kill two birds
> > with the one stone, however I've uprooted some more nasties which I'm now
> > trying to address. When configuring Nutchgora with Cassandra I'm getting
> > the following
> >
> > lewis@lewis-01:~/ASF/nutchgora/runtime/local$ bin/nutch inject urls
> > crawldb
> > InjectorJob: starting
> > InjectorJob: urlDir: urls
> > InjectorJob: org.apache.gora.util.GoraException: java.io.IOException:
> > java.io.IOException: Property with base name "servers" could not be
> found,
> > make sure to include this property in gora.properties file
> > at
> >
> org.apache.gora.store.DataStoreFactory.createDataStore(DataStoreFactory.java:110)
> > at
> >
> org.apache.gora.store.DataStoreFactory.createDataStore(DataStoreFactory.java:93)
> > at
> >
> org.apache.nutch.storage.StorageUtils.createWebStore(StorageUtils.java:59)
> > at org.apache.nutch.crawl.InjectorJob.run(InjectorJob.java:243)
> > at org.apache.nutch.crawl.InjectorJob.inject(InjectorJob.java:268)
> > at org.apache.nutch.crawl.InjectorJob.run(InjectorJob.java:282)
> > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> > at org.apache.nutch.crawl.InjectorJob.main(InjectorJob.java:292)
> > Caused by: java.io.IOException: java.io.IOException: Property with base
> > name "servers" could not be found, make sure to include this property in
> > gora.properties file
> > at
> >
> org.apache.gora.cassandra.store.CassandraStore.readMapping(CassandraStore.java:462)
> > at
> >
> org.apache.gora.cassandra.store.CassandraStore.initialize(CassandraStore.java:91)
> > at
> >
> org.apache.gora.store.DataStoreFactory.initializeDataStore(DataStoreFactory.java:81)
> > at
> >
> org.apache.gora.store.DataStoreFactory.createDataStore(DataStoreFactory.java:104)
> > ... 7 more
> > Caused by: java.io.IOException: Property with base name "servers" could
> > not be found, make sure to include this property in gora.properties file
> > at
> >
> org.apache.gora.store.DataStoreFactory.findPropertyOrDie(DataStoreFactory.java:254)
> > at
> >
> org.apache.gora.cassandra.store.CassandraStore.createClient(CassandraStore.java:394)
> > at
> >
> org.apache.gora.cassandra.store.CassandraStore.readMapping(CassandraStore.java:425)
> > ... 10 more
> >
> > Can someone please explain a bit about what kind of properties we
> > can/should add to gora.properties for cassandra setup. I've tried editing
> > gora.properties as follows with no luck
> >
> > #gora.sqlstore.jdbc.driver=org.hsqldb.jdbcDriver
> > #gora.sqlstore.jdbc.url=jdbc:hsqldb:hsql://localhost/nutchtest
> > servers=localhost/127.0.0.1:9160
> >
> > If there are any resources people are aware of on the net then I'll begin
> > getting my head around them.
> >
> > Thanks in advance
> >
> > Lewis
> >
> >
> > --
> > *Lewis*
> >
> >
>
>
> --
> *Lewis*
>
Re: Setting properties in gora.properties
Posted by Enis Söztutar <en...@gmail.com>.
Hi Lewis,
I guess in gora-cassandra/src/test/conf/gora.properties, the servers are
listed as:
gora.cassandrastore.servers=localhost:9160
In setting the properties for gora data stores, you have to supply the data
store that it applies to. The documentation at
https://incubator.apache.org/gora/docs/current/gora-conf.html
seems fair enough, but we(as gora-devs) have to fill out data-store
specific configuration documentation
https://incubator.apache.org/gora/docs/current/gora-hbase.html
https://incubator.apache.org/gora/docs/current/gora-cassandra.html
Thanks,
Enis
On Wed, Nov 2, 2011 at 12:20 PM, Lewis John Mcgibbney <
lewis.mcgibbney@gmail.com> wrote:
> Hi,
>
> (I'm keeping gora-dev's in on this one as well)
>
> A quick update to this one, having submitted a work in progress patch for
> NUTCH-1189, so that properties are easier for the gora-cassandra module to
> find I'm trying to get the configuration to work as follows
>
> # Cassandra store properties
> # gora.CassandraMapping.keyspace=
> # gora.CassandraMapping.name=
> # gora.CassandraMapping.class=
> # gora.CassandraMapping.qualifier=
> # gora.CassandraMapping.family=
> # gora.CassandraMapping.type=
> # gora.CassandraMapping.cluster=
> gora.CassandraMapping.host=localhost://127.0.0.1:9160
>
> However I'm still getting
>
> Caused by: java.io.IOException: Property with base name "servers" could not
> be found, make sure to include this property in gora.properties file
>
> This is caused by a null value being returned for baseKey parameter in
> DataStoreFactory.
>
> I think it's important to note, that in order to get Nutchgora to build
> with the gora-cassandra properties uncommented in NUTCHGORA/ivy/ivy.xml it
> was necessary to include the following exclusions
>
> <dependency org="org.apache.gora" name="gora-cassandra"
> rev="0.1.1-incubating" conf="*->default">
> <exclude org="org.apache.thrift" />
> <exclude org="org.apache.cassandra" />
> </dependency>
>
> Firstly, does anyone have a solution regarding the removing or substituting
> the exclusions? Enis has mentioned that they should be included within
> gora-cassandra/ivy/ivy.xml, however as this has not been loaded to maven
> repos it is not currently a possibility therefore they need to be included
> in the Nutchgora side of things.
> Secondly, does anyone know how the illusive baseKey parameter should be
> included within the gora.properties?
> Presumably something similar to
>
> # gora.CassandraMapping.servers=somethingSomething.com
>
> This is nearly sorted (I hope) but there are a couple of areas that need
> attention, therefore anyone with some insight would be great.
>
> Thanks, and sorry for ridiculous thread
>
> Lewis
>
> On Tue, Nov 1, 2011 at 2:01 PM, Lewis John Mcgibbney <
> lewis.mcgibbney@gmail.com> wrote:
>
> > Hi,
> >
> > I'm currently trying to complete NUTCH-902 and GORA-39 and kill two birds
> > with the one stone, however I've uprooted some more nasties which I'm now
> > trying to address. When configuring Nutchgora with Cassandra I'm getting
> > the following
> >
> > lewis@lewis-01:~/ASF/nutchgora/runtime/local$ bin/nutch inject urls
> > crawldb
> > InjectorJob: starting
> > InjectorJob: urlDir: urls
> > InjectorJob: org.apache.gora.util.GoraException: java.io.IOException:
> > java.io.IOException: Property with base name "servers" could not be
> found,
> > make sure to include this property in gora.properties file
> > at
> >
> org.apache.gora.store.DataStoreFactory.createDataStore(DataStoreFactory.java:110)
> > at
> >
> org.apache.gora.store.DataStoreFactory.createDataStore(DataStoreFactory.java:93)
> > at
> >
> org.apache.nutch.storage.StorageUtils.createWebStore(StorageUtils.java:59)
> > at org.apache.nutch.crawl.InjectorJob.run(InjectorJob.java:243)
> > at org.apache.nutch.crawl.InjectorJob.inject(InjectorJob.java:268)
> > at org.apache.nutch.crawl.InjectorJob.run(InjectorJob.java:282)
> > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> > at org.apache.nutch.crawl.InjectorJob.main(InjectorJob.java:292)
> > Caused by: java.io.IOException: java.io.IOException: Property with base
> > name "servers" could not be found, make sure to include this property in
> > gora.properties file
> > at
> >
> org.apache.gora.cassandra.store.CassandraStore.readMapping(CassandraStore.java:462)
> > at
> >
> org.apache.gora.cassandra.store.CassandraStore.initialize(CassandraStore.java:91)
> > at
> >
> org.apache.gora.store.DataStoreFactory.initializeDataStore(DataStoreFactory.java:81)
> > at
> >
> org.apache.gora.store.DataStoreFactory.createDataStore(DataStoreFactory.java:104)
> > ... 7 more
> > Caused by: java.io.IOException: Property with base name "servers" could
> > not be found, make sure to include this property in gora.properties file
> > at
> >
> org.apache.gora.store.DataStoreFactory.findPropertyOrDie(DataStoreFactory.java:254)
> > at
> >
> org.apache.gora.cassandra.store.CassandraStore.createClient(CassandraStore.java:394)
> > at
> >
> org.apache.gora.cassandra.store.CassandraStore.readMapping(CassandraStore.java:425)
> > ... 10 more
> >
> > Can someone please explain a bit about what kind of properties we
> > can/should add to gora.properties for cassandra setup. I've tried editing
> > gora.properties as follows with no luck
> >
> > #gora.sqlstore.jdbc.driver=org.hsqldb.jdbcDriver
> > #gora.sqlstore.jdbc.url=jdbc:hsqldb:hsql://localhost/nutchtest
> > servers=localhost/127.0.0.1:9160
> >
> > If there are any resources people are aware of on the net then I'll begin
> > getting my head around them.
> >
> > Thanks in advance
> >
> > Lewis
> >
> >
> > --
> > *Lewis*
> >
> >
>
>
> --
> *Lewis*
>