You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by reddibabu <re...@gmail.com> on 2014/07/24 13:11:54 UTC

Not able to put Twitter4j.Place object attributes from scala application

Hi,

I am working on Twitter API for retrieve the twitter information and store
those details into Hbase table.
Here, I am not able to put twitter4j.Place attributes into Hbase table, I am
able to retrieve Place and GeoLocation properties from Twitter API and able
to see on Client application. The below mentioned is the line of code for
put operation in my scala application, able to put twitter4j.User and
twitter4j.Status properties correctly into hbase table.

rowkey.add(Bytes.toBytes("columnFamily"),Bytes.toBytes("latitude"),Bytes.toBytes("latitude
information"))

Is there any another specific way not like above mentioned line of code to
put those Place/GeoLocation properties into HBase table.

Any assistance is greatful.

Thanks,
Reddi Babu





--
View this message in context: http://apache-hbase.679495.n3.nabble.com/Not-able-to-put-Twitter4j-Place-object-attributes-from-scala-application-tp4061914.html
Sent from the HBase User mailing list archive at Nabble.com.

Re: Not able to put Twitter4j.Place object attributes from scala application

Posted by Jean-Marc Spaggiari <je...@spaggiari.org>.
Indeed, interesting.

Here is an extract from the exception Reddi as posted:

NoSuchColumnFamilyException: Column  family 'cf'  does not exist ... in
table 'twitter_mobile_data', {NAME => 'cf', .....

What hbck gives you? what do you have in the HDFS folder for this region?

JM

2014-07-27 22:36 GMT-04:00 reddibabu <re...@gmail.com>:

> Hi Ted,
> Thanks for your response.
>
> I have 'cf' columnfamily in my table, twitter_mobile_data.
> e.geoLocation is the Spark RDD, I have called getLatitude() on it.
>
> trowkey.add(cf,Bytes.toBytes("latitude"),Bytes.toBytes(e.geoLocation.getLatitude()))
>
> trowkey.add(cf,Bytes.toBytes("longitude"),Bytes.toBytes(e.geoLocation.getLongitude()))
>
> I have tried one more way like,
>  var latitude=Bytes.toBytes(e.geoLocation.getLatitude())
>  var longitude=Bytes.toBytes(e.getLocation.getLongitude())
>
>  trowkey.add(cf,Bytes.toBytes("latitude"),Bytes.toBytes(latitude))
>  trowkey.add(cf,Bytes.toBytes("longitude"),Bytes.toBytes(longitude))
>
> both ways are not helpful.
>
> Here, I am not able to put Twitter4j.Place object attributes into Hbase
> table like,
>          GeoLocation.getLatitude()
>          GeoLocation.getLongitude()
>          Place.getCountry()
>          Place.getCountryCode()
>          Place.getFullName()
>          Place.getId()
>          Place.getName()
>          Place.getPlaceType()
>          Place.getStreetAddress()
>          Place.getGeometryType()
>
>
>
> Thanks in advance,
> Reddi Babu
>
>
>
>
> --
> View this message in context:
> http://apache-hbase.679495.n3.nabble.com/Not-able-to-put-Twitter4j-Place-object-attributes-from-scala-application-tp4061914p4061994.html
> Sent from the HBase User mailing list archive at Nabble.com.
>

Re: Not able to put Twitter4j.Place object attributes from scala application

Posted by reddibabu <re...@gmail.com>.
Hi Ted,
Thanks for your response.

I have 'cf' columnfamily in my table, twitter_mobile_data.
e.geoLocation is the Spark RDD, I have called getLatitude() on it. 
trowkey.add(cf,Bytes.toBytes("latitude"),Bytes.toBytes(e.geoLocation.getLatitude()))
trowkey.add(cf,Bytes.toBytes("longitude"),Bytes.toBytes(e.geoLocation.getLongitude()))

I have tried one more way like,
 var latitude=Bytes.toBytes(e.geoLocation.getLatitude())
 var longitude=Bytes.toBytes(e.getLocation.getLongitude())

 trowkey.add(cf,Bytes.toBytes("latitude"),Bytes.toBytes(latitude))
 trowkey.add(cf,Bytes.toBytes("longitude"),Bytes.toBytes(longitude))

both ways are not helpful.

Here, I am not able to put Twitter4j.Place object attributes into Hbase
table like,
         GeoLocation.getLatitude()
         GeoLocation.getLongitude()
         Place.getCountry()
         Place.getCountryCode()
         Place.getFullName()
         Place.getId()
         Place.getName()
         Place.getPlaceType()
         Place.getStreetAddress()
         Place.getGeometryType()
         


Thanks in advance,
Reddi Babu




--
View this message in context: http://apache-hbase.679495.n3.nabble.com/Not-able-to-put-Twitter4j-Place-object-attributes-from-scala-application-tp4061914p4061994.html
Sent from the HBase User mailing list archive at Nabble.com.

Re: Not able to put Twitter4j.Place object attributes from scala application

Posted by Ted Yu <yu...@gmail.com>.
The "Column family 'cf'  does not exist in region" error seemed to be
associated with openScanner request.

Does twitter_mobile_data contain such column family ?

bq. Bytes.toBytes("latitude information")

I guess the above is pseudo code. Can you give a real sample ?

Thanks


On Thu, Jul 24, 2014 at 8:03 PM, reddibabu <re...@gmail.com>
wrote:

> Hi Ted,
> Thanks for responding my post.
>
> I am using HBase 0.94.15-cdh4.6.0. & same application can able to put
> twitter4j.Status and twitter4j.User attributes into hbase table. I have't
> seen any error messages on console. Few errors in hbase RegionServer logs
> as
> specified below...
>
> 2014-07-24 02:25:26,638 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: Client tried to access
> missing scanner -1
> 2014-07-24 02:25:56,270 ERROR
> org.apache.hadoop.hbase.regionserver.HRegionServer: Failed openScanner
> org.apache.hadoop.hbase.regionserver.NoSuchColumnFamilyException: Column
> family 'cf'  does not exist in region
> twitter_mobile_data,,1404930369033.aa8c4d9370c3e8882beb81659734e2c0. in
> table 'twitter_mobile_data', {NAME => 'cf', DATA_BLOCK_ENCODING => 'NONE',
> BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', COMPRESSION => 'NONE',
> VERSIONS => '3', TTL => '2147483647', MIN_VERSIONS => '0',
> KEEP_DELETED_CELLS => 'false', BLOCKSIZE => '65536', ENCODE_ON_DISK =>
> 'true', IN_MEMORY => 'false', BLOCKCACHE => 'true'}
>         at
> org.apache.hadoop.hbase.regionserver.HRegion.checkFamily(HRegion.java:5475)
>         at
> org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1814)
>         at
> org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1794)
>         at
>
> org.apache.hadoop.hbase.regionserver.HRegionServer.internalOpenScanner(HRegionServer.java:2567)
>         at
>
> org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:2538)
>         at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source)
>         at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at
>
> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:323)
>         at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1428)
>
>
> 2014-07-24 02:26:27,797 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: Client tried to access
> missing scanner -1
> 2014-07-24 02:27:36,141 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner
> 3395680567130916233 lease expired on region
> twitter_mobile_data,,1404930369033.aa8c4d9370c3e8882beb81659734e2c0.
> 2014-07-24 02:27:49,200 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner
> 3463114407687997552 lease expired on region
> twitter_mobile_data,,1404930369033.aa8c4d9370c3e8882beb81659734e2c0.
> 2014-07-24 02:27:50,353 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner
> -8911756666985052814 lease expired on region
> twitter_mobile_data,,1404930369033.aa8c4d9370c3e8882beb81659734e2c0.
> 2014-07-24 02:28:02,469 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner
> 2586945861118350525 lease expired on region
> twitter_mobile_data,,1404930369033.aa8c4d9370c3e8882beb81659734e2c0.
> 2014-07-24 02:28:31,974 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner
> 6246362279302788198 lease expired on region
> twitter_mobile_data,,1404930369033.aa8c4d9370c3e8882beb81659734e2c0.
> 2014-07-24 02:28:57,261 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner
> 4048750095102992142 lease expired on region
> twitter_mobile_data,,1404930369033.aa8c4d9370c3e8882beb81659734e2c0.
> 2014-07-24 02:29:38,633 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner
> 4557453676796441030 lease expired on region
> twitter_mobile_data,,1404930369033.aa8c4d9370c3e8882beb81659734e2c0.
> 2014-07-24 02:29:51,891 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner
> -3356919412101426008 lease expired on region
> twitter_mobile_data,,1404930369033.aa8c4d9370c3e8882beb81659734e2c0.
> 2014-07-24 02:42:52,816 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server
> listener on 60020: readAndProcess threw exception java.io.IOException:
> Connection reset by peer. Count of bytes read: 0
> java.io.IOException: Connection reset by peer
>         at sun.nio.ch.FileDispatcher.read0(Native Method)
>         at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
>         at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:198)
>         at sun.nio.ch.IOUtil.read(IOUtil.java:171)
>         at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:243)
>         at
> org.apache.hadoop.hbase.ipc.HBaseServer.channelRead(HBaseServer.java:1798)
>         at
>
> org.apache.hadoop.hbase.ipc.HBaseServer$Connection.readAndProcess(HBaseServer.java:1181)
>         at
>
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener.doRead(HBaseServer.java:750)
>         at
>
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:541)
>         at
>
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:516)
>         at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>         at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>         at java.lang.Thread.run(Thread.java:662)
>
>
>
>
>
> --
> View this message in context:
> http://apache-hbase.679495.n3.nabble.com/Not-able-to-put-Twitter4j-Place-object-attributes-from-scala-application-tp4061914p4061944.html
> Sent from the HBase User mailing list archive at Nabble.com.
>

Re: Not able to put Twitter4j.Place object attributes from scala application

Posted by reddibabu <re...@gmail.com>.
Hi Ted,
Thanks for responding my post.

I am using HBase 0.94.15-cdh4.6.0. & same application can able to put
twitter4j.Status and twitter4j.User attributes into hbase table. I have't
seen any error messages on console. Few errors in hbase RegionServer logs as
specified below...

2014-07-24 02:25:26,638 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Client tried to access
missing scanner -1
2014-07-24 02:25:56,270 ERROR
org.apache.hadoop.hbase.regionserver.HRegionServer: Failed openScanner
org.apache.hadoop.hbase.regionserver.NoSuchColumnFamilyException: Column
family 'cf'  does not exist in region
twitter_mobile_data,,1404930369033.aa8c4d9370c3e8882beb81659734e2c0. in
table 'twitter_mobile_data', {NAME => 'cf', DATA_BLOCK_ENCODING => 'NONE',
BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', COMPRESSION => 'NONE',
VERSIONS => '3', TTL => '2147483647', MIN_VERSIONS => '0',
KEEP_DELETED_CELLS => 'false', BLOCKSIZE => '65536', ENCODE_ON_DISK =>
'true', IN_MEMORY => 'false', BLOCKCACHE => 'true'}
        at
org.apache.hadoop.hbase.regionserver.HRegion.checkFamily(HRegion.java:5475)
        at
org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1814)
        at
org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1794)
        at
org.apache.hadoop.hbase.regionserver.HRegionServer.internalOpenScanner(HRegionServer.java:2567)
        at
org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:2538)
        at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:323)
        at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1428)


2014-07-24 02:26:27,797 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Client tried to access
missing scanner -1
2014-07-24 02:27:36,141 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner
3395680567130916233 lease expired on region
twitter_mobile_data,,1404930369033.aa8c4d9370c3e8882beb81659734e2c0.
2014-07-24 02:27:49,200 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner
3463114407687997552 lease expired on region
twitter_mobile_data,,1404930369033.aa8c4d9370c3e8882beb81659734e2c0.
2014-07-24 02:27:50,353 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner
-8911756666985052814 lease expired on region
twitter_mobile_data,,1404930369033.aa8c4d9370c3e8882beb81659734e2c0.
2014-07-24 02:28:02,469 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner
2586945861118350525 lease expired on region
twitter_mobile_data,,1404930369033.aa8c4d9370c3e8882beb81659734e2c0.
2014-07-24 02:28:31,974 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner
6246362279302788198 lease expired on region
twitter_mobile_data,,1404930369033.aa8c4d9370c3e8882beb81659734e2c0.
2014-07-24 02:28:57,261 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner
4048750095102992142 lease expired on region
twitter_mobile_data,,1404930369033.aa8c4d9370c3e8882beb81659734e2c0.
2014-07-24 02:29:38,633 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner
4557453676796441030 lease expired on region
twitter_mobile_data,,1404930369033.aa8c4d9370c3e8882beb81659734e2c0.
2014-07-24 02:29:51,891 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner
-3356919412101426008 lease expired on region
twitter_mobile_data,,1404930369033.aa8c4d9370c3e8882beb81659734e2c0.
2014-07-24 02:42:52,816 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server
listener on 60020: readAndProcess threw exception java.io.IOException:
Connection reset by peer. Count of bytes read: 0
java.io.IOException: Connection reset by peer
        at sun.nio.ch.FileDispatcher.read0(Native Method)
        at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
        at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:198)
        at sun.nio.ch.IOUtil.read(IOUtil.java:171)
        at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:243)
        at
org.apache.hadoop.hbase.ipc.HBaseServer.channelRead(HBaseServer.java:1798)
        at
org.apache.hadoop.hbase.ipc.HBaseServer$Connection.readAndProcess(HBaseServer.java:1181)
        at
org.apache.hadoop.hbase.ipc.HBaseServer$Listener.doRead(HBaseServer.java:750)
        at
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:541)
        at
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:516)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)





--
View this message in context: http://apache-hbase.679495.n3.nabble.com/Not-able-to-put-Twitter4j-Place-object-attributes-from-scala-application-tp4061914p4061944.html
Sent from the HBase User mailing list archive at Nabble.com.

Re: Not able to put Twitter4j.Place object attributes from scala application

Posted by Ted Yu <yu...@gmail.com>.
Do you see any exception in region server logs ?

Which release of hbase are you using ?

Cheers

On Jul 24, 2014, at 3:11 AM, reddibabu <re...@gmail.com> wrote:

> Hi,
> 
> I am working on Twitter API for retrieve the twitter information and store
> those details into Hbase table.
> Here, I am not able to put twitter4j.Place attributes into Hbase table, I am
> able to retrieve Place and GeoLocation properties from Twitter API and able
> to see on Client application. The below mentioned is the line of code for
> put operation in my scala application, able to put twitter4j.User and
> twitter4j.Status properties correctly into hbase table.
> 
> rowkey.add(Bytes.toBytes("columnFamily"),Bytes.toBytes("latitude"),Bytes.toBytes("latitude
> information"))
> 
> Is there any another specific way not like above mentioned line of code to
> put those Place/GeoLocation properties into HBase table.
> 
> Any assistance is greatful.
> 
> Thanks,
> Reddi Babu
> 
> 
> 
> 
> 
> --
> View this message in context: http://apache-hbase.679495.n3.nabble.com/Not-able-to-put-Twitter4j-Place-object-attributes-from-scala-application-tp4061914.html
> Sent from the HBase User mailing list archive at Nabble.com.