[jira] [Comment Edited] (GORA-173) Iterating over StatefulHashMap in HBaseStore throws ConcurrentModificationException

["raf shin (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/GORA-173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13485370#comment-13485370 ] 

raf shin edited comment on GORA-173 at 10/27/12 7:13 AM:
---------------------------------------------------------

We have changed Nutch so that writing fetched pages is done in separate threads than that of hadoop or fetchers. so we have created a store for each of our writer threads. The writer threads just create the store and call store.put(key, webpage).
Because we have a store for each writer thread it is not the case that we have multiple threads on one store.


We have been using the same code from 6 months ago but we didn't face this exception until recently that we have increased our bandwidth from 10Mbps to 100Mbps.
private class WriteThread extends Thread {
    private DataStore<String, WebPage> storeWebPage = null;
   public WriteThread(){
         storeWebPage = StorageUtils.createWebStore(conf, String.class,
						WebPage.class);
         ....
}
public void run(){
   ...
   fit.page.clearDirty("content");
   extractOutlinks(fit);
  updateWebPage(fit.url, fit.page);
  storeWebPage.put(key, fit.page);
  ...
}
						
                
      was (Author: rafshin):
    We have changed Nutch so that writing fetched pages is done in separate threads than that of hadoop or fetchers. so we have created a store for each of our writer threads. The writer threads just create the store and call store.put(key, webpage).
Because we have a store for each writer thread it is not the case that we have multiple threads on one store.


We have been using the same code from 6 months ago but we didn't face this exception until recently that we have increased our bandwidth from 10Mbps to 100Mbps.
                  
> Iterating over StatefulHashMap in HBaseStore throws ConcurrentModificationException
> -----------------------------------------------------------------------------------
>
>                 Key: GORA-173
>                 URL: https://issues.apache.org/jira/browse/GORA-173
>             Project: Apache Gora
>          Issue Type: Bug
>    Affects Versions: 0.1.1-incubating
>            Reporter: raf shin
>
> We are using just one thread working with HBaseStore in Nutch.
> HBaseStore.put throws the following exception. I think the same problem was in CassandraStore that is
> solved now. But the problem in HBaseStore remains. We used the same solution for HBaseStore but
> it didn't work.
> java.util.ConcurrentModificationException
> 	at java.util.HashMap$HashIterator.nextEntry(HashMap.java:793)
> 	at java.util.HashMap$EntryIterator.next(HashMap.java:834)
> 	at java.util.HashMap$EntryIterator.next(HashMap.java:832)
> 	at org.apache.gora.hbase.store.HBaseStore.put(HBaseStore.java:267)
> 	at org.nise.hiva.fetcher.FetcherReducer$WriteThread.run(FetcherReducer.java:1557)
> Line 266 in HBaseStore:
>  StatefulHashMap<Utf8, ?> map = (StatefulHashMap<Utf8, ?>) o;
>             for (Entry<Utf8, State> e : map.states().entrySet()) {
>               Utf8 mapKey = e.getKey();
>               switch (e.getValue()) {
>                 case DIRTY:
>                   byte[] qual = Bytes.toBytes(mapKey.toString());
>                   byte[] val = toBytes(map.get(mapKey), field.schema().getValueType());
>                   put.add(hcol.getFamily(), qual, val);
>                   hasPuts = true;
>                   break;
>                 case DELETED:
>                   qual = Bytes.toBytes(mapKey.toString());
>                   hasDeletes = true;
>                   delete.deleteColumn(hcol.getFamily(), qual);
>                   break;
>               }
> ***************************************************************************
> Our solution was :
> Set<Utf8> keys = map.keySet();
> Utf8 [] arr = (Utf8[])keys.toArray();
> for(Utf8 key: arr){....}
> But it didn't work and we throw the same exception.
> Our version is 0.1 but i checked the trunk and did't see much change in that code so the trunk code
> may throw the same exception.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira