You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@gora.apache.org by "raf shin (JIRA)" <ji...@apache.org> on 2012/10/25 10:12:12 UTC
[jira] [Created] (GORA-173) Iterating over StatefulHashMap in
HBaseStore throws ConcurrentModificationException
raf shin created GORA-173:
-----------------------------
Summary: Iterating over StatefulHashMap in HBaseStore throws ConcurrentModificationException
Key: GORA-173
URL: https://issues.apache.org/jira/browse/GORA-173
Project: Apache Gora
Issue Type: Bug
Affects Versions: 0.1.1-incubating
Reporter: raf shin
We are using just one thread working with HBaseStore in Nutch.
HBaseStore.put throws the following exception. I think the same problem was in CassandraStore that is
solved now. But the problem in HBaseStore remains. We used the same solution for HBaseStore but
it didn't work.
java.util.ConcurrentModificationException
at java.util.HashMap$HashIterator.nextEntry(HashMap.java:793)
at java.util.HashMap$EntryIterator.next(HashMap.java:834)
at java.util.HashMap$EntryIterator.next(HashMap.java:832)
at org.apache.gora.hbase.store.HBaseStore.put(HBaseStore.java:267)
at org.nise.hiva.fetcher.FetcherReducer$WriteThread.run(FetcherReducer.java:1557)
Line 266 in HBaseStore:
StatefulHashMap<Utf8, ?> map = (StatefulHashMap<Utf8, ?>) o;
for (Entry<Utf8, State> e : map.states().entrySet()) {
Utf8 mapKey = e.getKey();
switch (e.getValue()) {
case DIRTY:
byte[] qual = Bytes.toBytes(mapKey.toString());
byte[] val = toBytes(map.get(mapKey), field.schema().getValueType());
put.add(hcol.getFamily(), qual, val);
hasPuts = true;
break;
case DELETED:
qual = Bytes.toBytes(mapKey.toString());
hasDeletes = true;
delete.deleteColumn(hcol.getFamily(), qual);
break;
}
***************************************************************************
Our solution was :
Set<Utf8> keys = map.keySet();
Utf8 [] arr = (Utf8[])keys.toArray();
for(Utf8 key: arr){....}
But it didn't work and we throw the same exception.
Our version is 0.1 but i checked the trunk and did't see much change in that code so the trunk code
may throw the same exception.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GORA-173) Iterating over StatefulHashMap in
HBaseStore throws ConcurrentModificationException
Posted by "raf shin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GORA-173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13485415#comment-13485415 ]
raf shin commented on GORA-173:
-------------------------------
The problem resolved. The problem was not related to Gora HBaseStore. It was related to our own code.
Two writer threads removed the same object from a Queue because the remove operation was not synchronized.
Thread1 HBaseStore.put(key, aWebPage)
Thread2 HBaseStore.put(key, theSameWebPagePointer)
so it caused a ConcurrentModificationException.
We synchronized removing from the queue and it resolved the issue. So i think the issue should be closed.
> Iterating over StatefulHashMap in HBaseStore throws ConcurrentModificationException
> -----------------------------------------------------------------------------------
>
> Key: GORA-173
> URL: https://issues.apache.org/jira/browse/GORA-173
> Project: Apache Gora
> Issue Type: Bug
> Affects Versions: 0.1.1-incubating
> Reporter: raf shin
>
> We are using just one thread working with HBaseStore in Nutch.
> HBaseStore.put throws the following exception. I think the same problem was in CassandraStore that is
> solved now. But the problem in HBaseStore remains. We used the same solution for HBaseStore but
> it didn't work.
> java.util.ConcurrentModificationException
> at java.util.HashMap$HashIterator.nextEntry(HashMap.java:793)
> at java.util.HashMap$EntryIterator.next(HashMap.java:834)
> at java.util.HashMap$EntryIterator.next(HashMap.java:832)
> at org.apache.gora.hbase.store.HBaseStore.put(HBaseStore.java:267)
> at org.nise.hiva.fetcher.FetcherReducer$WriteThread.run(FetcherReducer.java:1557)
> Line 266 in HBaseStore:
> StatefulHashMap<Utf8, ?> map = (StatefulHashMap<Utf8, ?>) o;
> for (Entry<Utf8, State> e : map.states().entrySet()) {
> Utf8 mapKey = e.getKey();
> switch (e.getValue()) {
> case DIRTY:
> byte[] qual = Bytes.toBytes(mapKey.toString());
> byte[] val = toBytes(map.get(mapKey), field.schema().getValueType());
> put.add(hcol.getFamily(), qual, val);
> hasPuts = true;
> break;
> case DELETED:
> qual = Bytes.toBytes(mapKey.toString());
> hasDeletes = true;
> delete.deleteColumn(hcol.getFamily(), qual);
> break;
> }
> ***************************************************************************
> Our solution was :
> Set<Utf8> keys = map.keySet();
> Utf8 [] arr = (Utf8[])keys.toArray();
> for(Utf8 key: arr){....}
> But it didn't work and we throw the same exception.
> Our version is 0.1 but i checked the trunk and did't see much change in that code so the trunk code
> may throw the same exception.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GORA-173) Iterating over StatefulHashMap in
HBaseStore throws ConcurrentModificationException
Posted by "Lewis John McGibbney (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GORA-173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13484922#comment-13484922 ]
Lewis John McGibbney commented on GORA-173:
-------------------------------------------
Which operation on the Nutch side executes the put? Also can you clarify which lines and in which file your solution (which unfortunately didn't work) was for?
> Iterating over StatefulHashMap in HBaseStore throws ConcurrentModificationException
> -----------------------------------------------------------------------------------
>
> Key: GORA-173
> URL: https://issues.apache.org/jira/browse/GORA-173
> Project: Apache Gora
> Issue Type: Bug
> Affects Versions: 0.1.1-incubating
> Reporter: raf shin
>
> We are using just one thread working with HBaseStore in Nutch.
> HBaseStore.put throws the following exception. I think the same problem was in CassandraStore that is
> solved now. But the problem in HBaseStore remains. We used the same solution for HBaseStore but
> it didn't work.
> java.util.ConcurrentModificationException
> at java.util.HashMap$HashIterator.nextEntry(HashMap.java:793)
> at java.util.HashMap$EntryIterator.next(HashMap.java:834)
> at java.util.HashMap$EntryIterator.next(HashMap.java:832)
> at org.apache.gora.hbase.store.HBaseStore.put(HBaseStore.java:267)
> at org.nise.hiva.fetcher.FetcherReducer$WriteThread.run(FetcherReducer.java:1557)
> Line 266 in HBaseStore:
> StatefulHashMap<Utf8, ?> map = (StatefulHashMap<Utf8, ?>) o;
> for (Entry<Utf8, State> e : map.states().entrySet()) {
> Utf8 mapKey = e.getKey();
> switch (e.getValue()) {
> case DIRTY:
> byte[] qual = Bytes.toBytes(mapKey.toString());
> byte[] val = toBytes(map.get(mapKey), field.schema().getValueType());
> put.add(hcol.getFamily(), qual, val);
> hasPuts = true;
> break;
> case DELETED:
> qual = Bytes.toBytes(mapKey.toString());
> hasDeletes = true;
> delete.deleteColumn(hcol.getFamily(), qual);
> break;
> }
> ***************************************************************************
> Our solution was :
> Set<Utf8> keys = map.keySet();
> Utf8 [] arr = (Utf8[])keys.toArray();
> for(Utf8 key: arr){....}
> But it didn't work and we throw the same exception.
> Our version is 0.1 but i checked the trunk and did't see much change in that code so the trunk code
> may throw the same exception.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GORA-173) Iterating over StatefulHashMap in
HBaseStore throws ConcurrentModificationException
Posted by "raf shin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GORA-173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13485370#comment-13485370 ]
raf shin commented on GORA-173:
-------------------------------
We have changed Nutch so that writing fetched pages is done in separate threads than that of hadoop or fetchers. so we have created a store for each of our writer threads. The writer threads just create the store and call store.put(key, webpage).
Because we have a store for each writer thread it is not the case that we have multiple threads on one store.
We have been using the same code from 6 months ago but we didn't face this exception until recently that we have increased our bandwidth from 10Mbps to 100Mbps.
> Iterating over StatefulHashMap in HBaseStore throws ConcurrentModificationException
> -----------------------------------------------------------------------------------
>
> Key: GORA-173
> URL: https://issues.apache.org/jira/browse/GORA-173
> Project: Apache Gora
> Issue Type: Bug
> Affects Versions: 0.1.1-incubating
> Reporter: raf shin
>
> We are using just one thread working with HBaseStore in Nutch.
> HBaseStore.put throws the following exception. I think the same problem was in CassandraStore that is
> solved now. But the problem in HBaseStore remains. We used the same solution for HBaseStore but
> it didn't work.
> java.util.ConcurrentModificationException
> at java.util.HashMap$HashIterator.nextEntry(HashMap.java:793)
> at java.util.HashMap$EntryIterator.next(HashMap.java:834)
> at java.util.HashMap$EntryIterator.next(HashMap.java:832)
> at org.apache.gora.hbase.store.HBaseStore.put(HBaseStore.java:267)
> at org.nise.hiva.fetcher.FetcherReducer$WriteThread.run(FetcherReducer.java:1557)
> Line 266 in HBaseStore:
> StatefulHashMap<Utf8, ?> map = (StatefulHashMap<Utf8, ?>) o;
> for (Entry<Utf8, State> e : map.states().entrySet()) {
> Utf8 mapKey = e.getKey();
> switch (e.getValue()) {
> case DIRTY:
> byte[] qual = Bytes.toBytes(mapKey.toString());
> byte[] val = toBytes(map.get(mapKey), field.schema().getValueType());
> put.add(hcol.getFamily(), qual, val);
> hasPuts = true;
> break;
> case DELETED:
> qual = Bytes.toBytes(mapKey.toString());
> hasDeletes = true;
> delete.deleteColumn(hcol.getFamily(), qual);
> break;
> }
> ***************************************************************************
> Our solution was :
> Set<Utf8> keys = map.keySet();
> Utf8 [] arr = (Utf8[])keys.toArray();
> for(Utf8 key: arr){....}
> But it didn't work and we throw the same exception.
> Our version is 0.1 but i checked the trunk and did't see much change in that code so the trunk code
> may throw the same exception.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (GORA-173) Iterating over StatefulHashMap
in HBaseStore throws ConcurrentModificationException
Posted by "raf shin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GORA-173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13485370#comment-13485370 ]
raf shin edited comment on GORA-173 at 10/27/12 7:13 AM:
---------------------------------------------------------
We have changed Nutch so that writing fetched pages is done in separate threads than that of hadoop or fetchers. so we have created a store for each of our writer threads. The writer threads just create the store and call store.put(key, webpage).
Because we have a store for each writer thread it is not the case that we have multiple threads on one store.
We have been using the same code from 6 months ago but we didn't face this exception until recently that we have increased our bandwidth from 10Mbps to 100Mbps.
private class WriteThread extends Thread {
private DataStore<String, WebPage> storeWebPage = null;
public WriteThread(){
storeWebPage = StorageUtils.createWebStore(conf, String.class,
WebPage.class);
....
}
public void run(){
...
fit.page.clearDirty("content");
extractOutlinks(fit);
updateWebPage(fit.url, fit.page);
storeWebPage.put(key, fit.page);
...
}
was (Author: rafshin):
We have changed Nutch so that writing fetched pages is done in separate threads than that of hadoop or fetchers. so we have created a store for each of our writer threads. The writer threads just create the store and call store.put(key, webpage).
Because we have a store for each writer thread it is not the case that we have multiple threads on one store.
We have been using the same code from 6 months ago but we didn't face this exception until recently that we have increased our bandwidth from 10Mbps to 100Mbps.
> Iterating over StatefulHashMap in HBaseStore throws ConcurrentModificationException
> -----------------------------------------------------------------------------------
>
> Key: GORA-173
> URL: https://issues.apache.org/jira/browse/GORA-173
> Project: Apache Gora
> Issue Type: Bug
> Affects Versions: 0.1.1-incubating
> Reporter: raf shin
>
> We are using just one thread working with HBaseStore in Nutch.
> HBaseStore.put throws the following exception. I think the same problem was in CassandraStore that is
> solved now. But the problem in HBaseStore remains. We used the same solution for HBaseStore but
> it didn't work.
> java.util.ConcurrentModificationException
> at java.util.HashMap$HashIterator.nextEntry(HashMap.java:793)
> at java.util.HashMap$EntryIterator.next(HashMap.java:834)
> at java.util.HashMap$EntryIterator.next(HashMap.java:832)
> at org.apache.gora.hbase.store.HBaseStore.put(HBaseStore.java:267)
> at org.nise.hiva.fetcher.FetcherReducer$WriteThread.run(FetcherReducer.java:1557)
> Line 266 in HBaseStore:
> StatefulHashMap<Utf8, ?> map = (StatefulHashMap<Utf8, ?>) o;
> for (Entry<Utf8, State> e : map.states().entrySet()) {
> Utf8 mapKey = e.getKey();
> switch (e.getValue()) {
> case DIRTY:
> byte[] qual = Bytes.toBytes(mapKey.toString());
> byte[] val = toBytes(map.get(mapKey), field.schema().getValueType());
> put.add(hcol.getFamily(), qual, val);
> hasPuts = true;
> break;
> case DELETED:
> qual = Bytes.toBytes(mapKey.toString());
> hasDeletes = true;
> delete.deleteColumn(hcol.getFamily(), qual);
> break;
> }
> ***************************************************************************
> Our solution was :
> Set<Utf8> keys = map.keySet();
> Utf8 [] arr = (Utf8[])keys.toArray();
> for(Utf8 key: arr){....}
> But it didn't work and we throw the same exception.
> Our version is 0.1 but i checked the trunk and did't see much change in that code so the trunk code
> may throw the same exception.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GORA-173) Iterating over StatefulHashMap in
HBaseStore throws ConcurrentModificationException
Posted by "raf shin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GORA-173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13485382#comment-13485382 ]
raf shin commented on GORA-173:
-------------------------------
our solution was in the HBaseStore Line 266 as stated above where we have a for clause on map.states (where nextentry causes concurrent modification). Our solution was to copy keys into a Utf8[] array and loop over it instead of map.states.keyset() but it didn't work.
Our new solution is to cast map.states into a ConcurrentHashMap(Utt8, State) and then loop over it.
> Iterating over StatefulHashMap in HBaseStore throws ConcurrentModificationException
> -----------------------------------------------------------------------------------
>
> Key: GORA-173
> URL: https://issues.apache.org/jira/browse/GORA-173
> Project: Apache Gora
> Issue Type: Bug
> Affects Versions: 0.1.1-incubating
> Reporter: raf shin
>
> We are using just one thread working with HBaseStore in Nutch.
> HBaseStore.put throws the following exception. I think the same problem was in CassandraStore that is
> solved now. But the problem in HBaseStore remains. We used the same solution for HBaseStore but
> it didn't work.
> java.util.ConcurrentModificationException
> at java.util.HashMap$HashIterator.nextEntry(HashMap.java:793)
> at java.util.HashMap$EntryIterator.next(HashMap.java:834)
> at java.util.HashMap$EntryIterator.next(HashMap.java:832)
> at org.apache.gora.hbase.store.HBaseStore.put(HBaseStore.java:267)
> at org.nise.hiva.fetcher.FetcherReducer$WriteThread.run(FetcherReducer.java:1557)
> Line 266 in HBaseStore:
> StatefulHashMap<Utf8, ?> map = (StatefulHashMap<Utf8, ?>) o;
> for (Entry<Utf8, State> e : map.states().entrySet()) {
> Utf8 mapKey = e.getKey();
> switch (e.getValue()) {
> case DIRTY:
> byte[] qual = Bytes.toBytes(mapKey.toString());
> byte[] val = toBytes(map.get(mapKey), field.schema().getValueType());
> put.add(hcol.getFamily(), qual, val);
> hasPuts = true;
> break;
> case DELETED:
> qual = Bytes.toBytes(mapKey.toString());
> hasDeletes = true;
> delete.deleteColumn(hcol.getFamily(), qual);
> break;
> }
> ***************************************************************************
> Our solution was :
> Set<Utf8> keys = map.keySet();
> Utf8 [] arr = (Utf8[])keys.toArray();
> for(Utf8 key: arr){....}
> But it didn't work and we throw the same exception.
> Our version is 0.1 but i checked the trunk and did't see much change in that code so the trunk code
> may throw the same exception.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Closed] (GORA-173) Iterating over StatefulHashMap in
HBaseStore throws ConcurrentModificationException
Posted by "raf shin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GORA-173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
raf shin closed GORA-173.
-------------------------
Resolution: Not A Problem
The problem was not related to Gora.
> Iterating over StatefulHashMap in HBaseStore throws ConcurrentModificationException
> -----------------------------------------------------------------------------------
>
> Key: GORA-173
> URL: https://issues.apache.org/jira/browse/GORA-173
> Project: Apache Gora
> Issue Type: Bug
> Affects Versions: 0.1.1-incubating
> Reporter: raf shin
>
> We are using just one thread working with HBaseStore in Nutch.
> HBaseStore.put throws the following exception. I think the same problem was in CassandraStore that is
> solved now. But the problem in HBaseStore remains. We used the same solution for HBaseStore but
> it didn't work.
> java.util.ConcurrentModificationException
> at java.util.HashMap$HashIterator.nextEntry(HashMap.java:793)
> at java.util.HashMap$EntryIterator.next(HashMap.java:834)
> at java.util.HashMap$EntryIterator.next(HashMap.java:832)
> at org.apache.gora.hbase.store.HBaseStore.put(HBaseStore.java:267)
> at org.nise.hiva.fetcher.FetcherReducer$WriteThread.run(FetcherReducer.java:1557)
> Line 266 in HBaseStore:
> StatefulHashMap<Utf8, ?> map = (StatefulHashMap<Utf8, ?>) o;
> for (Entry<Utf8, State> e : map.states().entrySet()) {
> Utf8 mapKey = e.getKey();
> switch (e.getValue()) {
> case DIRTY:
> byte[] qual = Bytes.toBytes(mapKey.toString());
> byte[] val = toBytes(map.get(mapKey), field.schema().getValueType());
> put.add(hcol.getFamily(), qual, val);
> hasPuts = true;
> break;
> case DELETED:
> qual = Bytes.toBytes(mapKey.toString());
> hasDeletes = true;
> delete.deleteColumn(hcol.getFamily(), qual);
> break;
> }
> ***************************************************************************
> Our solution was :
> Set<Utf8> keys = map.keySet();
> Utf8 [] arr = (Utf8[])keys.toArray();
> for(Utf8 key: arr){....}
> But it didn't work and we throw the same exception.
> Our version is 0.1 but i checked the trunk and did't see much change in that code so the trunk code
> may throw the same exception.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira