You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@accumulo.apache.org by "Christopher Tubbs (JIRA)" <ji...@apache.org> on 2013/04/30 23:40:16 UTC
[jira] [Resolved] (ACCUMULO-1355) Accumulo ingestion hangs/fails
for 300 MB+ file, local machine memory usage grows to 10GB+
[ https://issues.apache.org/jira/browse/ACCUMULO-1355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Christopher Tubbs resolved ACCUMULO-1355.
-----------------------------------------
Resolution: Cannot Reproduce
Assignee: Christopher Tubbs (was: Billie Rinaldi)
Closing ticket, as it doesn't appear to be an Accumulo bug.
But don't worry, we may still be able to help! Please follow up on the [user@accumulo.apache.org mailing list|http://accumulo.apache.org/mailing_list.html], or in the IRC #accumulo room on irc.freenode.net, for further assistance.
When asking for assistance on the mailing list, please provide the version of Accumulo, the version of Hadoop, the version of Thrift (if different than the one provided with Accumulo), the version of Zookeeper, and the OS/platform you're using.
If it turns out there is a bug that is causing the issue you're seeing, we can reopen this ticket or create a new one against the affected component(s).
> Accumulo ingestion hangs/fails for 300 MB+ file, local machine memory usage grows to 10GB+
> ------------------------------------------------------------------------------------------
>
> Key: ACCUMULO-1355
> URL: https://issues.apache.org/jira/browse/ACCUMULO-1355
> Project: Accumulo
> Issue Type: Bug
> Components: client, thrift, tserver
> Affects Versions: 1.4.2, 1.4.3
> Reporter: Nasheb Ismaily
> Assignee: Christopher Tubbs
> Priority: Blocker
>
> *Bug:*
> I am attempting to ingest a 300+ MB file into Accumulo, however the ingestion process hangs and my local machines memory consumption grows to 10GB+.
> The mutation is never put into accumulo.
> *The file I am ingesting can be found here (needs to be unzipped):* http://www.epa.gov/ttn/atw/nata2005/emissions_mdbzip/2005natav3_ei_ca.zip
>
> *I’ve attached a snippet of code used to ingest the data here:* http://pastebin.com/Yh4V6nng
> *Initial Investigation:*
> While attempting to upload the file, the thread which sends the mutation to the tablet ‘waits’ in the waitRTE() method which is called by synchronized void flush() in TabletServerBatchWriter.java and is never ‘woken up’ by a notify call. While ‘waiting’ my local machines memory allocation grows to 10 GB+.
> *Thread Stack Trace (from java profiler):*
> java.nio.Bits.copyFromArray(Bits.java:699)
> java.nio.DirectByteBuffer.put(DirectByteBuffer.java:360)
> java.nio.DirectByteBuffer.put(DirectByteBuffer.java:331)
> sun.nio.ch.IOUtil.write(IOUtil.java:35)
> sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:336)
> org.apache.hadoop.net.SocketOutputStream$Writer.performIO(SocketOutputStream.java:55)
> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142)
> org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:146)
> org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:107)
> java.io.BufferedOutputStream.write(BufferedOutputStream.java:105)
> org.apache.thrift.transport.TIOStreamTransport.write(TIOStreamTransport.java:145)
> org.apache.thrift.transport.TFramedTransport.flush(TFramedTransport.java:157)
> org.apache.accumulo.core.client.impl.ThriftTransportPool$CachedTTransport.flush(ThriftTransportPool.java:299)
> org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Client.send_applyUpdates(TabletClientService.java:449)
> org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Client.applyUpdates(TabletClientService.java:436)
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> java.lang.reflect.Method.invoke(Method.java:597)
> org.apache.accumulo.cloudtrace.instrument.thrift.TraceWrap$2.invoke(TraceWrap.java:84)
> com.sun.proxy.$Proxy7.applyUpdates(Unknown Source)
> org.apache.accumulo.core.client.impl.TabletServerBatchWriter$MutationWriter.sendMutationsToTabletServer(TabletServerBatchWriter.java:768)
> org.apache.accumulo.core.client.impl.TabletServerBatchWriter$MutationWriter.access$1400(TabletServerBatchWriter.java:536)
> org.apache.accumulo.core.client.impl.TabletServerBatchWriter$MutationWriter$SendTask.send(TabletServerBatchWriter.java:700)
> org.apache.accumulo.core.client.impl.TabletServerBatchWriter$MutationWriter$SendTask.run(TabletServerBatchWriter.java:671)
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> java.util.concurrent.FutureTask.run(FutureTask.java:138)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
> java.lang.Thread.run(Thread.java:680)
> *Tserver log error:*
> 2013-04-26 15:36:25,253 [server.TNonblockingServer] ERROR: Unexpected exception while invoking!
> java.lang.RuntimeException: No Such SessionID
> at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler.applyUpdates(TabletServer.java:1433)
> at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:601)
> at org.apache.accumulo.cloudtrace.instrument.thrift.TraceWrap$1.invoke(TraceWrap.java:59)
> at com.sun.proxy.$Proxy2.applyUpdates(Unknown Source)
> at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor$applyUpdates.process(TabletClientService.java:2315)
> at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor.process(TabletClientService.java:2037)
> at org.apache.accumulo.server.util.TServerUtils$TimedProcessor.process(TServerUtils.java:154)
> at org.apache.thrift.server.TNonblockingServer$FrameBuffer.invoke(TNonblockingServer.java:631)
> at org.apache.accumulo.server.util.TServerUtils$THsHaServer$Invocation.run(TServerUtils.java:202)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at org.apache.accumulo.core.util.LoggingRunnable.run(LoggingRunnable.java:34)
> at java.lang.Thread.run(Thread.java:722)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira